devonfw guide

Read: Framework Test Class

Every logged information is saved in a separate test file, as a result of parallel tests execution.

The places they are saved:

Console output:

You can generate a report using one of the following commands:

Since mrchecker-core-module version 5.6.2.1:

Prior to mrchecker-core-module version 5.6.2.1:

A report will be generated into temp folder. Web server with results will start. You can additionally configure the server timeout. The default value is "3600" (one hour).

System property allure.serve.timeout.

Since mrchecker-core-module version 5.6.2.1:

Prior to mrchecker-core-module version 5.6.2.1:

A report will be generated tо directory: target/site/allure-maven/index.html

NOTE: Please open index.html file under Firefox. Chrome has some limitations to presenting dynamic content. If you want to open a report with a Chromium based Web Browser, you need to launch it first with --allow-file-access-from-files argument.

A report is created here allure-app-under-test\target\site\allure-report\index.html

NOTE: Please open index.html file under Firefox. Chrome has some limitations to presenting dynamic content. If you want to open a report with a Chromium based Web Browser, you need to launch it first with --allow-file-access-from-files argument.

In our case, we’ll use the Allure Jenkins plugin. When integrating Allure in a Jenkins job configuration, we’ll have direct access to the build’s test report.

There are several ways to access the Allure Test Reports:

Afterwards you’ll be greeted with either the general Allure Dashboard (showing the newest build) or the Allure Dashboard for a specific (older) build.

The Dashboard provides a graphical overview on how many test cases were successful, failed or broken.

The defects tab lists out all the defects that occurred, and also descriptions thereof. Clicking on a list item displays the test case which resulted in an error. Clicking on a test case allows the user to have a look at the test case steps, as well as Log files or Screenshots of the failure.

The graph page includes a pie chart of all tests, showing their result status (failed, passed, etc.). Another graph allows insight into the time elapsed during the tests. This is a very useful information to find and eliminate possible bottlenecks in test implementations.

Regression testing is a type of software testing which verifies that software which was previously developed and tested still performs the same way after it was changed or interfaced with another software.

Since mrchecker-core-module version 5.6.2.1:

Where:

Most commonly used filters to build a Test Suite are ones using:

Example:

Prior to mrchecker-core-module version 5.6.2.1:

Where:

Most commonly used filters to build Test Suite are ones using:

Example:

Since mrchecker-core-module version 5.6.2.1:

Where:

Prior to mrchecker-core-module version 5.6.2.1:

Where:

Since mrchecker-core-module version 5.6.2.1:

Tag name: TestsTag1 annotation

Tag name: TestsSmoke annotation

Tag name: TestSelenium annotation

Prior to mrchecker-core-module version 5.6.2.1:

Tag name: TestsTag1.class

Tag name: TestsSmoke.class

Tag name: TestSelenium.class

OR

In the first case the arguments are directly mapped to the test method parameters. In the second case the array is passed as the argument.

Prior to mrchecker-core-module version 5.6.2.1

Parameters that are passed into tests using the @Parameters annotation must be _Object[]_s

The parameters must be primitive objects such as integers, strings, or booleans. Each set of parameters is contained within a single string and will be parsed to their correct values as defined by the test method’s signature.

A separate method can be defined and referred to for parameters. This method must return an Object[] and can contain normal objects.

A separate class can be used to define parameters for the test. This test must contain at least one static method that returns an Object[], and its name must be prefixed with provide. The class could also contain multiple methods that provide parameters to the test, as long as they also meet the required criteria.

and CSV file structure

and CSV file structure

and Mapper implementation

Prior to mrchecker-core-module version 5.6.2.1

Tests use the annotation @FileParameters to inject CSVs file.

A CSV can also be used to contain the parameters for the tests. It is pretty simple to set up, as it’s just a comma-separated list.

and CSV file structure

and CSV file structure

and Mapper implementation

Cucumber / Selenium

Business and IT don’t always understand each other. Very often misunderstandings between business and IT result in the costly failure of IT projects. With this in mind, Cucumber was developed as a tool to support human collaboration between business and IT.

Cucumber uses executable specifications to encourage a close collaboration. This helps teams to keep the business goal in mind at all times. With Cucumber you can merge specification and test documentation into one cohesive whole, allowing your team to maintain one single source of truth. Because these executable specifications are automatically tested by Cucumber, your single source of truth is always up-to-date.

Cucumber supports testers when designing test cases. To automate these test cases, several languages can be used. Cucumber also works well with Browser Automation tools such as Selenium Webdriver.

Dynamic variable values are stored under path mrchecker-app-under-test\src\resources\enviroments\environments.csv.

NOTE: As environments.csv is Comma-separated file, please be aware of any edition and then save it under Excel.

Some types of data you might want to store as environment settings are sensitive in nature (e.g. passwords). You might not want to store them (at least not in their plaintext form) in your repository. To be able to encrypt sensitive data you need to do following:

To map values from external file with Page class you ought to use class GetEnvironmentParam.

Therefore when you add new variable (row) in environments.csv you might need to add this variable to GetEnvironmentParam.

Dynamic variable values are stored under mrchecker-app-under-test\src\resources\enviroments\environments.csv.

NOTE: As environments.csv is a comma-separated file, please be careful while editing and then save it under Excel.

Some types of data you might want to store as environment settings are sensitive in nature (e.g. passwords). You might not want to store them (at least not in their plaintext form) in your repository. To be able to encrypt sensitive data you need to do following:

To map values from external file with Page class you ought to use class GetEnvironmentParam

Therefore when you add new variable (row) in environments.csv you might need to add this variable to GetEnvironmentParam.

Prior to mrchecker-core-module version 5.6.2.1

Read: My first Selenium Test

More information on Selenium on the Production Line can be found here.

tl;dr

The Production Line has containers running Chrome and Firefox Selenium Nodes. The communication with these nodes is accomplished using Selenium Grid.

Having issues using Selenium on the Production Line? Check the Production Line issue list, maybe it’s a known issue that can be worked around.

A tester, through their test script, can command WebDriver to perform certain actions on the WAUT on a certain browser. The way the user can command WebDriver to perform something is by using the client libraries or language bindings provided by WebDriver.

By using the language-binding client libraries, a tester can invoke browser-specific implementations of WebDriver, such as Firefox Driver, IE Driver, Opera Driver, and so on, to interact with the WAUT of the respective browser. These browser-specific implementations of WebDriver will work with the browser natively and execute commands from outside the browser to simulate exactly what the application user does.

After execution, WebDriver will send the test result back to the test script for developer’s analysis.

Source folder: allure-app-under-test/src/main/java

Name: com.example.selenium.pages.YOUR_PROJECT

Classes being created inside of this new package have to extend the BasePage class. As a result, a few abstract methods from BasePage have to be implemented.

The example above demonstrates a minimum valid Page Object class with all required methods included.

To initialize web elements, a large variety of selectors can be used.

We recommend creating a private and constant field for every web element you’d like to represent in Java. Use the guide above to find the preferred selector and place it in the code below at "WEB_ELEMENT_SELECTOR".

As soon as you create the selector above, you can make use of it to initialize a WebElement object.

Note: The examples displayed in the cssSelector.docx file use the Selenium method driver.findElement() to find elements. However, using this framework we recommend findDynamicElement() or findQuietlyElement().findDynamicElement() allows waiting for dynamic elements, for example buttons that pop up.

To interact with the page object, we recommend creating methods for each action.

Creating a method like the one above allows the test case to run something like googleSearchPage.enterGoogleSearchInput("Hello") to interact with the page object.

For code uniformity and readability, we provide a few method naming conventions.

A css selector is used to select elements from an HTML page.

Selection by element tag, class or id are the most common selectors.

This text element (p) can be found by using any one of the following selectors:

When a class or an id attribute is not sufficient to identify an element, other attributes can be used as well, by using "[attribute=value]": For example:

This can be selected by using the entire value: "a[href='https://ns.nl/example.html'\]". For selecting links starting with, containing, ending with see the list below.

The css selectors can be stacked, by appending them:

In the example above, the link element to nsinternational can be obtained with: "#2 a".

A good list with CSS Selectors can be found at W3Schools:
https://www.w3schools.com/cssref/css_selectors.asp

Service Virtualization has become recognized as one of the best ways to speed up testing and accelerate your time to market.

Service virtualization lets you automatically execute tests even when the application under test’s dependent system components (APIs, third-party applications, etc.) cannot be properly accessed or configured for testing. By simulating these dependencies, you can ensure that your tests will encounter the appropriate dependency behaviour and data each and every time that they execute.

Service virtualization is the simulation of interfaces – not the virtualization of systems.

According to Wikipedia’s service virtualization entry: Service virtualization emulates the behaviour of software components to remove dependency constraints on development and testing teams. Such constraints occur in complex, interdependent environments when a component connected to the application under test is:

For instance, instead of virtualizing an entire database (and performing all associated test data management as well as setting up the database for every test session), you monitor how the application interacts with the database, then you emulate the related database behaviour (the SQL queries that are passed to the database, the corresponding result sets that are returned, and so forth).

The most commonly discussed categories of test doubles are mocks, stubs and virtual services.

Stub: a minimal implementation of an interface that normally returns hardcoded data that is tightly coupled to the test suite. It is most useful when the suite of tests is simple and keeping the hardcoded data in the stub is not an issue. Some stubs are handwritten; some can be generated by tools. A stub is normally written by a developer for personal use. It can be shared with testers, but wider sharing is typically limited by interoperability issues related to software platform and deployment infrastructure dependencies that were hardcoded. A common practice is when a stub works in-process directly with classes, methods, and functions for the unit, module, and acceptance testing. Some developers will say that a stub can also be primed, but you cannot verify an invocation on a stub. Stubs can also be communicating "over the wire", for example, HTTP, but some would argue that they should be called virtual services in that case.

Mock: a programmable interface observer, that verifies outputs against expectations defined by the test. It is frequently created using a third party library, for example in Java that is Mockito, JMock or WireMock. It is most useful when you have a large suite of tests and a stub will not be sufficient because each test needs a different data set up and maintaining them in a stub would be costly. The mock lets us keep the data set-up in the test. A mock is normally written by a developer for personal use but it can be shared with testers. However, wider sharing is typically limited by interoperability issues related to software platform and deployment infrastructure dependencies that were hardcoded. They are most often work-in-progress directly with classes, methods, and functions for a unit, module, and acceptance testing. Mock provides responses based on a given request satisfying predefined criteria (also called request or parameter matching). A mock also focuses on interactions rather than state so mocks are usually stateful. For example, you can verify how many times a given method was called or the order of calls made to a given object.

Virtual service: a test double often provided as a Software-as-a-Service (SaaS), is always called remotely, and is never working in-process directly with methods or functions. A virtual service is often created by recording traffic using one of the service virtualization platforms instead of building the interaction pattern from scratch based on interface or API documentation. A virtual service can be used to establish a common ground for teams to communicate and facilitate artefact sharing with other development teams as well as testing teams. A virtual service is called remotely (over HTTP, TCP, etc.) normally supports multiple protocols (e.g. HTTP, MQ, TCP, etc.), while a stub or mock normally supports only one. Sometimes virtual services will require users to authorize, especially when deployed in environments with enterprise-wide visibility. Service virtualization tools used to create virtual services will most often have user interfaces that allow less tech-savvy software testers to hit the ground running, before diving into the details of how specific protocols work. They are sometimes backed by a database. They can also simulate non-functional characteristics of systems such as response times or slow connections. You can sometimes find virtual services that provide a set of stubbed responses for given request criteria and pass every other request to a live backend system (partial stubbing). Similar to mocks, virtual services can have quite complex request matchers, that allow having one response returned for many different types of requests. Sometimes, virtual services simulate system behaviours by constructing parts of the response based on request attributes and data.

It is often difficult to say definitely which of the following categories a test double fits into. They should be treated as a spectrum rather than strict definitions.

This is a quite common application structure, where we have any of the following in Application Under Test (AUT):

This classic application is quite fragile for development and/or test process. Especially so, if the component (WebAPI) connected to the Application Under Test is:

You can find the full list of such "classic application structure" limitations here What-is-service-virtualization.

*Service virtualization is the key solution to address such a list of impediments. *

For simplicity, AUT connects to other components by TCP/IP protocol. Therefore AUT has an IP address and port number where given components operate. To plug in virtualization server, the author of AUT ought to switch IP and port to "proxy server" instead of real endpoint component (WebAPI) . Finally, "proxy server" maps requests come from AUT with either virtual assets or real endpoint component (WebAPI). How do maps work in such a "proxy server"? Have a look here How-to-make-virtual-asset

Therefore AUT is build either with:

or

Full article here Wiremock record-playback.

First, start an instance of WireMock running standalone. Once that’s running, visit the recorder UI page at http://localhost:8080/__admin/recorder (assuming you started WireMock on the default port of 8080).

Enter the URL you wish to record from in the target URL field and click the Record button. You can use http://example.mocklab.io to try it out.

Now you need to make a request through WireMock to the target API so that it can be recorded. If you’re using the example URL, you can generate a request using curl:

Now click stop. You should see a message indicating that one stub was captured.

You should also see that a file has been created called something like recordables_123-40a93c4a-d378-4e07-8321-6158d5dbcb29.json under the mappings directory created when WireMock started up, and that a new mapping has appeared at http://localhost:8080/__admin/mappings.

Requesting the same URL again (possibly disabling your wifi first if you want a firm proof) will now serve the recorded result:

An example of how such a record can be achieved

EMPTY

Link to full file structure: REST_FarenheitToCelsiusMethod_Test.java

If docker is not already installed on machine (this should be checked during C2C creation), install docker, docker-compose, apache2-utils, openssl (You can use script to install docker & docker-compose OR refer to this post and add Alias for this machine <C2C_Alias_Name>):

Dockerfile:

Execute the following steps with a specified version to build a docker image and push it to the repository :

To run a docker image, execute the following command:

Where:

-p - publish a container’s port to the host

-v - bind mount a volume. WireMock server creates two directories under the current one: mappings and __files. It is necessary to mount directories with already created mappings and responses to make it work.

-restart always - restart policy to apply when a container exists

All of the parameters are described in: official docker documentation

Record

WireMock can create stub mappings from requests it has received. Combined with its proxying feature, this allows you to "record" stub mappings from interaction with existing APIs.

Record and playback (Legacy): documentation

Once it’s started and request is sent to it, it will be redirected to "http://search.twitter.com" and traffic (response) is saved to files in mappings and __files directories for further use.

Record and playback (New): documentation

When the WireMock server starts, it creates two directories under the current one: mappings and __files. To create a stub, it is necessary to drop a file with a .json extension under mappings.

Run docker with mounted volumes

Mappings are in a repository. It is necessary to mount directories with already created mappings and responses to make it work:

The description of how to build and run docker is available under: Docker run command description

Recorded mappings

Recorded mappings are kept in the project repository.

To enable the connection from Jenkins to Virtual Server (C2C), it is necessary to create a user and map them to docker group user. It can be done using the following command:

To set the password for a wiremock user:

SSH keys serve as a means of identifying yourself to an SSH server using public-key cryptography and challenge-response authentication. One immediate advantage this method has over traditional password is that you can be authenticated by the server without ever having to send your password over the network.

To create an SSH key, log in as wiremock (previously created user).

The .ssh directory is not by default created below user home directory. Therefore, it is necessary to create it:

Now we can proceed with creating an RSA key using ssh-keygen (a tool for creating new authentication key pairs for SSH):

A key should be created under /.ssh/id_rsa Appending the public keys to authorized_keys:

To add an SSH key to Jenkins, go to credentials in your job location. Choose the folder within credentials, then 'global credentials', 'Add credentials'. Fill in the fields. Finally, the entry should be created.

The description of how to use SSH Agent plugin in Jenkins pipeline can be found under: https://www.karthikeyan.tech/2017/09/ssh-agent-blue-ocean-via-jenkins.html

Example of use:

Where: env.WIREMOCK_CREDENTIALS is a credential id of previously created wiremock credentials. Now that it is present, we can execute commands on a remote machine, where in ssh command: env.WIREMOCK_USERNAME - user name of user connected with configured private key env.WIREMOCK_IP_ADDRESS - ip address of the machine where this user with this private key exists

To pull the repository on a remote machine, it is necessary to use the previously described SSH Agent plugin. An example of use:

Where:

withCredentials allows various kinds of credentials (secrets) to be used in idiosyncratic ways. Each binding will define an environment variable active within the scope of the step. Then the necessary commands are executed:

cd …​ - command will change from current directory to the specified directory with git repository

git fetch …​ ;git reset …​ ;git clean …​ - pull from GIT branch. Git pull or checkout are not used here to prevent the situation with wrong coding between Mac OSX/Linux etc.

PLEASE remember that when using this script for the first time, the code from previous block should be changed to:

New settings file is pushed to the repository. Example configuration:

Address of service (backend) should be changed to wiremock address as it is shown on listing to change the default route.

New versions of application are prepared by Jenkins job.

Installation of an application is actually executed in a non-automated way using SeeTest environment.

Jenkinsfile:

stageStartVirtualServer.groovy:

stageMapApiRequests.groovy:

stageInstallApplication.groovy:

stageUITests.groovy:

Configuration

It is possible to configure Jenkins job in two ways. First one is to edit the Jenkinsfile. All of the properties are in properties collection as below:

Second way is to add properties in 'Configure job'. All of the properties there are overriding properties from Jenkinsfile (the have the highest priority). They can then be set durring 'Build with Paremeters' process.

Reports

After a job execution 'Allure report' and 'Cucumber-JVM' reports should be visible. If any tests fail, You can check on which screen (printscreen from failures is attached, why and etc.)

Application Security is concerned with Integrity, Availability and Confidentiality of data processed, stored and transferred by the application.

Application Security is a cross-cutting concern which touches every aspect of the Software Development Lifecycle. You can introduce some SQL injection flaws in your application and make it exploitable, but you can also expose your secrets (which will have nothing to do with code itself) due to poor secret management process, and fail as well.

Because of this and many other reasons, not every aspect of security can be automatically verified. Manual tests and audits will still be needed. Nevertheless, every security requirement which is automatically verified will prevent code degeneration and misconfiguration in a continuous manner.

Security tests can be performed in many different ways, such as:

In the current release of the Security Module, the main focus will be Unit Tests.

Although the most common choice of environment for running security tests on will be integration(the environment offers the right stability and should mirror the production closely), it is not uncommon for some security tests to run on production as well. This is done for e.g. TLS configuration testing to ensure proper configuration of the most relevant environment in a continuous manner.

Database module is based on Object-Relational Mapping programming technique. All functionalities are built using Java Persistence API but examples use Hibernate as a main provider.

This module was written to allow the use of any JPA provider. The structure is represented in the schema below.

MrChecker E2E Mobile test Module is a suitable solution for testing Remote Web Design, Mobile Browsers and application. A user can write tests suitable for all mobile browsers with a full range of resolution. The way of working is similar to Selenium and uses the same rules and patterns as the Web Driver. For more information please look in the Selenium test module.

Creating Selenium test cases can result in an unmaintainable project. One of the reasons is that too many duplicated code is used. Duplicated code could be caused by the duplicated functionality and this will result in duplicated usage of locators. The disadvantage of duplicated code is that the project is less maintainable. If some locator will change, you have to walk through the whole test code to adjust locators where necessary. By using the page object model we can make non-brittle test code and reduce or eliminate duplicate test code. Beside of that it improves the readability and allows us to create interactive documentation. Last but not least, we can create tests with less keystroke. An implementation of the page object model can be achieved by separating the abstraction of the test object and the test scripts.

It is build on the top of the Appium library. Appium is an open-source tool for automating native, mobile web, and hybrid applications on iOS mobile, Android mobile, and Windows desktop platforms. Native apps are those written using iOS, Android, or Windows SDKs. Mobile web apps are web apps accessed using a mobile browser (Appium supports Safari on iOS and Chrome or the built-in 'Browser' app on Android). Hybrid apps have a wrapper around a "webview" - a native control that enables interaction with web content.

To execute each test with chosen connected mobile devices, it is required to use specific arguments in Run configuration.

Default supported arguments in MrChecker:

Example usage:

Check also:

+ My First Selenium Test

+ How to use Mobile test Module

+ Example TestCase

+ Full list of Generic Capabilities

+ List of additional capabilities for Android

+ List of additional capabilities for iOS

DevOps consists of a mixture of three key components in a technical project:

Using E2E MrChecker Test Framework it is possible to cover the majority of these areas.

For QA engineers, it is essential to take care of the product code quality.

Therefore, we have to understand, that a test case is also code which has to be validated against quality gates. As a result, we must test our developed test case like it is done during standard Software Delivery Life Cycle.

The CI procedure has been divided into transparent modules. This solution makes configuration and maintenance very easy because everyone is able to manage versions and customize the configuration independently for each module. A separate security module ensures the protection of your credentials and assigned access roles regardless of changes in other modules.

Your CI process will be matched to the current project. You can easily go back to the previous configuration, test a new one or move a selected one to other projects.

DevOps module supports a delivery model in which executors are made available to the user as needed. It has such advantages as:

Once you have implemented the module, you can learn more about it here:

There are two ways to set up your Continuous Integration environment:

Jenkins is an Open Source Continuous Integration Tool. It allows the user to create automated build jobs which will run remotely on so called Jenkins Slaves. A build job can be triggered by several events, for example on new pull request on specified repositories or timed (e.g. at midnight).

If you are starting a new Jenkins instance for your tests, we’d suggest using the pre-configured Docker image. This image already contains all the configurations and additional features.

The configurations are e.g. Plugins and Pre-Installed job setup samples. This way, you don’t have to set up the entire CI-Environment from the ground up.

Additional features from this docker image allow dynamic creation and deletion of Jenkins slaves, by creating Docker containers. Also, Cloud Solutions can be implemented to allow wide-spread load balancing.

CD from Jenkins point of view does not change a lot from Continuous Integration one.

Use the same Jenkins settings for Jenkins CD setup as for CI, please. link. The only difference is:

The default interaction with Jenkins required manual jobs. This keeps configuration of a job in Jenkins separate from source code. With Pipeline plugin users can implement a pipeline procedure in Jenkinsfile and store it in repository with other code. This approach is used in Mr Checker framework. More info: https://jenkins.io/solutions/pipeline/

Our CI & CD processes are divided into a few separate files: Jenkins_node.groovy is the file to manage all processes. It defines all operations executed on a Jenkins node, so all code in this file is closed in node closure. Workflow in Jenkinsfile:

Not all the steps must be present in the Jenkins files. This should be configured for particular job requirements.

Selenium Grid allows running web/mobile browsers test cases to fulfill basic factors, such as:

On a daily basis, a test automation engineer uses their local environments for test case execution/development. However, a created browser test case has to be able to run on any infrastructure. Selenium Grid enables this portability.

Full documentation of Selenium Grid can be found here and here.

'Vanilla flavour' Selenium Grid is based on two, not very complicated ingredients:

There are two options of Selenium Grid setup:

Advantages and disadvantages of both solutions:

Run the following command either in Eclipse or in Jenkins:

As a result of this command:

Pushing new image takes place in two steps. First save the image by adding container ID to the commit command and next use push: