Functional Testing with the Java Stack Test Runner

Disclaimer This is a training not a presentation. Please be prepared to: Learn Ask questions Participate in the lab Prerequisites: Knowledge of Maven (required) Java Stack ALM Training (recommended)

What we’ll cover… In this training, we will discuss What is functional testing? How is it unique from other forms of testing? What is the Java Stack Test Runner? How does it integrate with Maven and AnthillPro? How do I invoke Test Runner from my IDE? Where do I find my functional test results?

What we won’t cover… Topics that we will not have time to discuss are: Testing frameworks and APIs i.e. TestNG, Selenium, Spock, etc. Testing methodologies Writing tests Test case management tools

First, some background…

Stages of Testing What When 1. Unit Testing After code compilation 2. Integration testing After the build artifact is packaged. 3. Functional testing* After the build artifact is deployed and started. 4. Load testing After deployment to staging environment 5 End-user testing After code is “feature complete”. Testing generally falls into one of three categories based on the granularity of the test itself. We will discuss functional testing and how it differs from unit and integration testing.

Unit Testing Scope Purpose Responsible Party Failure: Narrow (single class or method) Purpose Validates small units of code Responsible Party Code developer Failure: Indicates code is broken Typically fails the build * Has direct dependencies on application code Unit Testing - SCOPE: narrow - PURPOSE: to validate a specific unit of code. For example: a. A single method or function. - RESPONSIBILITY: the developer of the code unit to be tested. - WHEN: Typically after code compilation. - TEST FAILURES a. An indication that the code is broken. b. Will typically fail the build.

Integration Testing Scope Purpose Responsible Party Failure: Medium (spans two integrated systems) Purpose Validate code that integrates with another system. (e.g. a database or web service) Responsible Party Code developer. Failure: Indicates failure of systems to communicate as expected. Does not typically fail the build. * Has direct dependencies on application code. Integration Testing - SCOPE: medium - PURPOSE: to validate a system of your code as it integrates with one or more other systems external to your code. For example: a. A database b. An external web service c. An operating system specific API. - RESPONSIBILITY: the developer(s) of the system whose integration is to be tested. - WHEN: Typically after unit test validation and packaging. (You don't want to run integration tests on broken code because these tests are meant to test the integration of two or more systems, not the code itself.) - TEST FAILURES a. As the integrated systems are typically outside the direct control of the system being tested, a test failure could mean one of many things: - A configuration issue. - A down system - etc. b. A test failure can, but does not typically, fail the build. c. A test failure indicates that the application may run, but with some broken functionality.

Functional Testing Scope Purpose Responsible Party Failure: Broad (end-to-end feature testing) Purpose Validate functional requirements and/or user stories Responsible Party QA resource or feature developer Failure: Indicates failure to meet user story requirements Results recorded in test case management and/or bug tracker * Has little or no direct dependencies on application code. Functional Testing - SCOPE: broad - PURPOSE: to validate a functional requirement or user story. This type of testing has also been called manual user testing, automated testing, or application testing. A tester or test agent represents a user or user agent executing some user story against a deployed and running application. For example: a. Access Login Page => Enter credentials => Click submit => Verify login success. b. Access a REST or SOAP endpoint => Enter authentication credentials => Request or post a resource => Validate the response. - RESPONSIBILITY: Either a dedicated QA resource or the developer(s) that wrote the application. For example, a QA resource will typically perform functional tests against a user-interactive web application, whereas a developer will typically be responsible for testing REST or SOAP APIs exposed by the application. In other words, functional testing is not solely the role of QA engineers! WHEN: After application deployment and startup, typically within a dedicated test environment. - TEST FAILURES a. Failed results indicate a regression or change in functionality or a failure to meet user story requirements. b. Failed results are often tracked and entered into a

Functional Testing and Application Life-cycle Management (ALM)

ALM Build Tools Maven Project definition Dependency management Plug-in configuration and management Build profiles AnthillPro Manages build processes Creation of build artifact(s) Deployment to server environment Functional test execution Maintains build life history Provides ability to re-execute a process on any build life ALM Functional Testing and Build Tools For maximum efficiency and productivity, unit, integration and functional tests should be capable of being executed in an automated fashion. In ICS, most projects use the following build tools: A. MAVEN: a life-cycle based tool for project build configuration. Provides a default life-cycle for compiling source code, running unit and integration tests, and deploying source artifacts to a shared repository. B. AnthillPro: a build management tool - UNIT TESTS: the Maven Surefire plug-in is designed for executing unit tests and reporting on their results. * Invoked during the "test" phase, just after the compile phase. * By default, unit test class names are expected to have the "Test" prefix or suffix. (The "TestCase" suffix is also allowed.) - INTEGRATION TESTS: The Maven Failsafe plug-in is designed for executing integration tests and reporting on their results. * Invoked during the "integration-test" phase, just after the "test" and "package" phases. * By default, integration test class names are expected to have the "IT" prefix or suffix. (The "ITCase" suffix is - FUNCTIONAL TESTS: Maven's default life-cycle provides no phase for the execution of functional tests

ALM Tools: Maven Life-cycles The default Maven build life-cycle (simplified): compile test (unit tests) package integration-test (integration tests) install (to local Maven repository) deploy (to remote Maven repository) Where do we plug in functional testing? ALM Tools & the Maven Life-cycle A simplified view of the default Maven life-cycle is as follows: compile test package integration-test install deploy Where does functional testing hook into this life-cycle?

ALM Tools: Maven Life-cycles The woes of having “one build to rule them all”: Functional Testing ≠ Integration Testing Confuses build with server deployment Must re-build to re-deploy or re-test Complicates build management Less productive use of build agents ALM Tools & the Maven Life-cycle A. In the past, we have typically deployed the application to be tested during a "pre-integration-test" phase, and then ran functional tests during the "integration-test" phase. This has a number of drawbacks: 1. It complicates the build process by introducing application deployment as part of integration testing. 2. It confuses functional tests with integration tests. 3. An application could not be re-deployed without re-building. 4. Functional tests could not be run separate from a build. 5. It unnecessarily mingles the "primary" intent of the build with the "secondary" processes of deployment and functional testing, resulting in various complicated configurations for build management tools, such as AntHill Pro. 6. The majority of the build process's time is spent on the secondary tasks of deploying the application and executing functional tests, leading to inefficient use of build agents in cases where only a build, deployment, or functional test execution was necessary.

ALM Tools: Maven Life-cycles The Java Stack ALM build life-cycle (simplified): alm-db (run database scripts) alm-deploy (deploy application to server) alm-test (run functional tests) alm-promote (run all previous phases) This life-cycle is available to the artifact produced by the “alm” module of a Java Stack project. The Java Stack ALM life-cycle for Maven builds In cooperation with the Development Platforms Build team, the Java Stack team defined a new life-cycle for secondary process execution: alm-db - for running database scripts alm-deploy - for application deployment alm-test - for functional testing alm-promote - for executing all previous phases The ALM life-cycle is made available to Maven project modules whose pom has the "alm" packaging type. Web application projects, web service projects, and standalone QA projects created from the Java Stack Starter will each contain an ALM module for the purpose of deploying to and/or running functional tests against a local or remote test environment.

ALM Tools: Maven Life-cycle Advantages of a second build life-cycle: Clean separation of configuration: One configuration for build One configuration for deployment and/or functional testing Separate processes: One process for building artifact One process for deployment and/or functional testing Simplified build management More productive use of build agents What are the advantages of running functional tests in a separate life-cycle? 1. Separate configuration. The configuration for building an application's artifact is not mingled and confused with the configuration for deploying the artifact to an environment and running functional tests against that environment. 2. Separate execution. The process for building the application's artifact is executed separately from the process that deploys the artifact to a test environment and that then runs tests against it. 3. Simplified build management 4. More productive use of build agents—that is, separating build tasks from deployment tasks allows each build agent to run in less time. So each build process is more targeted and focused on what is actually needed, rather than doing twenty minutes of extra work you didn’t need it to do.

ALM Tools: Test Runner In Stack 3.2, as part of its ALM tooling, the Java Stack introduced the “test-bundle” artifact as a part of its suite of testing plug-ins, collectively known as “Test Runner”: Why another artifact? Why another tool? In the past, functional tests could only be packaged in a non-executable jar file and could not be easily executed outside of Maven. This made executing functional tests from past code revisions cumbersome and time-consuming. For this reason, the Java Stack has introduced a new kind of artifact for packaging functional tests: the test bundle.

ALM Tools: The Test Bundle The “test” jar Not an executable jar Difficult to execute without help from Maven. Relied on maven to supply its dependencies. Difficult to execute tests from past code revisions. The “test-bundle” jar An executable jar Can be executed with or without Maven. Comes packaged with all its dependencies. Each test artifact represents a snapshot of the current code revision. The Test Bundle: A new kind of artifact

The “failsafe:test” goal The “stack-test:functional-test” goal ALM Tools: Test Runner The “failsafe:test” goal Supports TestNG Supports test suites Supports environment configuration Produces JUnit and TestNG reports. Executes tests from the target/classes folder The “stack-test:functional-test” goal Supports TestNG Supports test suites Supports environment configuration Produces JUnit and TestNG reports. Executes “test-bundle” artifact.

ALM Tools: Functional Test Goal Common configuration for stack-test:functional-test: testBundle – groupId and artifactId of QA test bundle testEnv – the test environment properties to load skipFTs – whether to skip functional tests suiteXmlResources – TestNG suites to load includes – test class patterns (e.g. **/*FT.class) systemPropertyVariables – custom system properties argLine – JVM arguments (e.g. memory options, debugger, etc.) For the additional options, see: http://code.lds.org/maven-sites/stack/modules/test/

ALM Tools: The Test Bundle Simple to upgrade existing QA projects. 1. Upgrade the project to Stack 3.2 Then change the QA pom’s packaging type from to 2. Add QA module as a dependency to the ALM module. <packaging>jar</packaging> <packaging>test-bundle</packaging> <dependency> <groupId>${project.groupId}</groupId> <artifactId>example-qa</artifactId> <version>${project.version}</version> <type>test-bundle</type> </dependency> 1. Simple to upgrade existing QA modules. Any existing Java Stack QA module created prior to Java Stack 3.2 can be changed into a test bundle by simply changing the packaging type from "jar" to "test-bundle". <packaging>test-bundle</packaging> 2. Can be used to package and run TestNG tests against any deployed web application regardless of the technology: Java, OpenWeb, XQuery, .NET, you name it! 3. Binds to the ALM module's "alm-test" phase so it can be run automatically after application deployment as part of a secondary process. Test Runner was designed specifically for functional testing. Test Runner was designed with the QA engineer in mind. It is meant to test functionality, not code and therefore can be used to test applications written on any platform or language Java Web Applications Web Services .NET web application XQuery web applications Open Web applications Etc. Can be executed as part of build, a secondary process in Anthill, or independently from the command-line.

ALM Tools: The Test Bundle 3. Bind the functional-test goal to the alm-test phase: <plugin> <groupId>org.lds.stack.test</groupId> <artifactId>stack-test-maven-plugin</artifactId> <executions> <execution> <id>test</id> <goals><goal>functional-test</goal></goals> <phase>alm-test</phase> <configuration> <testBundle> <groupId>org.lds.training.testrunner.lab1</groupId> <artifactId>testrunner-lab1-qa</artifactId> </testBundle> </configuration> </execution> </executions> </plugin> 1. Simple to upgrade existing QA modules. Any existing Java Stack QA module created prior to Java Stack 3.2 can be changed into a test bundle by simply changing the packaging type from "jar" to "test-bundle". <packaging>test-bundle</packaging> 2. Can be used to package and run TestNG tests against any deployed web application regardless of the technology: Java, OpenWeb, XQuery, .NET, you name it! 3. Binds to the ALM module's "alm-test" phase so it can be run automatically after application deployment as part of a secondary process. Test Runner was designed specifically for functional testing. Test Runner was designed with the QA engineer in mind. It is meant to test functionality, not code and therefore can be used to test applications written on any platform or language Java Web Applications Web Services .NET web application XQuery web applications Open Web applications Etc. Can be executed as part of build, a secondary process in Anthill, or independently from the command-line.

ALM Tools: The Test Bundle 4. Modify build profiles to fit your needs <profile> <id>test</id> <properties> <skipFTs>false</skipFTs> <testEnv>test</testEnv> </properties> </profile> 1. Simple to upgrade existing QA modules. Any existing Java Stack QA module created prior to Java Stack 3.2 can be changed into a test bundle by simply changing the packaging type from "jar" to "test-bundle". <packaging>test-bundle</packaging> 2. Can be used to package and run TestNG tests against any deployed web application regardless of the technology: Java, OpenWeb, XQuery, .NET, you name it! 3. Binds to the ALM module's "alm-test" phase so it can be run automatically after application deployment as part of a secondary process. Test Runner was designed specifically for functional testing. Test Runner was designed with the QA engineer in mind. It is meant to test functionality, not code and therefore can be used to test applications written on any platform or language Java Web Applications Web Services .NET web application XQuery web applications Open Web applications Etc. Can be executed as part of build, a secondary process in Anthill, or independently from the command-line.

Lab #1 A Simple QA Test Bundle Project https://tech. lds

Lab #2 Running Functional Tests in AnthillPro https://tech. lds

Credits The Apache Maven Project: http://maven.apache.org/ Urban Code, for it’s AnthillPro documentation: http://www.urbancode.com/html/products/anthillpro/ The Dark Lord Sauron, for the quote, “One ring to rule them all”. (The Lord of the Rings, by J.R.R. Tolkien)

<code snippet block>