Testing Workflow In the Unified Process and Agile/Scrum processes

Purpose Testing Workflow: Purpose Verify the result from implementation by testing each build – Plan the tests in each iteration – Integration tests for every build within the iteration – System tests for the end of the iteration Design and implement tests by creating – test cases that specify what to test – test procedures that specify how to perform the tests – executable test components to automate the tests

Review: Kinds of testing Unit testing – Does an individual software component (unit) live up to its contract? – Typically done by the same person who codes the given unit – Tests become part of regression test suite Integration testing System testing Regression testing

Where do unit tests fit into UP? Unit tests are part of the implementation workflow They come before the integration testing They are typically designed and written by the programmer who writes the given module/unit (See separate slides on unit testing.)

Review: Kinds of testing Unit testing Integration testing – Does a collection of modules function correctly together as a group? System testing Regression testing

Review: Kinds of testing Unit testing Integration testing System testing – Does the entire system meet the end user’s requirements? Regression testing

Review: Kinds of testing Unit testing – Shows that individual software components (units) live up to their contracts – Typically done by the same person who codes the given unit – Tests become part of regression test suite Integration testing System testing Regression testing

Flow Testing Workflow: Flow

UP Phases Testing Workflow in UP Phases

Testing Workflow in Phases Generally, when there is an implementation result, there is a test --- so Test at each build Inception phase: initial test planning, prototype testing Elaboration phase: test architectural baseline Construction phase: significant testing at each build Transition phase: re-test fixes and regression tests

Testing Workflow: Other Tasks Regression tests: in a new build, re-apply tests from old builds to make sure nothing “broke” in new build Evolve test model create new test cases for next build refine prior test cases into regression tests remove obsolete tests and corresponding test procedures and components

Models Testing Workflow: Models Test model artifacts Test cases Test procedures Test components Test subsystem packages for complex test Other artifacts Test Plan List of product Defects Test Evaluation

Model Testing Workflow: Model Describes how executable components (e.g., builds) in the implementation model are tested by integration tests and system tests Describes how specific system aspects are to be tested e.g., usability testing of user interface or user manual

Test Case Testing Workflow: Test Case Specifies one way to test the system What to test (requirements) With which inputs Expected results Test conditions and constraints

Test Case Examples Test a use case or a scenario in a use case (black box) actor interaction, pre-conditions, action sequences, post-conditions Test a use-case realization (white box) verify interaction between components installation tests, configuration tests, negative (abuse) tests, stress tests Organize similar tests in input/result matrix Family of tests for different paths through a use case

Special Tests Verify the system can be installed on the customer platform and that it executes correctly when installed Configuration tests verify the system works correctly in different configurations Negative tests (abuse tests) discover what causes failure Test outside the intended design, e.g., wrong configuration, wrong input, etc. Reveals weaknesses in design Stress tests show problems with insufficient or shared resources

Test Procedures Testing Workflow: Test Procedures Specify how to perform one or several test cases or parts of them Instructions for tester in manual test case Instructions for interacting with test automation tool to create executable test components, then instructions to integrate and execute those test components One test procedure may cover several test cases One test case may require several test procedures Test procedure instructions often reflect flow-of events descriptions, including input values, how to enter input values, and how to verify results

Test Components Testing Workflow: Test Components Automate one or several test procedures or parts Provide test inputs, control and monitor test execution, and possibly analyze and report results Developed with scripting languages, programming languages, or test automation tools test drivers, test harnesses, and test scripts Complex test cases and procedures may require formal analysis, design, and implementation modeling of the test components

Test Plan Testing Workflow: Test Plan Describes testing strategies, resources and schedule Strategy What kind of tests to perform for each iteration Test objectives Required level of test and code coverage Test acceptance criteria Trade-off cost vs. improved quality and reduced risk Test Plan may be defined in conjunction with the model Artifacts from other workflows analysis (black box) design and implementation (white box) “Tests” may include artifact reviews

Test Results Testing Workflow: Test Results Results are the output data of the test and/or the behavior of the system during test Analysis of results: pass or defect Defect: a system anomaly (software fault or a defect found in artifact review) Capture symptom in sufficient detail to allow problem resolution Record defect in problem and resolution tracking system Test evaluation: summary of overall test results test-case coverage, code coverage defects and defect status

Testing Workflow: Test Results

Testing Workflow: Workers Test designer: responsible for integrity of the test model plan tests: test goals and schedule select and define test cases, test procedures evaluate test results Component engineer: responsible for development of test components that automate some of the test procedures consider independent test

Testing Workflow: Workers Integration tester: perform integration tests and report defects may be same person as system integrator in implementation workflow System tester: perform system tests and report defects focus on externally observable behavior consider users, beta customers, etc. as testers

Testing Workflow – THE PLAN Plan the Tests Strategy and Resources Define Integration Tests Define System Tests Design Regression Tests Define and Mutate Test Cases Implement Tests Evaluate Tests

Testing Workflow – THE PLAN Plan the Tests Strategy and Resources Describe testing strategy Inputs: use case model, supplemental requirements, design model Develop test cases and procedures that have a minimum of overlap test the most important test cases test high risk requirement items Estimate resources Estimate human and system resources Schedule the testing effort

Testing Workflow – THE PLAN Define Integration Tests Derive test cases from use-case realizations – design (component interaction) Develop tests with minimal overlap that test an interesting path or scenario through use case Primarily consider interaction diagrams Combinations of actor input, output and system start state that lead to interesting scenarios that employ the classes that participate in the diagrams

Testing Workflow – THE PLAN Define System Tests Test system as a whole Test combinations of use cases instantiated under different conditions Hardware configurations System loads Number of actors Database size, etc. Prioritize combinations required or are likely to function in parallel likely to influence each other if in parallel involve multiple processes frequently use system resources or use them in complex and unpredictable ways

Testing Workflow – THE PLAN Design Regression Tests Modify test cases from earlier builds to be used as regression tests in subsequent builds Make more flexible and resilient to changes to the system under test Specifically address previously found defects Automate where possible

Testing Workflow – THE PLAN Define and Mutate Test Cases Work test case by test case and suggest testing procedures for each one Reuse existing test cases May need to modify them to be reusable Align test procedures to subsystems under test Minimize overlap Isolates change to test resulting from change to subsystem

Testing Workflow – THE PLAN Implement Tests Create test components from test procedures With test automation tools Perform or specify actions Record actions as a test component When programming explicitly Design and implement using test procedures Provide input and output data generation, visualization, interpretation and management utilities Use spreadsheets and database applications

Testing Workflow – THE PLAN Evaluate Tests Perform the integration tests relevant to a build Manually perform test procedures Compare test results with the expected test results Investigate test results that deviate from the expected Faulty system and/or faulty test? Report defects to the component engineer responsible for the components likely to contain the fault Report defects to the test designers for evaluation of overall test results

Testing Workflow – THE PLAN Evaluate Tests (cont) Test designers prepare metrics of test acceptance with respect to system quality level Test completeness and coverage Percentage of test cases executed Percentage of code and paths tested Reliability Categorize defect severity Analyze trends of discovered defects Compare with prior experience to predict amount of testing still required to reach an acceptable defect level

Testing Workflow – THE PLAN Evaluate Tests (cont) Suggest actions Suggest further tests Relax test criteria Deliver parts of the system that passed; rework and retest parts that failed Document test completeness, reliability and actions

Testing Workflow – SUMMARY Test Model Test cases specify what to test Test procedures specify how to perform tests Test components automate test procedures Test Plan Test Evaluation Defects to be fed into other core work