Contents Introduction Requirements Engineering Project Management Software Design Detailed Design and Coding Quality Assurance Maintenance Analysis Design Testing Coding Operation and Maintenance Installation Require- ments Requirements Specification Planning PVK-HT03 bella@cs.umu.se

Quality Assurance Introduction Testing Testing Phases and Approaches Black-box Testing White-box Testing System Testing PVK-HT03 bella@cs.umu.se

What is Quality Assurance? QA is the combination of planned and unplanned activities to ensure the fulfillment of predefined quality standards. Constructive vs analytic approaches to QA Qualitative vs quantitative quality standards Measurement Derive qualitative factors from measurable quantitative factors Software Metrics PVK-HT03 bella@cs.umu.se

Approaches to QA Constructive Approaches Analytic approaches Syntax-directed editors Type systems Transformational programming Coding guidelines ... Analytic approaches Inspections Static analysis tools (e.g. lint) Testing Usage of methods, languages, and tools that ensure the fulfillment of some quality factors. Usage of methods, languages, and tools to observe the current quality level. PVK-HT03 bella@cs.umu.se

?! Fault vs Failure Different types of faults can lead to can lead to human error fault failure Different types of faults Different identification techniques Different testing techniques Fault prevention and -detection strategies should be based on expected fault profile PVK-HT03 bella@cs.umu.se

HP´s Fault Classification Fault origin: WHERE? Specification/ requirements Design Code Environment/ support Documen- tation Other Requirements or specifications Functionality HW interface SW interface User interface Functional description (Inter-)Process communications Data definition Module design Logic description Error checking Standards Logic Computation Data handling Module interface/ implementation Standards Test HW Test SW Integration SW Development tools Fault type: WHAT? Missing Unclear Wrong Changed Better way Fault mode: WHY? bella@cs.umu.se

Fault Profile of a HP Division See [Pfleeger 98]. PVK-HT03 bella@cs.umu.se

Testing Phases . SYSTEM IN USE! Unit test Integration Function Performance Acceptance (,) test Installation Component code . Tested component Integrated modules Functioning system Verified software Accepted, validated SYSTEM IN USE! Design specifications System functional requirements Other Customer specification User environment Pre-implementation PVK-HT03

Pre-Implementation Testing Inspections evaluation of documents and code prior to technical review or testing Walkthrough In teams Examine source code/detailed design Reviews More informal Often done by document owners Advantages Effective High learning effect Distributing system knowledge Disadvantages Expensive

Testing vs “Proofing” Correctness Verification Check the design/code against the requirements Are we building the product right? Validation Check the product against the expectations of the customer Are we building the right product? Testing Testing can neither proof that a program is error free, nor that it is correct Testing is the process in which a (probably unfinished) program is executed with the goal to find errors. [Myers 76] Testing can only show the presence of errors, never their absence. [Dijkstra 69] PVK-HT03 bella@cs.umu.se

Goals of Testing Detect deviations from specifications Debugging Regression testing Establish confidence in software Operational testing Evaluate properties of software Reliability Performance Memory use/leakage Security Usability PVK-HT03 bella@cs.umu.se

Principles of Software Testing* Complete testing is not possible Testing is creative and difficult A major objective of testing is defect detection Testing must be risk-based Testing must be planned Testing requires independence * Hetzel, The Complete Guide to Software Testing PVK-HT03 bella@cs.umu.se

Test Cases and Test Data Test case  test data Test data: Inputs devised to test the system Test case: Situation to test Inputs to test this situation Expected outputs Test are reproducible PVK-HT03 bella@cs.umu.se

Fundamental Steps of Software Testing Understand requirements and specifications Create the execution environment Select test cases Execute & evaluate test cases Evaluate test progress PVK-HT03 bella@cs.umu.se

Unit Testing Defn: Tests the smallest individually executable code units. Objective: Find faults in the units. Assure correct functional behavior of units. By: Usually programmers. Tools: Test driver/harness Coverage evaluator Automatic test generator PVK-HT03 bella@cs.umu.se

Test Methods Functional testing (black-box) Structural testing (white-box, glass-box) Uses code/detailed design is to develop test cases Typically used in unit testing Approaches: Coverage-based testing Symbolic execution Data flow analysis ... Functional testing (black-box) Uses function specifications to develop test cases Typically used in system testing Equivalence partitioning Border case analysis develop black-box test cases develop white-box test cases perform white-box testing perform black-box testing time PVK-HT03 bella@cs.umu.se

Test Preparation Exhaustive testing is prohibited, because of the combinatorial explosion of test cases Choose representative test data i paths to test #tests 1 X, Y 2 2 XX, XY, YX, YY 4 3 XXX, XXY, ... 8 ... 100 2100 for i := 1 to 100 do if a = b then X else Y; 2  2100 - 2 > 2,5  1030 With 106 tests/sec this would take 81016 years Choose test data (test cases) PVK-HT03 bella@cs.umu.se

How to Choose Test Data Example 1 Example 2 Both paths must be tested! How can I know there is a “path”? if ((x + y + z)/3 = x) then writeln( “x, y, z are equal”) else writeln( “x, y, z are unequal”); Test case 1: x=1, y=2, z=3 Test case 2: x=y=z=2 if (d = 0) then writeln( “division by zero”) else x = y/n; (*-----------------------------*) PVK-HT03 bella@cs.umu.se

Black-box Testing Test generation without knowledge of software structure Also called specification-based or functional testing Equivalence partitioning Boundary-value analysis Error Guessing .... PVK-HT03 bella@cs.umu.se

Equivalence Partitioning Input data Input- and output data can be grouped into classes where all members in the class behave in a comparable way. Define the classes Choose representatives Typical element Borderline cases Inputs causing anomalous behaviour System Outputs which reveal the presence of faults Output data class 1: x < 25 class 2: x >= 25 and x <= 100 class 3: x > 100 x  [25 .. 100] PVK-HT03 bella@cs.umu.se

Equivalence Partitioning Examples Module keeps a running total of amount of money received Input: AmountofContribution, specified as being a value from $0.01 to $99,999.99 Module prints an appropriate response letter Input: MemberStatus, specified as having a value of {Regular, Student, Retiree, StudioClub} 3 equivalence classes: Values from $0.01 to $99,999.99 (valid) Values less than $0.01 (invalid) Values greater than $99,999.99 (invalid) 2 equivalence classes: Values of {Regular, Student, Retiree, StudioClub} (valid) All other input (invalid) PVK-HT03 bella@cs.umu.se

White-box Testing Methods based on internal structure of code Approaches: Coverage-based testing Statement coverage Branch coverage Path coverage Data-flow coverage Symbolic execution Data flow analysis ... a b d c if a then b else c; d while a do b; c PVK-HT03 bella@cs.umu.se

Statement Coverage Every statement is at least executed once in some test 4 2 7 6 1 8 3 5 2 test cases: 12467; 13567 PVK-HT03 bella@cs.umu.se

Branch Coverage For every decision point in the graph, each branch is at least chosen once 4 2 7 6 1 8 3 5 2 test cases: 12467; 1358 PVK-HT03 bella@cs.umu.se

Condition Coverage Test all combinations of conditions in boolean expressions at least once Why in boolean expressions only? if (X or not (Y and Z) and ... then b; c := (d + e * f - g) div op( h, i, j); PVK-HT03 bella@cs.umu.se

Path Coverage Assure that every distinct paths in the control-flow graph is executed at least once in some test PVK-HT03 bella@cs.umu.se

Path Coverage Assure that every distinct paths in the control-flow graph is executed at least once in some test 4 2 7 6 1 8 3 5 PVK-HT03 bella@cs.umu.se

Path Coverage Assure that every distinct paths in the control-flow graph is executed at least once in some test 4 2 7 6 1 8 3 5 4 test cases: 12467; 1358; 1248; 13567 PVK-HT03 bella@cs.umu.se

Test Coverage–Example Statement Coverage: 5/10 = 50% Branch Coverage: 2/6 = 33% Path Coverage: 1/4 = 25% PVK-HT03 bella@cs.umu.se

Data-flow testing Def-use analysis: match variable definitions (assignments) and uses. Example: x = 5; … if (x > 0) ... Does assignment get to the use? PVK-HT03 bella@cs.umu.se

Data Flow Coverage All-uses coverage x :=1 x :=3 z :=x+y y :=2 r :=4 x :=2 z := 2*r z := 2*x-y Red path covers the defs y :=2; r :=4; x :=1 Blue path covers y :=2; x :=3. Does not cover x :=2 PVK-HT03 bella@cs.umu.se

Coverage-based Testing Advantages Systematic way to develop test cases Measurable results (the coverage) Extensive tool support Flow graph generators Test data generators Bookkeeping Documentation support Disadvantages Code must be available Does not (yet) work well for data-driven programs PVK-HT03 bella@cs.umu.se

Integration Testing Defn: Testing two or more units or components Objectives Interface errors Functionality of combined units; look for problems with functional threads By: Developers or Testing group Tools: Interface analysis; call pairs Issues: Strategy for combining units PVK-HT03 bella@cs.umu.se

Integration Testing How to integrate & test the system Top-down Bottom-up Critical units first Functionality-oriented (threads) Implications of build order Top-down => stubs; more thorough top-level Bottom-up => drivers; more thorough bottom-level Critical => stubs & drivers. A Test all B C D Test B,E,F Test C Test D,G E F G Test E Test F Test G PVK-HT03 bella@cs.umu.se

System Testing Defn: Test the functionality, performance, reliability, security of the entire system. By: Separate test group. Objective: Find errors in the overall system behavior. Establish confidence in system functionality. Validate that system achieves its desired non-functional attributes. Tools: User simulator. Load simulator PVK-HT03 bella@cs.umu.se

Realities of System Testing Available time for testing is short Compressing development risks introducing problems Compressing testing risks missing critical problems Testers want to start testing early System testing requires an available system Developers resist testing until system is “ready” To optimize use of the existing resources, use risk analysis. PVK-HT03 bella@cs.umu.se

Acceptance Testing Defn: Operate system in user environment, with standard user input scenarios. By: End user Objective: Evaluate whether system meets customer criteria. Determine if customer will accept system. Tools: User simulator. Customer test scripts/logs from operation of previous system. PVK-HT03 bella@cs.umu.se

Regression Testing Defn: Test of modified versions of previously validated system. By: System or regression test group. Objective: Assure that changes to system have not introduced new errors. Tools: Regression test base, capture/replay Issues: Minimal regression suite, test prioritization PVK-HT03 bella@cs.umu.se

Test Automation Automation has several meanings Test generation: Produce test cases by processing of specifications, code, model. Test execution: Run large numbers of test cases/suites without human intervention. Test management: Log test cases & results; map tests to requirements & functionality; track test progress & completeness PVK-HT03 bella@cs.umu.se

Issues of Test Automation Automating Test Execution Does the payoff from test automation justify the expense and effort of automation? Learning to use the automation tool is non-trivial Testers become programmers All tests, including automated tests, have a finite lifetime. Complete automated execution implies putting the system into the proper state, supplying the inputs, running the test case, collecting the result, verifying the result. PVK-HT03 bella@cs.umu.se

Best uses of automated test execution Load/stress tests--Nearly impossible to have 1000 human testers simultaneously accessing a database. Regression test suites--Tests collected from previous releases; run to check that updates don’t break previously correct operation. Sanity tests (smoke tests)--run after every new system build to check for obvious problems. PVK-HT03 bella@cs.umu.se

Test documentation Test plan: describes system and plan for exercising all functions and characteristics Test specification and evaluation: details each test and defines criteria for evaluating each feature Test description: test data and procedures for each test Test analysis report: results of each test PVK-HT03 bella@cs.umu.se

The Key Problems of Software Testing Selecting or generating the right test cases. Knowing when a system has been tested enough. Knowing what has been discovered/demonstrated by execution of a test suite. PVK-HT03 bella@cs.umu.se