1. Wilfried Mausz Testing Presentation for Software Quality Management Testing and Inspections

2. Wilfried Mausz Introduction “We do not write any wrong programs. Testing is just for wimps.” [old programmer’s proverb]

3. Wilfried Mausz Introduction Software is written by humans and humans make errors. There is no way to write perfect programs. Software faults have to be detected and removed, before giving it to the customer. About 50% of project time in software development is used for tests

4. Wilfried Mausz Introduction Test process has a major impact on Quality of the software Project schedule Project costs Satisfaction of the customer

5. Wilfried Mausz Agenda Errors, software failures, bugs Code Reviews Tests in the software engineering process Types of tests Cleanroom software engineering Test driven development –Example with NUnit Bug tracking –Example with Bugzilla

6. Wilfried Mausz Software failures Definitions Reasons The term “bug” Famous examples of software failures

7. Wilfried Mausz Software failures Software does not do what the requirements describe Reasons: –Specification may be wrong or have missing requirements –Specification may describe requirements that can not be implemented –System design may contain a fault –Program design may contain a fault –Wrong program code [Pfleeger, 2001]

8. Wilfried Mausz Definitions Errors are human actions, that produces an incorrect result Faults are the manifestations of errors Failures are the result of faults (when a program is executed) [Vliet, 2002]

9. Wilfried Mausz Types of faults Omission Some key aspects of the code is missing Example (C#): OleDbConnection myConn; myConn.Open();

10. Wilfried Mausz Types of faults Comission The written code is incorrect Example (C#): OleDbConnection myConn = new OleDBConnection(“wrong_connection_string”); myConn.Open();

11. Wilfried Mausz Bugs Term for inexplicable defects in engineering for decades erroneously attributed to Grace Hopper while working on Mark II [http://en.wikipedia.org/wiki/Computer_bug, 2004-10-30] First computer bug

12. Wilfried Mausz Famous computer bugs Ariane V takeoff, 1996 NASA Mars Climate Orbiter, 1999 Therac-25, 1985 – 1987 Y2K – bug

13. Wilfried Mausz Ariane 5 Source Code begin sensor_get(vertical_veloc_sensor); sensor_get(horizontal_veloc_sensor); vertical_veloc_bias := integer(vertical_veloc_sensor); horizontal_veloc_bias := integer(horizontal_veloc_sensor);... exception when numeric_error => calculate_vertical_veloc(); when others => use_irs1(); end; [http://wwwzenger.informatik.tu-muenchen.de/lehre/seminare/ semsoft/unterlagen_02/ariane/website/Ariane.Htm, 2004-11-05] Ariane 5 takeoff and explosion

14. Wilfried Mausz Reviews Definition Code walkthroughs Code inspections Numbers of its success Advantages, disadvantages

15. Wilfried Mausz Reviews Objective group of experts reviews requirements, program code and documentation Intention of finding faults Team consists of the programmer and three or four technical experts no members of the customers organisation [Pfleeger, 2001]

16. Wilfried Mausz Code walkthroughs Programmer presents code and documentation to review team Review-team discusses its correctness Informal Focus on the code, not the programmer Focus on finding faults, not correcting them [Pfleeger, 2001]

17. Wilfried Mausz Code inspections More formal than walkthrough Review team checks code and documentation against prepared list of issues Each team member studies the code and notes faults In a final group meeting these faults are discussed Lead by a team moderator Focus on the code, not the coder [Pfleeger, 2001]

18. Wilfried Mausz Success of inspections Fagan (1976): –67% of faults detected in inspection –Inspection had 38% fewer failures than walkthrough Ackermann, Buchwald and Lewski (1986): –93% of faults detected by inspections Jones (1977, 1991): –85% of faults detected during inspections Recommended standard practice at companies like IBM, HP, AT&T, …

19. Wilfried Mausz Advantages of inspections Failures are found at an early state Not only failures in the code but also design faults will be found Saves testing and debugging time Knowledge transfer to other project members Extremely high number of detected faults

20. Wilfried Mausz Disadvantages of inspections Programmers do not want others to critically review their code Inspections could be used to evaluate the programmer himself instead of his code Coordination of the review meetings needed

21. Wilfried Mausz Tests in the software engineering process Definitions Tests in software engineering process Types of tests

22. Wilfried Mausz Validation and verification Validation: –System has implemented all requirements –“Developer is building the right product” Verification: –Each function works correct –“Quality of the implementation” is checked Both has to be done in the testing process [Pfleeger, 2001]

23. Wilfried Mausz Testing Testing is often seen as proof that a program runs correct … but testing is to demonstrate the existence of faults [Pfleeger, 2001] Write tests first, then start coding Tests are documentation for the code [Beck]

24. Wilfried Mausz Stats on testing Errors in about 2 – 5% of all lines of written program code [http://panko.cba.hawaii.edu/HumanErr/ProgNorm.htm, 2004-11-06] About 3 – 6 defects in 1000 lines of code (after testting) [Voas et al.] Testing needs about 50% of the project shedule [Brooks, 1975] Programmers should spend 25 – 50% on testing [Beck]

25. Wilfried Mausz Bugs found during tests [Brooks, 1975] Bugs found since installation

26. Wilfried Mausz Testing in the V-model Requirements analysis System design Program design Coding Unit & Integration testing System testing Acceptance testing Operation & Maintenance [Pfleeger, 2001] Validate requirements Verify design

27. Wilfried Mausz Type of Tests Alpha Beta Release Candidate Types of tests –Black box –White box

28. Wilfried Mausz Alpha versions Definition: An alpha version or alpha release represents the first version of a computer program or other product, likely to be very unstable but useful for demonstrating internally and to select customers. Some developers refer to this stage as a preview, as a technical preview or as an early access. [http://en.wikipedia.org/wiki/Alpha_test, 2004-10-31]http://en.wikipedia.org/wiki/Alpha_test

29. Wilfried Mausz Beta version Definition: The beta version of a product still awaits full debugging or full implementation of all its functionality, but satisfies a majority of the requirements. Beta versions (or just betas) stand at an intermediate step in the full development cycle. Developers release them to a group of beta testers (or, sometimes, to the general public) for a user test. The testers report any bugs that they found, features they would like to see in the final version, etc. [http://en.wikipedia.org/wiki/Alpha_test, 2004-10-31]http://en.wikipedia.org/wiki/Alpha_test

30. Wilfried Mausz Alpha, beta testing Alpha test: –Before giving a system to customers, it can be released for in- house use and test in the own company Beta test: –A pilot system at a selected group of customers. Commonly done when a system is relased to a wide varity of customers. [Pfleeger, 2001]

31. Wilfried Mausz Release Candidate The term release candidate can refer to a final product, ready to release unless fatal bugs emerge. In this stage, the product features all designed functionalities and no known showstopper class bugs. Microsoft Corporation often uses the term release candidate. [http://en.wikipedia.org/wiki/Alpha_test, 2004-10-31]http://en.wikipedia.org/wiki/Alpha_test

32. Wilfried Mausz Black box Internal working of a program is not known by the tester Tester only knows input of a component and its expected outcome Program Code is not reviewed by the test Only the specification of a component is needed Also knows as: closed box testing, functional testing [http://www.webopedia.com/TERM/B/Black_Box_Testing.html,2004-10-29] Component

33. Wilfried Mausz Black Box testing Advantages: –Programmer and tester are independent of each other –Tester does not need knowledge of used technologies –Tests from the users point of view –Tests can be designed as soon as specification is availible Disadvantages: –It is not possible to test every input combination, so many program path will not be tested –Difficult to find representative data for test [http://www.webopedia.com/TERM/B/Black_Box_Testing.html,2004-10-29]

34. Wilfried Mausz White box Knowledge of the source code is used to define tests Problem of finding test cases in black box testing is reduced to values relevant in the program code Errors by omission not detectable Tester needs information about code Design of tests can only be done after the implementation alias: open box or clear box testing, structural testing [http://www.webopedia.com/TERM/W/White_Box_Testing.html, 2004-10-29] Component

35. Wilfried Mausz Type of Tests Tests in the software engineering process Unit tests –Regression tests Integration tests Function tests Performance tests Acceptance tests Installation tests

36. Wilfried Mausz Testing steps Unit test Integration test Function test Performance test Acceptance test Installation test Unit test Component code Design specification System functional requirements Other Software requirements Customer requirements specification User environment Integrated modules Functioning system Verified, validated software Accepted system Tested component System in use [Pfleeger, 2001]

37. Wilfried Mausz Unit tests Each program component is first tested on its own Isolated from other components Controlled environment Predetermined test data Back-box and white-box test

38. Wilfried Mausz Regression testing Testing a program that has been modified to verify that modifications have not caused unintended effects and still complies with its specified requirements. [http://www.bitpipe.com/tlist/Regression-Testing.html, 2004-10-30] Part of the test phase of software development where, as new modules are integrated into the system and the added functionality is tested, previously tested functionality is re-tested to assure that no new module has corrupted the system. [Bennatan, 1992] write part 1 test part 1 write part 2 test part 2 write part 3 test part 3

39. Wilfried Mausz Regression testing Regression testing ensures that a system still runs correct after modifications to it has been made Should be done after every change regardless if new functionality was added or faults in existing program where fixed Manually done  extremely time-consuming When possible done with automated tests

40. Wilfried Mausz System integration test Software and system integration Interface faults are not found by unit testing Determine if components will work properly together Determine failures distributing the components Identify defects that are not easily identified during unit testing [Firesmith, 2004] Integration testing is the testing of two or more integrated software components to produce failures caused by interface defects.

41. Wilfried Mausz Acceptance tests A formal test conducted to determine whether or not a system satisfies its acceptance criteria and to enable the customer to determine whether or not to accept the system. [Beck, 2000] Black box Tested by customer Test cases are derived from the requirements (XP: user stories) alias: functional tests

42. Wilfried Mausz Cleanroom software engineering Overview Advantages Disadvantages Example of formal proofs

43. Wilfried Mausz Cleanroom software engineering Development process for high quality software Focus on writing correct software instead of detecting errors IBM and US Army since mid-1980 “Cleanroom” comes from electronics industry  prevents introduction of defects during hardware production [http://www.sei.cmu.edu/str/descriptions/cleanroom_body.html, 2004-10-31]http://www.sei.cmu.edu/str/descriptions/cleanroom_body.html Courtesy Stanford Linear Accelerator Center

44. Wilfried Mausz Cleanroom software engineering software development as rigorous, engineering-based practices Combination of: –correct by mathematically sound design –verification by formal proof of correctness –certified by testing (black box, white box) Incremental development, each increment runs through the complete process

45. Wilfried Mausz Cleanroom software engineering Advantages: –Failures found at an early state –Correctness is mathematically proven –High process quality Disadvantages: –Software is mainly seen as a combination of mathematical functions –High effort on specification and verification –Testing begins at a late state in development –Training on the process itself needed –Hardware analogy does not fit for software [Binder, 1997]

46. Wilfried Mausz Formal proof Mathematically correct proof that a program is correct Example: Input: Output: Flowchart: Transformation: Proof: A 1  A 2, A 2  A 3, … A n  A end [Pfleeger, 2001]

47. Wilfried Mausz Test driven developement Definitions Development Cycle Advantages Disadvantages Example using NUnit

48. Wilfried Mausz Test driven developement Test-driven development (TDD) is a programming technique heavily emphasized in Extreme Programming. Essentially the technique involves writing your tests first then implementing the code to make them pass. The goal of TDD is to achieve rapid feedback and implements the "illustrate the main line" approach to constructing a program. [http://en.wikipedia.org/wiki/Test_driven_development, 2004-11-04]

49. Wilfried Mausz Test driven developement Development Cycle: Write testWrite codeRun test Refactoring next component

50. Wilfried Mausz Test driven developement Tests are automated using a test framework Tests are “requirements” for implementation (“Design by Contract”) –First the test case is written –The implementation is done to comply with the test case Automated tests reused for regression testing Tests are run instantly after compiling heavily emphasized in Extreme Programming (Kent Beck) [http://en.wikipedia.org/wiki/Test_driven_development, 2004-10-31]

51. Wilfried Mausz Test driven developement Advantages Implementation is reduced to just comply the tests  source code gets more simple Tests can be rerun without human intervention  No extra costs for one test more Tests are additional documentation to the project Programmer gets permanent feedback about his work

52. Wilfried Mausz Test driven developement Disadvantages Effort for defining useful test cases Limits auf automated tests –GUI –Database –Non-object oriented systems “Test-driven Development is only as good as the tests are.” [http://en.wikipedia.org/wiki/Test_driven_development, 2004-10-31]

53. Wilfried Mausz Example NUnit Live demo…

54. Wilfried Mausz Bug tracking Definitions Why? Programs Example using Bugzilla

55. Wilfried Mausz Bug tracking Definition bug tracking system: A system for receiving and filing bugs reported against a software project, and tracking those bugs until they are fixed. Most major software projects have their own BTS, the source code of which is often available for use by other projects. [http://www.hyperdictionary.com/computing/bug+tracking+system, 2004-10-31] Bugzilla main page

56. Wilfried Mausz Bug tracking During tests a lot of failures are detected Bug tracking systems help to: –Report bugs –Track them until they are fixed –Get reports about failures and history of program components Programs: –Bugzilla (www.mozilla.org) –GNATS (www.gnu.org/software/gnats/) –Debbugs (www.debian.org)

57. Wilfried Mausz Bugzilla Live demo … https://bugzilla.mozilla.org/show_bug.cgi?id=65318

58. Wilfried Mausz Conclusion Software is written by humans and humans make errors. There is no perfect way to avoid failures during development. Testing is the main part of software development, improvement this process will instantly improve product quality and costs. V-model shows the role of testing (one of the two sides of development process, corresponding stages of the process) Combination of code reviews, smart usage of automated tests and consequent use of a bug tracking system

59. Wilfried Mausz Literature Beck, K., “Simple Smalltalk testing with patterns” http://www.xprogramming.com/testfram.htm, 2004-11-04 http://www.xprogramming.com/testfram.htm Bennatan, E.M., "Software Project Management", 2nd edition, McGraw-Hill International, 1992 Binder, R., “Can a manufactoring quality model work for software?”, 1997 http://www.rbsc.com/docs/IEEE-SW-6S.pdf, 2004-10-31 http://www.rbsc.com/docs/IEEE-SW-6S.pdf Brooks, F.P., “The mythical man-month”, Anniversary edition, Addison-Wesley, 1995 Firesmith, D., System integration testing, 2004 http://www.donald- firesmith.com/index.html?Components/WorkUnits/Activities/Testing/SoftwareIntegrationTesting.html~Contents, 2004-10-31 http://www.donald- firesmith.com/index.html?Components/WorkUnits/Activities/Testing/SoftwareIntegrationTesting.html~Contents Pfleeger, S.L., “Software Engineering, Theory and Practice”, 2nd edition, Prentice Hall, 2001 http://en.wikipedia.org/wiki/Alpha_test, 2004-10-31http://en.wikipedia.org/wiki/Alpha_test http://en.wikipedia.org/wiki/Computer_bug, 2004-10-30http://en.wikipedia.org/wiki/Computer_bug http://en.wikipedia.org/wiki/Test_driven_development, 2004-10-31http://en.wikipedia.org/wiki/Test_driven_development http://www.bitpipe.com/tlist/Regression-Testing.html, 2004-10-30http://www.bitpipe.com/tlist/Regression-Testing.html http://www.extremeprogramming.org/rules/functionaltests.html, 2004-10-29http://www.extremeprogramming.org/rules/functionaltests.html http://www.hyperdictionary.com/computing/bug+tracking+system, 2004-10-31http://www.hyperdictionary.com/computing/bug+tracking+system http://www.junit.org, 2004-10-29http://www.junit.org http://www.netnam.vn/unescocourse/se/51.htm, 2004-10-30http://www.netnam.vn/unescocourse/se/51.htm http://www.sei.cmu.edu/str/descriptions/cleanroom_body.html, 2004-10-31http://www.sei.cmu.edu/str/descriptions/cleanroom_body.html http://www.webopedia.com/TERM/B/Black_Box_Testing.html, 2004-10-29http://www.webopedia.com/TERM/B/Black_Box_Testing.html http://www.webopedia.com/TERM/W/White_Box_Testing.html, 2004-10-29http://www.webopedia.com/TERM/W/White_Box_Testing.html

60. Wilfried Mausz Images First computer bug, http://en.wikipedia.org/wiki/Computer_bug, 2004-10-30http://en.wikipedia.org/wiki/Computer_bug Ariane 5 takeoff and explosion, http://www.doc.ic.ac.uk/~jrm198/program_analysis/verification2.html, 2004-11-05http://www.doc.ic.ac.uk/~jrm198/program_analysis/verification2.html Bugs found since installation, Brooks, F.P., “The mythical man-month”, Anniversary edition, Addison-Wesley, 1995 Courtesy Stanford Linear Accelerator Center, http://www.slac.stanford.edu/slac/media-info/SPEAR3/, 2004-10-31http://www.slac.stanford.edu/slac/media-info/SPEAR3/ Formal proof, Pfleeger, S.L., “Software Engineering, Theory and Practice”, 2nd edition, Prentice Hall, 2001 NUnit screenshot Bugzilla main page https://bugzilla.mozilla.org/show_bug.cgi?id=65318, 2004-10-31https://bugzilla.mozilla.org/show_bug.cgi?id=65318

61. Wilfried Mausz Discussion Testing is about 50% in software development? Can this be true? What is your experience? What can you do instantly to optimize the process of faults detection and removing? What is your experience with test driven development? Can you imagine using a test framework for your projects? Where are the limits? Do you think that code reviews are appropriate for your company? Do you have experience with bug tracking systems? Which ones are you using?