1. Software Engineering Modern Approaches Eric Braude and Michael Bernstein 1

2. © 2010 John Wiley & Sons Ltd. Chapter 28: Testing at the System Level 2

3. Learning Goals of This Chapter How do functional and non-functional testing differ? How do you conduct performance tests? What is load / stress event testing? Reliability and Availability testing? Recoverability testing? How do you test usability? How do you validate security? How do you test when requirements are sparse or even non-existent? What is acceptance testing? What is testing like for agile processes? What are compatibility, installation, and serviceability testing? How do alpha- and beta- releases relate to testing? What is automated testing? Requirements analysis Design Implementation Testing Maintenance Planning The Software Development Lifecycle 3

4. © 2010 John Wiley & Sons Ltd. Performance … Test speed of application Load / Stress … Test against heavy event traffic Reliability and Availability … Determine percentage up-time Recoverability … Test ease with which applications recovers from crash Non-Functional Tests 4

5. © 2010 John Wiley & Sons Ltd. Usability … Determine degree of user satisfaction. Security … Determine susceptibility to intended and unintended breaches Compatibility … Application compatible with other systems as required Install-ability … Test ease of installation Serviceability … Test ease of keeping up-to-date at customer site Non-Functional Tests (cont.) 5

6. © 2010 John Wiley & Sons Ltd. System Testing Encounter : Engage Foreign Character Use Case Test Step 1 Steps 2, 3, and 4 Step 5 6

7. © 2010 John Wiley & Sons Ltd. System Testing Encounter 7

8. © 2010 John Wiley & Sons Ltd. System Testing Encounter 2 8

9. © 2010 John Wiley & Sons Ltd. Accessibility How easily can users enter, navigate & exit? e.g., measure by average time taken to... Responsiveness How ready is the application to allow the user to accomplish specified goals? How often is GUI display accurate (percentage)? How quickly are user actions acknowledged? How often is the application ready for user action? Efficiency Degree to which the number of required steps for selected functionality is minimal “minimal” calculated theoretically e.g., measure by minimal time / average time Comprehensibility How easy is the product to understand and use with documentation and help? e.g., measure time taken for standard queries Key Attributes for Usability Testing* * Partly adapted from Kit [Ki]. 9

10. © 2010 John Wiley & Sons Ltd. Example of Usability Scores QualityScoreVariance Industry Average Variance Accessibility8.12.1 8.23.5 Responsiveness9.33.2 5.03.0 Efficiency5.62.0 7.21.1 Comprehensibility2.30.5 6.00.3 10

11. © 2010 John Wiley & Sons Ltd. Usability Testing 1 11

12. Usability Testing 2 © 2010 John Wiley & Sons Ltd. 12

13. Usability Testing 3 © 2010 John Wiley & Sons Ltd. 13

14. © 2010 John Wiley & Sons Ltd. Testing for Security Confidentiality oSniffers validate that data passed not visible to unauthorized parties oHire professional hackers? Nonrepudiation oExperiments to validate Integrity oAlter data and validate consequences Authentication oHire professional hackers? Authorization oHire professional hackers? 14

15. © 2010 John Wiley & Sons Ltd. Why Test Without Requirements? Provides a “feel” for the application Even complete requirements do not do this well The Requirements document omits implicit ones The Requirements may -- -- be incomplete -- be inconsistent with the code -- be missing -- never have been written down at all You can’t access the requirements because the product is from a third party Vendor software that you must test before using 15

16. Directed Graphs node edge © 2010 John Wiley & Sons Ltd. 16

17. Initial Directed Graph for OpenOffice 1 2 Actions Launch text document processing. Enter text Hit file / save Enter a file name and press save 34 An action. Edge indicates sequence (only one in this case). © 2010 John Wiley & Sons Ltd. 17

18. Directed Graph for OpenOffice 1 2 Actions 1.Launch text document processing. 2.Enter text 3.Hit file / save as 4.Enter a file name and press save 5.Convert 3 line to headings at levels 1, 2, and 3 6.Bold some text 7.File exists? 8.Hit save / “already exists” pops up / yes 3 4 5 save headings 6 boldsave headings 8 7 [Predicate node] T F © 2010 John Wiley & Sons Ltd. 18

19. Using the Directed Graph for OpenOffice for Testing 1 2 Actions 1.Launch text document processing. 2.Enter text 3.Hit file / save as 4.Enter a file name and press save 5.Convert 3 line to headings at levels 1, 2, and 3 6.Bold some text 7.File exists? 8.Hit save / “already exists” pops up / yes 3 4 5 save headings 6 bold headings 8 7 [Predicate node] T F save Paths: © 2010 John Wiley & Sons Ltd. 19

20. © 2010 John Wiley & Sons Ltd. Order of Testing With Directed Graphs The most expected paths with no error or anomaly conditions The most expected paths with error or anomaly conditions The paths most likely to yield defects 20

21. © 2010 John Wiley & Sons Ltd. Alpha- and Beta- Releases In-house and highly trusted users  Multiplies testing  Previews customer reaction  Benefits third-party developers  Forestalls competition Selected customers  Multiplies testing activity  Obtains customer reaction Alpha Beta 21

22. © 2010 John Wiley & Sons Ltd. Test TypeApproachCorresponding document sections Unit White- and black box; method and class tests; test against D- requirements and design. SRS section(s): 3.2 Classes/Objects SDD section(s): 6. Detailed design Integration Gray box; mostly package-level; oriented to builds (1, 2, and 3); test against architecture and C- requirements. SRS section(s): 2. Overall description, 3.1 External interfaces, validate representative requirements in 3.2 Classes/Objects SDD section(s): 3. Decomposition description, 4. Dependency description, 5. Interface description. System Black box; all packages; whole system (Build 3); test against non- functional requirements, architecture and C-requirements. SRS section(s): 2. Overall description, 3.1 External interfaces, validate representative requirements in 3.2 Classes/Objects, 3.3 Performance requirements, 3.4 Design constraints, 3.5 Software system attributes, 3.6 Other requirements SDD section(s): 3. Decomposition description, 4. Dependency description, 5. Interface description ; validate representative requirements in 6. Detailed design. Approaches and Documentation for Test Types 22

23. © 2010 John Wiley & Sons Ltd. 23

24. © 2010 John Wiley & Sons Ltd. DocumentSectionRequirement Title SRS2.1.2.2User interface for setting quality values 3.1.1.2User interface for setting quality values 3.2.ECEncounter characters 3.2.FCForeign characters 3.2.PPlayer characters 3.2.PQThe player quality window SDD for RPG framework 3.1.2Characters package 5.0Interface description SDD for Encounter 3.1.2EncounterCharacters package 4.2Inter-process dependencies 5.1.2 Interface to the EncounterCharacters package Features to be Tested in Build 1 24

25. © 2010 John Wiley & Sons Ltd. Quality Player input value Foreign input value OtherAction Intera- tion Test # B1.1 N/A Get player character Verify by name B1.2 N/A Get foreign character Verify by name B1.3 Concentra- tion 3040N/A Verify output values == input values B1.4 Stamina3040N/A Verify output values == input values B1.5..... Integration Test Inputs, Outputs, and Actions 25

26. © 2010 John Wiley & Sons Ltd. 26

27. Architecture and Modularization of Encounter Role-playing Game EncounterCharacters EncounterGame EncounterCast «facade» EncounterGame «facade» EncounterEnvironment «facade» © 2010 John Wiley & Sons Ltd. 27

28. 4. select exit for character 3a. enter quality value 1b. «create» 2. «create» 3b. setQuality() :Encounter- Game :Player quality window main player character: Player Character dressing room: Area 1a*. «create» 5. move() Sequence Diagram for Initialize Use Case * Numbering keyed to use case User © 2010 John Wiley & Sons Ltd. 28

29. 3.1 «create» engagement: Engagement 2.1 setPlayerQuality() 1.3 «create» 3.2 displayResult() :Player’s main character :Encounter game :Engagement Display freddie: Foreign Character 1.2 display() :Encounter Cast 1.1 displayForeignChar() 2.2 setQuality() 2.3 setForeignQuality() 2.4 setQuality() 2. execute() Sequence Diagram for Encounter Foreign Character Use Case © 2010 John Wiley & Sons Ltd. 29

30. A Comparison of Method Invocations in Eclipse http://www.eclipse.org/tptp/platform/documents/probekit/probekit.html © 2010 John Wiley & Sons Ltd. 30

31. Graphical Comparison of Method Execution Time in Eclipse http://www.eclipse.org/tptp/platform/documents/probekit/probekit.html © 2010 John Wiley & Sons Ltd. 31