1. CSCI 3428: Software Engineering Tami Meredith Chapter 9 Testing the System

2. TESTING The LAST LINE OF DEFENSE! Relying on testing is dangerous and leads to low-quality products # of Faults  # LOC Faults are often clustered: Where there is one, there is usually another Fixing faults frequently introduces new faults, or reveals previously undetectable faults Faults frequently NOT fixed because the actual problem has not been identified

3. System Testing Functional Tests Non-Functional Tests (Performance) Acceptance & Installation Tests SYSTEM IN USE! (Live)

4. Configuration Management Versions and Releases Release 1.0: John Madden Football (1988) Versions: PC, C-64, Apple 2 Release 5: John Madden Football ’93 (1992) Versions: PC, SNES, Genesis Release 13: Madden NFL 2000 (1999) Version: PC, PS, N64, GameBoy Colour, Mac Release 27: Madden NFL’13 (2012) Versions: XBox360, PS3, Wii (U), PS Vita, iOS, Android MS-DOS, C64/128, Apple II, Genesis, SNES, Amiga, Windows, GameBoy (+ colour, Advance), Game Gear (TV), PS, Saturn, PS2, N64, Mac, NGC, Xbox, Nindendo DS, PSP, Win. Mobile, Xbox 360, PS3, Tapwave Zodiac, Wii (+ U), OS-X, iOS, Android, 3DS

5. Regression Testing Tests applied to a new version/release Verifies it functions in the same manner as the previous version/release Should be used in iterative/incremental development Often automated and done by build team Daily builds advocated by agile methods, done by some companies (e.g., Microsoft)

6. Function(al) Testing Does the software do what its supposed? Has the SRS been implemented? Generally black-box (as opposed to white-box for unit/integration testing) Try to minimise the number of tests while still testing the software fully Optimal (complete) testing is unfeasible! Too many paths Too many different input combinations (including bad data) Could be destructive Cause and Effect Graphs … Yeah, whatever …

7. Non-Functional Testing (Performance) Stress – max users, requests, devices Volume – max data Configuration – all hardware combinations Compatibility – with other systems Regression – with system being replaced Security – is it safe Timing – performance, efficiency, response speed Environmental – use site (heat, humidity, vibration) Quality – MTF, MTBF Recovery – equipment failure, data loss Maintenance – diagnostic tools and procedures work Documentation – sufficient/correct documentation exists Human factors – “usability”

8. Failure Classification 1. Catastrophic – loss of system or life 2. Critical – severe injury or damage & mission failure 3. Marginal – injury or damage that delays or degrades mission 4. Minor – no mission impact but requires maintenance or repair Can you tell this specific classification is MIL-STD-1629A (US)? Others exist using different criteria.

9. Measurement Software Metrics MTBF = MTTF + MTTR Reliability 0..1, R = MTTF / (1 + MTTF) Availability 0..1, A = MTBF / (1 + MTBF) Maintainability 0..1, M = 1 / (1 + MTTR)

10. Acceptance Testing Convince the customer it works Generally involves the requirements team Benchmark Test: Typical Cases representative of use Pilot Test: Use on an experimental basis Alpha Testing: Testing of installed software by development organisation (e.g., programmers or professional testers) Beta Testing: Testing of installed software by client organisation (e.g., users) Parallel Testing: New system run in parallel with existing system and results compared

11. Test Plans Specialised for purpose (integration, security, etc.) Rationale, purpose, overview of testing Methods used to perform each test Identifies tools, stubs, drivers, and other support needed Detailed list of test cases and outputs for those cases Descriptions of how to evaluate outputs Notes on how to record, track, describe faults

12. Test Plans II Objectives Document References System Summary Major Tests Identify testing approaches Broken down as required Correspondences to SRS indicated Test Specifications: input/source, output/recording, limitations, requirements, ordering of sub-tests, procedures Schedule Materials Needed

13. Fault Reports A. Admin – tester, test #, site, equipment, date B. Location – where did it occur C. Timing – any details possible, sequencing D. Symptom – how you knew it was a fault E. End-result – failure caused, consequences F. Mechanism – how to make it happen again G. Cause – type of error that caused it H. Severity – with respect to performance and application domain I. Cost – to fix and its impact on application domain Not all of these can be clearly answered Components often overlap REPRODUCIBILITY is the key!

14. Fault (Bug) Trackers Also known as issue trackers Large number exist Generally an interface to a DB system Wide variety of features May be integrated with other systems (RCS, IDE) http://en.wikipedia.org/wiki/Comparison_of_issue-tracking_systems

15. Fault Flow

16. Thoughts Testing cannot remove all faults Safety, reliability, and security are NOT the same thing Assume users will make every mistake possible (and then some) Review, review, and then review again Short-term costs should not overshadow long-term risks and costs

17. Famous Moments in Software Engineering Marvy Medical Stuff 1986 to 1996: 450 reports to USFDA regarding software failure 24 Incidents lead to death or injury IV pump rand dry and injected air into patient Monitor failed to sound when patients stopped breathing Respirator gave extra breaths to patient Digital display put data of patient X beside name of patient Y Software failures responsible for 24% of device recalls (FDA 2011) Causes of failure (Abouzahra 2011) for IT Healthcare projects 43% Incomplete scope 30% Unidentified risks 14% Unidentified stakeholders (and interfaces) 8% Communication (developer  customer) 5% Other 66% of device failures are triggered by a single variable value