1. Software Testing

2. Definition To test a program is to try to make it fail

3. Testing In testing we’re searching for –Errors Discrepancy between actual and predicted results Unexpected inputs resulting in unexpected behavior –Faults Causes software to malfunction Related to errors –Failures Doesn’t do what it’s supposed to do Caused by faults/errors

4. Test Plan For successful testing, you must have a plan – otherwise your will miss things –Specifies how we will demonstrate that the software is free of errors, faults, and failures –A series of tests each relating back to a portion of the original specification –Includes a Test Oracle (benchmark) Specifies what the results should be

5. Test Cases Inputs that relate to specific functionality Each case has an associated oracle An exhaustive set would be nice but not always feasible White Box approach to test case development –Study code to determine tests –Also known as Structure tests Black Box approach to test case development –Study specified behaviors to determine tests –Also know as Functional tests

6. Test Cases Stress tests –Test the system under various external loading conditions –Test the system under various capacity conditions (file sizes, number of users, …) Performance tests –Execution time –Memory usage Regression tests to verify system upgrades

7. Test Cases Unit tests –Test individual pieces (classes, methods ) Integrated tests –Test assemblies (collaborations)

8. Test Cases And don’t forget the platform! –Various operating systems Windows 98 Windows 2000 Windows ME Windows XP They’re all different! –Various microprocessors –Various form factors

9. Testing Bottom line(s) –Know what the system is supposed to do –Know how it does what it is supposed to do –Develop test cases to get from beginning to end via all possible processing paths –Make sure actual results match predicted results EXACTLY!

10. Tests versus specs Tests are no substitute for specifications –Using the test plan as a product specification is popular among agile methods –But, doing so may cause you to miss extreme cases Yes, that thing that happens with probability 1 in a million does happen…consider a real-time, high throughput system –Use product specifications to create test cases

11. Regression testing Any failed execution must yield a test case, to remain a permanent part of the project’s test suite –Just because you found an error and fixed it, doesn’t mean you don’t have to retest it –When you make changes to the system, you should run all tests to make sure the fix didn’t break something else

12. Applying oracles Determining success or failure of tests must be an automatic process –If you automate your testing, you are more likely to perform the tests –If you automate the testing, the system can flag errors for you rather than you having to read test results –Consider a testing framework such as JUnit

13. Manual and automatic test cases An effective testing process must include both manually and automatically produced test cases –Manually generated tests are based on the knowledge of the developer, which is typically “deep” –Automatically generated tests are based on a syntactical look at the system, which is typically “broad” –Tools for automatically generating tests are being researched (NIST, Microsoft, et al.)

14. Empirical assessment of testing strategies Evaluate any testing strategy, however attractive in principle, through objective assessment using explicit criteria in a reproducible testing process –Test the simple and the complicated things –Don’t overlook tests because this code “obviously” works

15. Assessment criteria A testing strategy’s most important property is the number of faults it uncovers as a function of time –This is a matter of “when do I stop testing?” –You’ll likely see the number of failures reduce over development time –Ideally you stop when there are no failures but time may force a delivery with known failures

16. Source “Seven Principles of Software Testing” –Bertrand Meyer –IEEE Computer, August 2008

17. Project Deliverables Sequence diagram(s) User interface design (mock-up) State machine diagram for the user interface Revised specification Test plan