1. CS 5150 Software Engineering Lecture 22 Reliability 3

2. CS 5150 2 Administrivia Quiz 3 next Monday Today is the deadline for rescheduling! Design patterns Security/privacy Reliability

3. CS 5150 3 Final Milestone Client satisfaction In previous milestones my evaluation of client satisfaction has been informal This time I will ask them to send me an email a few days after receiving the final deliverable, with answers to a couple simple questions about how well the project went

4. CS 5150 4 SE in the News Petraeus and privacy

5. CS 5150 5 Testing Testing Testing Testing comes in many flavors User testing Unit testing Integration testing Performance testing Stress testing Regression testing Acceptance testing Manual/automated testing Close/open-box (black/white-box) testing

6. CS 5150 6 Fundamental Facts About Testing Testing is indispensable for software verification, but can never prove that non-trivial software is defect-free Each test proves that the system works in one particular circumstance Most non-trivial software has infinitely many possible execution contexts (i.e., input, environment, etc) Every project should have a verification plan that includes testing strategies

7. CS 5150 7 Unit Testing Each unit test should verify that a single ‘unit’ of software works in isolation Can be challenging to write unit tests for code buried deep in a project Overlap with type systems Basic sanity checks Should run automatically e.g. overnight

8. CS 5150 8 Integration Testing Motivation: components developed by different people/teams; bugs can easily crop up at the boundaries Definition a little nebulous More useful to explicitly do other kinds of testing

9. CS 5150 9 Regression Testing Every time you find a bug in your project, make a test case that reproduces the bug The test case should be committed to your repository no later than the bug fix itself A good set of regression tests provide a ‘ratcheting’ mechanism Helps prevent the recurrence of bugs

10. CS 5150 10 Fuzz Testing Run your program many times with a random sequence of inputs Black-box vs white-box

11. CS 5150 11 Manual Testing Has a bad reputation in some computer science circles, but is absolutely essential Not all testing can be automated Write test scripts

12. CS 5150 12 Dealing with Bugs Any change in complex software has a non- trivial chance of introducing bugs Major reason for managerial conservatism Bug avoidance Bug tolerance Bug detection

13. CS 5150 13 Defensive Programming Write code to check assumptions (numerical value ranges, complex data structure invariants) Call it all over the place JPL rules about assertion frequency “Impossible” things can happen in libraries, compilers and even hardware Avoid risky programming techniques Many coding standards ban ‘goto’ entirely

14. CS 5150 14 Fault Tolerance Instead of bombing out when unexpected conditions occur, find some way to keep going Checkpoints Logs Fault tolerance mechanisms are extremely difficult to test (so it’s best to keep them simple)

15. CS 5150 15 Fixing Bugs Reproducibility Test case complexity Workaround vs “proper” fix

16. CS 5150 16 Determinism => Reproducibility It is very hard to fix bugs that cannot be reproduced reliably Do not introduce unnecessary non- determinism into your programs Random numbers Address-based data structures Multithreading Time limits vs effort limits Environment dependencies

17. CS 5150 17 Safety-Critical Systems Much more emphasis on validation and verification Beyond the scope of 5150