CS 5150 Software Engineering Lecture 22 Reliability 3

CS Administrivia Quiz 3 next Monday Today is the deadline for rescheduling! Design patterns Security/privacy Reliability

CS Final Milestone Client satisfaction In previous milestones my evaluation of client satisfaction has been informal This time I will ask them to send me an a few days after receiving the final deliverable, with answers to a couple simple questions about how well the project went

CS SE in the News Petraeus and privacy

CS Testing Testing Testing Testing comes in many flavors User testing Unit testing Integration testing Performance testing Stress testing Regression testing Acceptance testing Manual/automated testing Close/open-box (black/white-box) testing

CS Fundamental Facts About Testing Testing is indispensable for software verification, but can never prove that non-trivial software is defect-free Each test proves that the system works in one particular circumstance Most non-trivial software has infinitely many possible execution contexts (i.e., input, environment, etc) Every project should have a verification plan that includes testing strategies

CS Unit Testing Each unit test should verify that a single ‘unit’ of software works in isolation Can be challenging to write unit tests for code buried deep in a project Overlap with type systems Basic sanity checks Should run automatically e.g. overnight

CS Integration Testing Motivation: components developed by different people/teams; bugs can easily crop up at the boundaries Definition a little nebulous More useful to explicitly do other kinds of testing

CS Regression Testing Every time you find a bug in your project, make a test case that reproduces the bug The test case should be committed to your repository no later than the bug fix itself A good set of regression tests provide a ‘ratcheting’ mechanism Helps prevent the recurrence of bugs

CS Fuzz Testing Run your program many times with a random sequence of inputs Black-box vs white-box

CS Manual Testing Has a bad reputation in some computer science circles, but is absolutely essential Not all testing can be automated Write test scripts

CS Dealing with Bugs Any change in complex software has a non- trivial chance of introducing bugs Major reason for managerial conservatism Bug avoidance Bug tolerance Bug detection

CS Defensive Programming Write code to check assumptions (numerical value ranges, complex data structure invariants) Call it all over the place JPL rules about assertion frequency “Impossible” things can happen in libraries, compilers and even hardware Avoid risky programming techniques Many coding standards ban ‘goto’ entirely

CS Fault Tolerance Instead of bombing out when unexpected conditions occur, find some way to keep going Checkpoints Logs Fault tolerance mechanisms are extremely difficult to test (so it’s best to keep them simple)

CS Fixing Bugs Reproducibility Test case complexity Workaround vs “proper” fix

CS Determinism => Reproducibility It is very hard to fix bugs that cannot be reproduced reliably Do not introduce unnecessary non- determinism into your programs Random numbers Address-based data structures Multithreading Time limits vs effort limits Environment dependencies

CS Safety-Critical Systems Much more emphasis on validation and verification Beyond the scope of 5150