CS 5150 Software Engineering Lecture 21 Reliability 2

CS Administrivia Presentation Feedback Generally a big improvement Transitions Demos Code Jargon Conclusion Quiz 3 Design patterns Security/privacy Reliability

CS Final Milestone Presentation Demo Report Cumulative

CS SE in the News

CS Reliability Analysis - Consequences

CS Iron Law of Software Reliability For most non-trivial applications, 100% reliability is effectively impossible

CS Verification and Validation Does the software do what it is intended to do? Validation Does the design match the intended behavior? Verification Does the implementation match the design? Informal and formal

CS Verification Does the implementation exhibit any particular bad behaviors? Can we guarantee that the implementation does what it is designed to do? “How did software get so good without proof?” Dijkstra

CS Static and Dynamic Static Examination of source code without actually running it Can be manual or tool-assisted Dynamic Testing Debugging Profiling

CS Informal Static Verification Code reviews Can be light or heavy Colleague reads over patch before committing to repository Meetings with multiple participants Benefits: “given enough eyeballs, all bugs are shallow” Opportunity to share knowledge Opportunity to do housekeeping Exposes inconsistent assumptions between developers

CS Getting the Most from Code Reviews Senior team members must show leadership heavy-weight reviews require preparation The tone must remain constructive Code reviews are not mini personnel reviews

CS Review Checklists Data faults: Initialization, constants, array bounds, character strings Control faults: Conditions, loop termination, compound statements, case statements Input/output faults: All inputs used; all outputs assigned a value Interface faults: Parameter numbers, types, and order; structures and shared memory Storage management faults: Modification of links, allocation and de-allocation of memory Exceptions: Possible errors, error handlers

CS Working in Pairs Idea: achieve benefits of continuous review by doing all work in pairs Hard for management to swallow because it looks like doubling the personnel cost of development Some evidence that it works well Generally highest value with a senior/junior pair Junior person should ‘drive’

CS Static Analysis Tools Scan source code for possible problems Compiler warnings on steroids Superficial syntactic warnings Deeper analysis based on symbolic execution void foo( int *p ) { printf( “%i”, *p ); }

CS Scaling of Static Analysis In general, static analyses cannot be both complete and sound Complete - no false positives Sound - no false negatives Static analysis tools have become more common in the last 10 years Careful engineering trade-offs between false positives and false negatives

CS Heavy-Weight Formal Verification Hoare logic Pre-conditions/post-conditions Invariants Model checking Assertions

CS Testing Testing Testing Testing comes in many flavors User testing Unit testing Integrations testing Performance testing Stress testing Regression testing Acceptance testing Manual/automated testing Close/open-box (black/white-box) testing

CS Fundamental Facts About Testing Testing is indispensable for software verification, but can never prove that non-trivial software is defect-free Each test proves that the system works in one particular circumstance Most non-trivial software has infinitely many possible execution contexts (i.e., input, environment, etc) Every project should have a verification plan that includes testing strategies

CS Unit Testing Each unit test should verify that a single ‘unit’ of software works in isolation Can be challenging to write unit tests for code buried deep in a project Overlap with type systems Basic sanity checks Should run automatically e.g. overnight

CS Regression Testing Every time you find a bug in your project, make a test case that reproduces the bug The test case should be committed to your repository no later than the bug fix itself A good set of regression tests provide a ‘ratcheting’ mechanism Helps prevent the recurrence of bugs

CS Determinism => Reproducibility It is very hard to fix bugs that cannot be reproduced reliably Do not introduce unnecessary non- determinism into your programs Random numbers Address-based data structures Multithreading