Jason I. Hong CS169 - Software Engineering, Fall 1998 Testing Strategies

Types of Testing

Scope –Unit Testing tests a subroutine, module, or class –Integration Testing tests multiple units together –System Testing tests entire system as a whole Some Testing Goals –Functional Testing tests functions in system –Performance Testing tests system performance primarily execution time, memory footprint –Reliability Testing certifies system reliability predictable execution time, low number of failures, etc.

Black Box Testing Define test cases based only on specifications Try all boundary conditions –Go from empty to full, full to empty –Array bounds –Empty or null values Should be done by someone other than the module programmer –Needs a shift in mindset away from programming –Programmers don’t want to find bugs, testers do

White Box Testing Define test cases by looking at code Not as useful as code review or code coverage Useful for regression testing

Code Coverage Create test cases that execute every block of code at least once Complete code coverage does not imply correct code Ideally, we would like to test every path but this is exponential –e.g. all possibilities in switch statements Useful for regression testing

Code Review Read through someone else’s code –Extremely effective at finding defects Look for common errors –Possible unhandled error conditions –Potential array out of bounds (e.g. gets) –Bad or dangerous type casting –Memory leaks (e.g. alloc w/o dealloc) –Too many global variables (high complexity) –Not cleaning up after self (e.g. goto, return) –...

Regression Testing Maintain suite of tests Build product periodically and run test suite –Automatically verify that output is correct –Find reason for failure (test wrong or code wrong) Good metric for quality –Can always tell if going forward or not –Requires significant process maturity from company Continually add new tests as bugs are fixed –Ensures that old bugs do not reappear –Forces discipline on developers

Stress Testing Place large load on product (typically a server) Each client sends multiple requests –Some valid requests, some junk requests Look for: –Increasing memory footprint (memory leak?) –Running out of resources (sockets, file descriptor) –Increased response time (slows over time?) –Incorrect results (possible race conditions?)

User Testing Perhaps the most important kind of testing –Can customers use the product easily? –How many errors do users make? –How long does it take users to learn the system? –Do customers like the product? Enjoy using it? –Without customers, you have no product. Best strategy is iterative development –Expensive to change product at end, so build cheap prototypes and evaluate with users at beginning –Hire real designers! Programmers != designers –Take CS160 for more informationCS160

Alpha and Beta Testing Alpha testing –Product is released to very small and focused group Beta testing –Product is released to wider range of people Difficult to make significant changes once Alpha and Beta Testing is reached –Use for polishing, finding compatibility problems, fixing simple errors, and finding workarounds –Do not use for evaluating usability or quality, a common mistake still made by many companies

Some Statistics Unit testing finds 10-50% of defects System testing finds 20-60% of defects Combined rate < 60% of defects Informal code review finds 30-70% of defects Formal inspection finds 60-90% of defects, net savings of 10-30% Reviews are more cost effective than testing –Using reviews with testing helps software quality See Rapid Development pp72-75

Development Notes Back-to-Back algorithms –Use two implementations and compare results –Microsoft Excel does this for cell calculations –UWash Kimera Verifier found bugs in Netscape, Sun, and Microsoft implementations of JavaKimera Use Static Verification –Use tools that go through code statically –Have no errors or warnings on Lint and gcc -Wall Self-Testing Code –Have each class be able to test itself –Each Java class can have its own main() method

Development Notes (cont.) Use good tools –Purify, test harnesses, code coverage analysis, etc Iterative Development –Plan a little, build a little, verify a little, and repeat –Fix defects as you find them, not “at the end” –If not, others will waste time on known problem 80/20 rule applies for defects –80% of bugs likely to be in 20% of code –Use defect tracking to find this code –If a unit seems to have lots of errors, it may be easier just to rewrite from scratch!

Testing Process Notes Begin test planning from beginning –Start test cases once requirements are known –Use User Manual and Specification Document –Microsoft uses one tester per developer –Space Shuttle people use ten testers per developer Daily Build –Product is built every day and put through tests –Supports incremental development –Minimizes integration (prevents dis-integration) –Provides project visibility

Testing Process Notes (cont.) Find the reason for error and ask questions –Is it a common occurrence? Mistake? Sloppiness? –Programmers fix own bugs at Microsoft, prevents programmer from introducing new bugs elsewhere –How much will it cost to fix? Is it worth it? –How can bugs like this can be prevented in future? Keep a database of bugs –Track all defects to closure –Graph open bugs and closed bugs over time

Testing Process Notes (cont.) Remember that testing only shows the presence of defects, not absence –Testing assesses quality, does not assure quality Have an environment of quality –Inktomi hangs a toy pig over person’s cubicle for a short while if he/she messes up –Microsoft head once threw chair out window on broken build –Lives depend on Space Shuttle software developers –Lives may depend on software you write!

Summary Plan quality from the very beginning Plan testing from the very beginning –People and resources needed, test cases Preventing bugs is easier than finding bugs –Use assertions, preconditions, invariants, reviews Use a variety of methods for reducing defects, assessing quality, and assuring quality The earlier you find a defect, the cheaper and easier it is to correct

References Rapid Development by Steve McConnell Code Complete by Steve McConnell Software Engineering by Ian Sommerville Software Project Dynamics by Jim McCarthy Software Management by Donald Reifer Construx Software Eric Brewer (CS169 Spring 1998)