Outline Types of errors Component Testing Testing Strategy Unit testing Integration testing Testing Strategy Design Patterns & Testing System testing Function testing Structure Testing Performance testing

How do we deal with Errors & Faults?

Verification?

Modular Redundancy?

Declaring the Bug a Feature?

Patching?

Testing?

Examples of Faults Interface specification Faults Algorithmic Faults Expectation mismatch between the client & the server Cannot happen with Jini Mismatch between requirements & implementation Algorithmic Faults Mechanical Faults Documentation does not match actual conditions Timing errors Throughput / performance errors

Some Observations It is impossible to completely test any nontrivial module Theoretical limitations: Halting problem Practical limitations: Prohibitive in time & cost Testing only shows the presence of bugs, not their absence

Testing Requires Creativity Is testing dirty work? An effective test requires: Detailed understanding of the system Knowledge of test techniques Skill to apply these techniques in an effective & efficient manner Book says: Testing is done best by independent testers XP says: developers devise/do unit tests

Testing Activities Unit T est Unit T est Integration Functional Test Requirements Analysis Document Code Unit Requirements Analysis Document System Design Document T est Tested Subsystem User Manual Code Unit T est Tested Subsystem Integration Functional Test Test Functioning System Integrated Subsystems Tested Subsystem Code Unit T est All tests by developer

Testing Activities … Performance Acceptance Installation Test Test Client’s Understanding of Requirements Global Requirements User Environment Validated System Accepted System Functioning System Performance Acceptance Installation Test Test Test Tests by developer Tests by client

Fault Handling Techniques Fault Avoidance Fault Detection Fault Tolerance Design Methodology Reviews Atomic Transactions Modular Redundancy Verification Configuration Management Testing Component Testing Integration Testing System Testing Correctness Debugging Performance Debugging

Component Testing Unit Testing Integration Testing Developers confirm that class functions correctly. Integration Testing Developers confirm that subsystem interface functions correctly

System Testing System Testing Acceptance Testing Developers confirm that the system meets its requirements Acceptance Testing Clients confirm that the system meets its requirements

Unit Testing Informal Static Analysis Testing Incremental coding Hand execution: Read the source code Walk-Through: Informal presentation to others Code Inspection: Formal presentation to others Testing Black-box testing: Test the input/output behavior White-box testing: Test the internal logic of the class

Black-box Testing I/O behavior If for any input, we can predict the output, the module passes the test. Generally intractable to test all possible inputs Reduce number of tests by equivalence partitioning Partition the inputs into equivalence classes Devise a test case for each equivalence class. Example: If an object is supposed to accept a negative number, include a test of 1 negative number.

Black-box Testing (Continued) Selection of equivalence classes No rules, only guidelines Input is valid across range of values. Select 4 test cases from 3 equivalence classes: Just below the lower boundary Just above the lower boundary Just below the upper boundary Just above the upper boundary Input is valid if it is from a discrete set. Select test cases from 2 equivalence classes: Valid discrete value Invalid discrete value

White-box Testing Thoroughness (Coverage) 4 types of white-box testing Every statement in the component is executed at least once. 4 types of white-box testing Statement Testing Loop Testing Path Testing Branch Testing

White-box Testing … Statement Testing Loop Testing Test single statements Loop Testing Execution of the loop is skipped. Exception: Do-While loops Loop is executed exactly once Loop is executed more than once Path testing (implies conditional testing) Make sure all paths in the program are executed

White-box Testing … public int sumPositives( int[] a ) { int total = 0; for ( int i = 0; i < a.length; i++ ) if ( a[i] > 0 ) total += a[i]; return total; } t1 = sumPositives( null ); t2 = sumPositives( { 0 } ); t3 = sumPositives( { 1 } ); t5 = sumPositives( { 1, 2, 3 } ); t6 = sumPositives( { 2, 0, 0 } ); // ?

Comparison of White & Black-box Testing Black-box tests are devised when: Functional requirement is defined Class is defined White-box tests are devised when class is implemented. Create a test harness: A program that executes the tests. First examination of test results is by eye. Test Oracle: Save output of tests in a file. Automate regression testing.

Bottom-up Testing Strategy Subsystem classes in the lowest layer of the call hierarchy are tested individually The next subsystems are tested that call the previously tested subsystems Do until all subsystems are included in the testing Test driver A routine that passes a test case to a subsystem.

Bottom-up Integration C D G F E Layer I Layer II Layer III Test E Test B, E, F Test F Test A, B, C, D, E, F, G Test C Test D,G Test G

Bottom-Up Integration Testing Useful for integrating: Object-oriented systems Systems with strict performance requirements Real-time systems

Top-down Testing Strategy Test the top layer subsystem first Then test that layer + all subsystems called by that layer. Do until all subsystems are incorporated into the test A test stub is needed A program or a method that simulates a missing method or subsystem by returning fake data.

Top-down Integration Testing B C D G F E Layer I Layer II Layer III Test A, B, C, D, E, F, G Test A Test A, B, C, D Layer I Layer I + II All Layers

Sandwich Testing Strategy Combines top-down strategy with bottom-up strategy The system is viewed as having 3 layers A target layer in the middle A layer above the target A layer below the target The target layer is tested.

1. Unit test all the classes in the selected component. 2. Do functional testing: Define test cases that exercise all uses cases with the selected component 3. Execute performance tests Repeat steps 1 to 3 until the full system is tested. Keep records of the test cases.

System Testing Functional Testing Structure Testing Performance Testing Acceptance Testing Installation Testing Impact of requirements on system testing: The more explicit the requirements, the easier they are to test. Quality of use cases determines the ease of functional testing Quality of subsystem decomposition determines the ease of structure testing Quality of nonfunctional requirements & constraints determines the ease of performance tests

Structure Testing Essentially the same as white box testing. Cover all paths in the system design Exercise all input / output parameters of each component. Exercise all components & methods Use conditional & iteration testing as in unit testing.

Functional Testing Essentially the same as black box testing . Test functionality of system Test cases are designed from the RAD, center around requirements & use cases The system is treated as black box. . .

Performance Testing Timing testing Stress Testing Environmental test Stress limits of system (maximum # of users, data, peak demands, extended operation) Configuration testing Test the various software and hardware configurations Security testing Try to violate security requirements Timing testing Evaluate response times and time to perform a function Environmental test Test tolerances for heat, humidity, motion, portability Recovery testing Tests system’s response to presence of errors or loss of data.

Testing has its own Life Cycle (?) Establish the test objectives Design the test cases Write the test cases Test the test cases Execute the tests Evaluate the test results Change the system Do regression testing

Test Team Test Team Professional Tester Analyst System Designer User too familiar Programmer with code Analyst System Designer Test User Team Configuration Management Specialist

Summary Testing is still a black art Testing consists of: component-testing (unit testing, integration testing) system testing Testing has its own lifecycle (?)