1. Test Plan: Introduction o Primary focus: developer testing –Implementation phase –Release testing –Maintenance and enhancement o Secondary focus: formal system verification –Addressed within current test plan via examples –Assumed to be primarily independent

2. Tests to be Performed o Bottom-up integration testing –Build a module, build a test to simulate how it is used o Black-box –Based on specification and ‘educated guess’ stress tests o White-box –Based on code inspection o Platform testing –Establish baseline capability of hardware/OS, detect differences o Performance –Per module, and peer to peer distributed costs o Test coverage –Coverage level dependent on resources, time, and cost of failure

3. Allocation of Testing Responsibilities o Assumption: development team performs –Internal (module-level) performance –Sample system performance (limited example federation) –Full white-box –Limited black-box (basic system functionality) o Assumption: external group performs –Independent, detailed system verification (Spec compliant) – Standardized performance tests o Test coverage –Coverage will be measured, analyzed for relative effectiveness –Coverage levels TBD

4. Testing Philosophy o Testing is not just pre-release! Continual process within development phase o Catch defects as early as possible o Make sure defects stay fixed o Track cause of defects: repair problem, do not keep re- patching the tire o Need support at both design level and implementation level to accomplish these goals

5. Continual Testing Process o Tests created during development o Central code repository: modules, test suites tied together –Tests are treated as live code to be maintained, not ‘once-offs’ –Test documentation: how to run, what is tested, and why o Revision control on both modules and tests o Modular development –System broken down into hierarchies of testable components o Automated, incremental integration testing –As code is developed, it is tested standalone, then incrementally within the confines of the system o Continual feedback into development, maintenance cycle –Weekly postings of performance results, current defect status

6. Design Support o Standard testing and debugging methods on each module / class (peek, dump, exercise, performance) o Self-checking code (pre and post parameter asserts, valid state calls) o Debug levels, controlled via runtime flags o Centralized logging mechanisms to collect distributed traces o Logs used in both developer testing and sample user traces from the field

7. Development Tool Support o Shadow development trees for automated and individual testing (ClearCase ‘views’) o Common testing tools, testing approach to simplify testing process, and to allow automated testing o Examples: –Standard test harnesses, method of invocation –Standard testing directories, makefile commands per module –Standard set of test-record-evaluate tools –Central I/O mechanisms to collect distributed test results –Standard system-level drivers –Sequential test harnesses and emulated distributed harnesses to emulate determinism during development

8. Levels of Testing per Module o Basic: used in initial development, later in porting –Minimal level of functionality to establish module operation –Simplicity is critical during development, porting o Detailed: used to verify module correctness before checking into current baseline –Does module meet interface contract –Tests for common coding errors o Regression: replicates previous use which caused failure –Used to verify defect has been corrected –Used to ensure defect does not re-occur over time o Performance: tracks the CPU usage per module and peer to peer performance across distributed components

9. Complex Functional Areas: Testing Examples o Memory: continual test for memory leaks and over-writes –Automated use of Purify within weekly test suites across all modules, all levels of the system o Threads: platform-specific performance and implementation variances must be established per platform –Standard set of tests which mimic system use of threads o Causality: complex areas of the system (such as zero- lookahead ordering) are difficult to establish correctness across all use cases (dropped packets, simultaneous time stamps with variable arrival times, cancelled events, …) –Detailed test scripts, executed in deterministic test harnesses with varying error conditions

10. Periodic Testing Activities o Code walkthroughs: encompass both system code and associated test suites. Testing focus: –Do the test suites sufficiently stress the module –Do the test suites still accurately represent the expected use of the module –Has the underlying platform changed, and has performance changed accordingly o Change Control Board: formal tracking process and tools to establish, record and monitor status of functional change requests, defect priority and status data