Testing Internet Services Sudipto Ghosh Sambhrama Mundkur Aditya P. Mathur: PI Ramkumar Natarajan Baskar Sridharan Department of Computer Sciences Purdue University @ HP Software Technology Lab Palo Alto Wednesday July 5, 2000 Last updated: June 28, 2000

Test and Management of Internet Services Areas of Concern [1] Test Methodology Adequacy assessment and test enhancement Load testing/Performance measurement Testing for fault tolerance In-use testing and debugging (dynamic testing) Test and Management of Internet Services

Test and Management of Internet Services Areas of Concern [2] Management Uniform interface to a heterogeneous environment Generic event and control specification and processing mechanisms Variable overhead mechanisms Test and Management of Internet Services

Structure of an Internet Service Component Component Client/Server Client/Server Request/data Request/data Stub/Skeleton Stub/Skeleton ORB ORB Component ORB: Object Request Broker Communication . Test and Management of Internet Services

Test and Management of Internet Services Service Domain Client An Internet Service Client A component Client Test and Management of Internet Services

Test and Management of Internet Services Areas of Concern Test Methodology Adequacy assessment and test enhancement Testing for fault tolerance Load testing/Performance measurement In-use testing and debugging (dynamic testing) Test and Management of Internet Services

Elements of a Proposed Test Methodology Test individual components using traditional black-box and white-box techniques. For COTS, use interface testing. Prior to deployment, test applications using Interface Testing. Test for compliance with requirements for FT. Load test an application prior to or during use. While in-use, test and debug an application using dynamic testing. Test and Management of Internet Services

Test and Management of Internet Services Areas of Concern Test Methodology Adequacy assessment and test enhancement Testing for fault tolerance Load testing/Performance measurement In-use testing and debugging (dynamic testing) Test and Management of Internet Services

The Coverage Principle Measuring test adequacy and improving a test set against a sequence of well defined, increasingly strong, coverage domains leads to improved reliability of the system under test. Test and Management of Internet Services

Coverage Domains and Elements Coverage Elements Requirements Classes Functions Mutations Exceptions Data-flows Test and Management of Internet Services

Interface Testing 100% Method Coverage Methods: m1, m2, …,mk # methods executed # methods defined Methods: m1, m2, …,mk 100% Exception Coverage # exceptions raised # exceptions defined Exceptions: e1, e2, …,ek 100% iMutation Score # distinguished mutants total # imutants - #equivalent imutants Interface Component Test and Management of Internet Services

What is Interface Mutation ? Test Suite T contains Request-A. Client Server Interface Request-A Response-A Mutated Interface Client Server Request-A Response-B Test and Management of Internet Services

Interface Mutation: Experimental Evaluation Select components Number of components = 3 Number of interfaces = 6 Number of methods = 82 Number of mutants = 220 Seed errors one by one in the components Errors related to: Portability, data, algorithm, language-specific Test and Management of Internet Services

Characteristics of applications Customer Group Service Manager Client Account Server Component-1 Component-3 APP2 Account Database Manager Activity Logger Component-2 APP1 Administrator APP1 Component-3 Component-2 Component-1 LOC # Interfaces # Methods # Exceptions # Mutants 1484 1268 2950 2 27 10 45 1 69 24 127 APP2 Activity Logger Account Database Manager Object Group Server LOC 1893 2799 5559 # Interfaces 1 2 2 # Methods 11 7 9 # Exceptions 3 4 # Mutants 15 6 13 Test and Management of Internet Services

Control-flow coverage obtained 100% 80% 60% Block Coverage with TME Coverage 40% 20% Component-1 Component-2 Component-3 Block Coverage with TIM APP1 100% 80% Decision Cov- erage with TME Coverage 60% 40% 20% Decision Cov-erage with TIM ActivityLogger AccountDBManager GroupServiceManager APP2 Test and Management of Internet Services

Number of errors revealed 100% 80% 60% Errors 40% Using TME 20% Component-1 Component-2 Component-3 APP1 Using TIM 100% 80% Using TCF 60% Errors 40% 20% ActivityLogger AccountDBManager GroupServiceManager APP2 Test and Management of Internet Services

Number of tests required 30 20 | TME | Tests 10 | TIM | Component-1 Component-2 Component-3 | TCF | APP1 10 Tests 5 ActivityLogger AccountDBManager GroupServiceManager APP2 Test and Management of Internet Services

Test and Management of Internet Services Observations [1] Interface mutation leads to fewer tests that reveal almost as many errors as revealed by statement and decision coverage. Interface mutation is a scalable alternative to using code coverage. Test and Management of Internet Services

Test and Management of Internet Services Observations [2] Reveals programming errors in components. errors in the use of component interfaces Reveals certain types of deadlocks 4 1 2 5 3 Client Server Client Request Server callback Test and Management of Internet Services

Test and Management of Internet Services Areas of Concern Test Methodology Adequacy assessment and test enhancement Testing for fault tolerance Load testing/Performance measurement In-use testing and debugging (dynamic testing) Test and Management of Internet Services

Testing for fault tolerance Problem: Often error recovery code is not executed by test inputs How do we know if the fault recovery code adequately meets the requirements? Solution: Simulate the occurrence of faults by injecting them Fault injection testing at the interface Increases coverage of fault recovery code Reveals inadequacies in fault recovery code Test and Management of Internet Services

Interface fault injection Simulate effects of problems in server, network and ORB Fault Injected m Client Server Stub Skeleton ORB Test and Management of Internet Services

Faults suitable for injection Time delays caused due to Network delays Server crashes or malfunctions Overloaded server Overflowing request buffer in ORB Unexpected exceptions received by client Server crashes Unexpected value User-defined exceptions raised by the server Implementation object invalid Test and Management of Internet Services

Test and Management of Internet Services Areas of Concern Test Methodology Adequacy assessment and test enhancement Testing for fault tolerance Load testing/Performance measurements In-use testing and debugging (dynamic testing) Test and Management of Internet Services

Test and Management of Internet Services Load Testing On: Avg. Latency Effect of: Avg. Load C1 network Clients Server C2 Test and Management of Internet Services

Test and Management of Internet Services Server Statistics Test and Management of Internet Services

Test and Management of Internet Services Client Statistics Test and Management of Internet Services

Performance Instrumentation Goals Low intrusion Selective on-line monitoring Selective on-line filtering To be applied in the Measurement of the latency and CPU consumption (and others) of individual methods for use in capacity planning Demonstration of performance improvement techniques Test and Management of Internet Services

Test and Management of Internet Services Areas of Concern Test Methodology Adequacy assessment and test enhancement Testing for fault tolerance Load testing/Performance measurement In-use testing and debugging (dynamic testing) Test and Management of Internet Services

Test and Management of Internet Services Dynamic Testing Question: How to test an Internet Service while it is in use? Answer: Use the dynamic testing procedure. What is the dynamic testing procedure? Test and Management of Internet Services

Test and Management of Internet Services Dynamic Testing Client Faulty Server group Ra Client 1 Isolated server 3 Faulty server Test Client Ra 2 Test and Management of Internet Services

Limitations of Dynamic Testing Test client might generate undesirable actions: Persistent data modification. Irreversible actions. Application limited to: Closed and well understood domains. Simulated or isolated service environments. Test and Management of Internet Services

Test and Management of Internet Services Areas of Concern [2] Management Uniform interface to a heterogeneous environment Generic event and control specification and processing mechanisms Variable overhead mechanisms Test and Management of Internet Services

Organization of the Service Domain Why organize ? Efficient and scalable management Personalized management Assignment of individual responsibilities Test and Management of Internet Services

Dimensions of Organization Component Geography Client Test and Management of Internet Services

Test and Management of Internet Services Sample Organization Test and Management of Internet Services

Test and Management of Internet Services Areas of Concern [2] Management Uniform interface to a heterogeneous environment Generic event and control specification and processing mechanisms Variable overhead mechanisms Test and Management of Internet Services

Test and Management of Internet Services Monitoring Why? Resource planning Control Testing and debugging What? State of individual or a group of components at different levels of abstraction. Test and Management of Internet Services

Test and Management of Internet Services Control Why? Resource allocation Prevention of unauthorized use Testing and debugging What? Component behavior and location. Test and Management of Internet Services

Test and Management of Internet Services Monitoring: Strategy User Event(s) specifies State Monitor Application State monitors Event detector User specified events detects notifies Event notification Waiting user Test and Management of Internet Services

Test and Management of Internet Services Control: Strategy specifies User Control action(s), trigger event(s) Event notifier Controller signals executes Controller Control action(s) Test and Management of Internet Services

Test and Management of Internet Services Areas of Concern [2] Management Uniform interface to a heterogeneous environment Generic event and control specification and processing mechanisms Variable overhead mechanisms Test and Management of Internet Services

Observer-listener architecture M1 O5 O6 C3 O7 O8 C4 L2 M2 Zonal Manager GUI GUI Test and Management of Internet Services

Test and Management of Internet Services Architecture of Wabash 2.0 [1] LLI MC AR db Zonal Manager L CS LOG Host 1 GUI requests data from the Zonal Manager Wabash GUI Zonal Manager returns data collected from LL to GUI LL returns data to Zonal Manager Zonal Manager requests data from corresponding LL LL determines whether request can be passed or not Client sends a request to a managed object Request C If the request is allowed, LL forwards it to the CORBA Server after time-stamping it If the request is not allowed, LL throws exception and does not forward request to the CORBA Server Response LL gets the response from the CORBA Server LL stores information about the request, records it in a log, sends the response back to client Test and Management of Internet Services

Test and Management of Internet Services Architecture of Wabash 2.0 [2] Zonal Manager MC AR Manager sends command (‘ANY’, DENY, OBJECT_A) db Wabash GUI LLI Command is forwarded to corresponding LL for OBJECT_A Client sends request to OBJECT_A LL CS LOG Host 1 Request C LL does not forward request to CS Throws exception back to client LL determines that request has to be denied Test and Management of Internet Services

Test and Management of Internet Services Architecture of Wabash 3.0 Test and Management of Internet Services

Test and Management of Internet Services Ongoing Research Non-intrusive procedures for dynamic testing. Generalized event-control model and its implementation. Implementation of the unified architecture to assist with the management of JMX, JINI, and CORBA objects. Light-version of Wabash for SmartHome management. Test and Management of Internet Services

The Wabash Project: History Progress: August 1998: Wabash project launched. August 1999 TDS 1.1 available to SERC affiliates. August 2000 Wabash 2.0 available to SERC affiliates. Experiments to assess goodness of proposed interface testing criteria completed. December 2000 Uniform interface for Jini/JMX/CORBA objects. Test and Management of Internet Services