1. Testing Internet Services
Sudipto Ghosh
Sambhrama Mundkur
Aditya P. Mathur: PI
Ramkumar Natarajan
Baskar Sridharan
Department of Computer Sciences
Purdue University
@ HP Software Technology Lab
Palo Alto
Wednesday July 5, 2000
Last updated: June 28, 2000

2. Test and Management of Internet Services
Areas of Concern [1]
Test
Methodology
Adequacy assessment and test enhancement
Load testing/Performance measurement
Testing for fault tolerance
In-use testing and debugging (dynamic testing)
Test and Management of Internet Services

3. Test and Management of Internet Services
Areas of Concern [2]
Management
Uniform interface to a heterogeneous environment
Generic event and control specification and processing mechanisms
Variable overhead mechanisms
Test and Management of Internet Services

4. Structure of an Internet Service
Component
Component
Client/Server
Client/Server
Request/data
Request/data
Stub/Skeleton
Stub/Skeleton
ORB
ORB
Component
ORB: Object Request
Broker
Communication .
Test and Management of Internet Services

5. Test and Management of Internet Services
Service Domain
Client
An Internet Service
Client
A component
Client
Test and Management of Internet Services

6. Test and Management of Internet Services
Areas of Concern
Test
Methodology
Adequacy assessment and test enhancement
Testing for fault tolerance
Load testing/Performance measurement
In-use testing and debugging (dynamic testing)
Test and Management of Internet Services

7. Elements of a Proposed Test Methodology
Test individual components using traditional black-box and white-box techniques.
For COTS, use interface testing.
Prior to deployment, test applications using Interface Testing.
Test for compliance with requirements for FT.
Load test an application prior to or during use.
While in-use, test and debug an application using dynamic testing.
Test and Management of Internet Services

8. Test and Management of Internet Services
Areas of Concern
Test
Methodology
Adequacy assessment and test enhancement
Testing for fault tolerance
Load testing/Performance measurement
In-use testing and debugging (dynamic testing)
Test and Management of Internet Services

9. The Coverage Principle
Measuring test adequacy and improving a test set against a sequence of well defined, increasingly strong, coverage domains leads to improved reliability of the system under test.
Test and Management of Internet Services

10. Coverage Domains and Elements
Coverage Elements
Requirements
Classes
Functions
Mutations
Exceptions
Data-flows
Test and Management of Internet Services

11. Interface Testing 100% Method Coverage Methods: m1, m2, …,mk
# methods executed
# methods defined
Methods:
m1, m2, …,mk
100% Exception Coverage
# exceptions raised
# exceptions defined
Exceptions:
e1, e2, …,ek
100% iMutation Score
# distinguished mutants
total # imutants - #equivalent imutants
Interface
Component
Test and Management of Internet Services

12. What is Interface Mutation ?
Test Suite T contains Request-A.
Client
Server
Interface
Request-A
Response-A
Mutated Interface
Client
Server
Request-A
Response-B
Test and Management of Internet Services

13. Interface Mutation: Experimental Evaluation
Select components
Number of components = 3
Number of interfaces = 6
Number of methods = 82
Number of mutants = 220
Seed errors one by one in the components
Errors related to:
Portability, data, algorithm, language-specific
Test and Management of Internet Services

14. Characteristics of applications
Customer
Group Service
Manager
Client
Account
Server
Component-1
Component-3
APP2
Account
Database
Manager
Activity
Logger
Component-2
APP1
Administrator
APP1
Component-3
Component-2
Component-1
LOC
# Interfaces
# Methods
# Exceptions
# Mutants
1484
1268
2950 2 27 10 45 1 69 24
127
APP2
Activity
Logger
Account
Database
Manager
Object
Group
Server
LOC
1893
2799
5559
# Interfaces 1 2 2
# Methods 11 7 9
# Exceptions 3 4
# Mutants 15 6 13
Test and Management of Internet Services

15. Control-flow coverage obtained
100%
80%
60%
Block Coverage
with TME
Coverage
40%
20%
Component-1
Component-2
Component-3
Block Coverage with TIM
APP1
100%
80%
Decision Cov-
erage with TME
Coverage
60%
40%
20%
Decision Cov-erage with TIM
ActivityLogger
AccountDBManager
GroupServiceManager
APP2
Test and Management of Internet Services

16. Number of errors revealed
100%
80%
60%
Errors
40%
Using TME
20%
Component-1
Component-2
Component-3
APP1
Using TIM
100%
80%
Using TCF
60%
Errors
40%
20%
ActivityLogger
AccountDBManager
GroupServiceManager
APP2
Test and Management of Internet Services

17. Number of tests required30 20
| TME |
Tests 10
| TIM |
Component-1
Component-2
Component-3
| TCF |
APP1 10
Tests 5
ActivityLogger
AccountDBManager
GroupServiceManager
APP2
Test and Management of Internet Services

18. Test and Management of Internet Services
Observations [1]
Interface mutation leads to fewer tests that reveal almost as many errors as revealed by statement and decision coverage.
Interface mutation is a scalable alternative to using code coverage.
Test and Management of Internet Services

19. Test and Management of Internet Services
Observations [2]
Reveals
programming errors in components.
errors in the use of component interfaces
Reveals certain types of deadlocks 4 1 2 5 3
Client
Server
Client Request
Server callback
Test and Management of Internet Services

20. Test and Management of Internet Services
Areas of Concern
Test
Methodology
Adequacy assessment and test enhancement
Testing for fault tolerance
Load testing/Performance measurement
In-use testing and debugging (dynamic testing)
Test and Management of Internet Services

21. Testing for fault tolerance
Problem:
Often error recovery code is not executed by test inputs
How do we know if the fault recovery code adequately meets the requirements?
Solution:
Simulate the occurrence of faults by injecting them
Fault injection testing at the interface
Increases coverage of fault recovery code
Reveals inadequacies in fault recovery code
Test and Management of Internet Services

22. Interface fault injection
Simulate effects of
problems in server,
network and ORB
Fault
Injected m
Client
Server
Stub
Skeleton
ORB
Test and Management of Internet Services

23. Faults suitable for injection
Time delays caused due to
Network delays
Server crashes or malfunctions
Overloaded server
Overflowing request buffer in ORB
Unexpected exceptions received by client
Server crashes
Unexpected value
User-defined exceptions raised by the server
Implementation object invalid
Test and Management of Internet Services

24. Test and Management of Internet Services
Areas of Concern
Test
Methodology
Adequacy assessment and test enhancement
Testing for fault tolerance
Load testing/Performance measurements
In-use testing and debugging (dynamic testing)
Test and Management of Internet Services

25. Test and Management of Internet Services
Load Testing
On: Avg. Latency
Effect of: Avg. Load C1
network
Clients
Server C2
Test and Management of Internet Services

26. Test and Management of Internet Services
Server Statistics
Test and Management of Internet Services

27. Test and Management of Internet Services
Client Statistics
Test and Management of Internet Services

28. Performance Instrumentation Goals
Low intrusion
Selective on-line monitoring
Selective on-line filtering
To be applied in the
Measurement of the latency and CPU consumption (and others) of individual methods for use in capacity planning
Demonstration of performance improvement techniques
Test and Management of Internet Services

29. Test and Management of Internet Services
Areas of Concern
Test
Methodology
Adequacy assessment and test enhancement
Testing for fault tolerance
Load testing/Performance measurement
In-use testing and debugging (dynamic testing)
Test and Management of Internet Services

30. Test and Management of Internet Services
Dynamic Testing
Question:
How to test an Internet Service while it is in use?
Answer: Use the dynamic testing procedure.
What is the dynamic testing procedure?
Test and Management of Internet Services

31. Test and Management of Internet Services
Dynamic Testing
Client
Faulty
Server group Ra
Client 1
Isolated
server 3
Faulty
server
Test
Client Ra 2
Test and Management of Internet Services

32. Limitations of Dynamic Testing
Test client might generate undesirable actions:
Persistent data modification.
Irreversible actions.
Application limited to:
Closed and well understood domains.
Simulated or isolated service environments.
Test and Management of Internet Services

33. Test and Management of Internet Services
Areas of Concern [2]
Management
Uniform interface to a heterogeneous environment
Generic event and control specification and processing mechanisms
Variable overhead mechanisms
Test and Management of Internet Services

34. Organization of the Service Domain
Why organize ?
Efficient and scalable management
Personalized management
Assignment of individual responsibilities
Test and Management of Internet Services

35. Dimensions of Organization
Component
Geography
Client
Test and Management of Internet Services

36. Test and Management of Internet Services
Sample Organization
Test and Management of Internet Services

37. Test and Management of Internet Services
Areas of Concern [2]
Management
Uniform interface to a heterogeneous environment
Generic event and control specification and processing mechanisms
Variable overhead mechanisms
Test and Management of Internet Services

38. Test and Management of Internet Services
Monitoring
Why?
Resource planning
Control
Testing and debugging
What?
State of individual or a group of components at different levels of abstraction.
Test and Management of Internet Services

39. Test and Management of Internet Services
Control
Why?
Resource allocation
Prevention of unauthorized use
Testing and debugging
What?
Component behavior and location.
Test and Management of Internet Services

40. Test and Management of Internet Services
Monitoring: Strategy
User
Event(s)
specifies
State Monitor
Application State
monitors
Event detector
User specified events
detects
notifies
Event notification
Waiting user
Test and Management of Internet Services

41. Test and Management of Internet Services
Control: Strategy
specifies
User
Control action(s),
trigger event(s)
Event notifier
Controller
signals
executes
Controller
Control action(s)
Test and Management of Internet Services

42. Test and Management of Internet Services
Areas of Concern [2]
Management
Uniform interface to a heterogeneous environment
Generic event and control specification and processing mechanisms
Variable overhead mechanisms
Test and Management of Internet Services

43. Observer-listener architectureM1 O5 O6 C3 O7 O8 C4 L2 M2
Zonal
Manager
GUI
GUI
Test and Management of Internet Services

44. Test and Management of Internet Services
Architecture of Wabash 2.0 [1]
LLI MC AR db
Zonal Manager L CS
LOG
Host 1
GUI requests data from
the Zonal Manager
Wabash
GUI
Zonal Manager returns
data collected from LL
to GUI
LL returns data
to Zonal Manager
Zonal Manager requests data
from corresponding LL
LL determines whether request can be passed or not
Client sends a request
to a managed object
Request C
If the request is allowed,
LL forwards it to the
CORBA Server after
time-stamping it
If the request is not allowed,
LL throws exception and does
not forward request to the
CORBA Server
Response
LL gets the response
from the CORBA Server
LL stores information about the request, records it in a log, sends the response back to client
Test and Management of Internet Services

45. Test and Management of Internet Services
Architecture of Wabash 2.0 [2]
Zonal Manager MC AR
Manager sends command
(‘ANY’, DENY, OBJECT_A) db
Wabash
GUI
LLI
Command is forwarded to
corresponding LL for OBJECT_A
Client sends request
to OBJECT_A LL CS
LOG
Host 1
Request C
LL does not forward
request to CS
Throws exception back
to client
LL determines that request
has to be denied
Test and Management of Internet Services

46. Test and Management of Internet Services
Architecture of Wabash 3.0
Test and Management of Internet Services

47. Test and Management of Internet Services
Ongoing Research
Non-intrusive procedures for dynamic testing.
Generalized event-control model and its implementation.
Implementation of the unified architecture to assist with the management of JMX, JINI, and CORBA objects.
Light-version of Wabash for SmartHome management.
Test and Management of Internet Services

48. The Wabash Project: History
Progress:
August 1998:
Wabash project launched.
August 1999
TDS 1.1 available to SERC affiliates.
August 2000
Wabash 2.0 available to SERC affiliates.
Experiments to assess goodness of proposed interface testing criteria completed.
December 2000
Uniform interface for Jini/JMX/CORBA objects.
Test and Management of Internet Services