1. Imperial College - Department of Computing Continuous Performance Testing in Virtual Time Nikos Baltas & Tony Field Department of Computing Imperial College London

2. Imperial College - Department of Computing Motivation Performance analysis:  Early stages: performance modelling  Late stages: profiling & performance testing  Intermediate stages: extract models from code, calibrate them and analyse... Can we simplify this? Can we performance test incomplete applications?

3. Imperial College - Department of Computing Idea Integrate models and code Don't model existing code – execute it in virtual time! Use performance models that describe the expected response times of missing (or existing) methods Combine measurements and model simulations to guide the thread progress of the application

4. Imperial College - Department of Computing Virtual Time Execution

5. Imperial College - Department of Computing VEX Virtual Execution framework (VEX) Java applications Components  Thread control  Profiling  I/O handling

6. Imperial College - Department of Computing VEX simulation Model existing code by profiling it on the simulation host Map the (possibly scaled) measurements on a virtual timeline Control the thread progress to enforce a fair scheduling policy in virtual time

7. Imperial College - Department of Computing Integration of models to VEX Don't just scale method durations: replace them Use time acquired by the simulation of performance models (open queuing networks) Decouple profiled time from virtual time measurements Use for either implemented methods or code stubs Continuous performance testing

8. Imperial College - Department of Computing Example

9. Imperial College - Department of Computing Issues How does the model-simulation time relate to the virtual time of executed code? How do we control the model simulation? What if the code of the model-described method interacts with the application?

10. Imperial College - Department of Computing Remote and local resources

11. Imperial College - Department of Computing Model simulation in VEX Thread adds job Thread state depends on job state in queue:  L-Think: Running/able  R-Think: Timed-waiting  Queuing: Waiting  L-Serv: Running/able  R-Serv: Timed-waiting Thread and job join at a Sink node

12. Imperial College - Department of Computing Model description: JMT [1] [1] M. Bertoli, G. Casale, and G. Serazzi, “JMT: performance engineering tools for system modeling,” SIGMETRICS Perform.Eval.Rev., vol. 36, no. 4, pp. 10–15, 2009.

13. Imperial College - Department of Computing Functional consistency T1T1 T2T2 Performance model for M() synchronized(obj) {resume suspend M() { synchronized(obj) { wait for T 1 to resume blocked on obj monitor Deadlock! violate virtual schedule and resume anyway } exit monitor objsuspend enter/exit monitor } M() Join with model resume

14. Imperial College - Department of Computing Usage -Xbootclasspath/p:vex.jar –javaagent:jine.jar –agentpath:libjinevex.sojavaClass void foo() { ….. bar(); ….. } @virtualtime.ModelPerformance( jmtModelFilename=“file.jsimg”, replaceMethodBody=true, sourceNodeLabel=“Source0”)

15. Imperial College - Department of Computing Evaluation Case study to demonstrate usage and verify functionality of approach Sample results and parameters Validate process by comparing predicted results to real times Study: remote DB server with local cache

16. Imperial College - Department of Computing Pure model

17. Imperial College - Department of Computing Partially complete code

18. Imperial College - Department of Computing Complete code

19. Imperial College - Department of Computing Results

20. Imperial College - Department of Computing Conclusion Integration of models and code within VEX Models and code can be used interchangeably Continuous performance testing without extracting models from code Prediction errors due to disregard for background system load and I/O approximation

21. Imperial College - Department of Computing Future work Handle more behaviours (asynchronous I/O) Lower-level VEX and/or distributed virtual execution Application on larger scale projects Release VEX under open source license Extensions to other languages (C++)

22. Imperial College - Department of Computing Questions? Thank you

23. Imperial College - Department of Computing Virtual Time Execution II

24. Imperial College - Department of Computing Remote and local resources II

25. Imperial College - Department of Computing Local resources visualisation