1. Measuring Interactive Performance with VNCplay Nickolai Zeldovich, Ramesh Chandra Stanford University

2. Measuring performance Scientific computing or server workloads: –How long does it take for my program to run? Measure total running time

3. Measuring performance Users of interactive applications: –I asked for something, when will I get it? But we still measure total runtime –WinBench runs Word, PowerPoint, etc as fast as possible, and measures total runtime We are measuring the wrong thing

4. Case in point Use Microsoft PowerPoint for 5 minutes Two machines: 2.0 GHz and 300 MHz –Expect to see different performance! Two performance metrics: –Total runtime: from start to finish –Response time: from click to output on screen

5. Performance Metrics Total RuntimeResponse Time

6. Users pause a lot... 0 sec0.1 sec0.2 sec5 sec... Response time is noise in total runtime

7. Outline Overview of VNCplay Current approaches How VNCplay works Evaluation –Quantitative results –Qualitative experience Summary

8. Overview of VNCplay To compare interactive performance, we need reproducible workloads –Record interactive user session –Replay session in different environments Compare response time between replays

9. Current approaches Display protocols: X11, VNC, Windows Key strokes, mouse clicks Screen updates

10. Current approaches Screen updates... 0 sec1 sec5 sec8 secTime Input Event Recorder Tools like Xnee, Visual Test, AutoIt, rfbplaymacro

11. Current approaches Screen updates... 0 sec1 sec5 sec8 secTime Input Event Replayer

12. What if system is slow? 0 sec0.5 sec1.0 sec Need feedback of when the system responds 1.5 sec Time

13. What should happen? 0 sec0.5 sec1.0 sec Need feedback of when the system responds 1.5 sec Time

14. Workarounds? Current tools allow user to insert delay statements to compensate for slowness –Creates unrealistic user behavior –Tedious and error-prone

15. VNCplay approach Big problem is mouse clicks Take a screen snapshot at each mouse click during recording Wait for the same thing during replay

16. VNCplay approach 0 sec1 sec5 sec8 secTime Input Event Screen updates Recorder

17. VNCplay approach Screen updates 0 sec1 sec5 sec8 secTime Input Event Replayer

18. VNCplay details Only snapshot the screen around the cursor –Reduces trace file size –Ignores irrelevant things (e.g. system clock) Other tricks described in the paper

19. Analyzing Replays Time Response Time System 1: System 2: Match

20. Evaluation OpenOffice on Linux Scaled CPU frequency from 2.0 GHz down to 300 MHz Compare total running time and interactive response time

21. Total Runtime

22. Median Response Time

23. Other experiments Paper has details on more experiments –Microsoft Windows, Linux –Evaluate interactive performance of different disk IO schedulers

24. Experience using VNCplay Platform independence –Measured interactive performance of PowerPoint, Word, OpenOffice Reliable replay –Replayed 5-minute PowerPoint session on a slow system, taking over an hour – works OK!

25. Caveats Near-perfect reproducibility needed –VMware: start the same VM every time –In Linux, reset user account (dotfiles)

26. Caveats Keyboard input can be a problem –Use mouse instead of keyboard shortcuts –Click before and after you type Couple of tries to get a reliable session –Tooltips, pop-ups get in the way

27. Using VNCplay vncplay record server:5901 trace.vnc vncplay play server1:5901 trace.vnc out1.rfb vncplay play server2:5901 trace.vnc out2.rfb vncplay analyze out1.rfb out2.rfb > analyze.out vncanalyze median analyze.out vncanalyze cdf analyze.out > cdf.plot Use gnuplot, Excel to plot cdf.plot

28. Other uses Automated testing –Make sure your application behaves the same way with a new release of Linux / Windows GUI ``automation’’ –Automate GUI-intensive tasks that have no scriptable interface

29. Future directions Integrate some GUI toolkit knowledge for smarter matching (e.g. ignore tooltips) Better keyboard input handling –Machine learning –Details in paper If you want to help, let us know!

30. In conclusion … Need to measure interactive performance –Runtime benchmarks are not appropriate VNCplay –Portable, reliable session replay –Response time measurement

31. Questions? http://suif.stanford.edu/vncplay/