Measuring Interactive Performance with VNCplay Nickolai Zeldovich, Ramesh Chandra Stanford University

Measuring performance Scientific computing or server workloads: –How long does it take for my program to run? Measure total running time

Measuring performance Users of interactive applications: –I asked for something, when will I get it? But we still measure total runtime –WinBench runs Word, PowerPoint, etc as fast as possible, and measures total runtime We are measuring the wrong thing

Case in point Use Microsoft PowerPoint for 5 minutes Two machines: 2.0 GHz and 300 MHz –Expect to see different performance! Two performance metrics: –Total runtime: from start to finish –Response time: from click to output on screen

Performance Metrics Total RuntimeResponse Time

Users pause a lot... 0 sec0.1 sec0.2 sec5 sec... Response time is noise in total runtime

Outline Overview of VNCplay Current approaches How VNCplay works Evaluation –Quantitative results –Qualitative experience Summary

Overview of VNCplay To compare interactive performance, we need reproducible workloads –Record interactive user session –Replay session in different environments Compare response time between replays

Current approaches Display protocols: X11, VNC, Windows Key strokes, mouse clicks Screen updates

Current approaches Screen updates... 0 sec1 sec5 sec8 secTime Input Event Recorder Tools like Xnee, Visual Test, AutoIt, rfbplaymacro

Current approaches Screen updates... 0 sec1 sec5 sec8 secTime Input Event Replayer

What if system is slow? 0 sec0.5 sec1.0 sec Need feedback of when the system responds 1.5 sec Time

What should happen? 0 sec0.5 sec1.0 sec Need feedback of when the system responds 1.5 sec Time

Workarounds? Current tools allow user to insert delay statements to compensate for slowness –Creates unrealistic user behavior –Tedious and error-prone

VNCplay approach Big problem is mouse clicks Take a screen snapshot at each mouse click during recording Wait for the same thing during replay

VNCplay approach 0 sec1 sec5 sec8 secTime Input Event Screen updates Recorder

VNCplay approach Screen updates 0 sec1 sec5 sec8 secTime Input Event Replayer

VNCplay details Only snapshot the screen around the cursor –Reduces trace file size –Ignores irrelevant things (e.g. system clock) Other tricks described in the paper

Analyzing Replays Time Response Time System 1: System 2: Match

Evaluation OpenOffice on Linux Scaled CPU frequency from 2.0 GHz down to 300 MHz Compare total running time and interactive response time

Total Runtime

Median Response Time

Other experiments Paper has details on more experiments –Microsoft Windows, Linux –Evaluate interactive performance of different disk IO schedulers

Experience using VNCplay Platform independence –Measured interactive performance of PowerPoint, Word, OpenOffice Reliable replay –Replayed 5-minute PowerPoint session on a slow system, taking over an hour – works OK!

Caveats Near-perfect reproducibility needed –VMware: start the same VM every time –In Linux, reset user account (dotfiles)

Caveats Keyboard input can be a problem –Use mouse instead of keyboard shortcuts –Click before and after you type Couple of tries to get a reliable session –Tooltips, pop-ups get in the way

Using VNCplay vncplay record server:5901 trace.vnc vncplay play server1:5901 trace.vnc out1.rfb vncplay play server2:5901 trace.vnc out2.rfb vncplay analyze out1.rfb out2.rfb > analyze.out vncanalyze median analyze.out vncanalyze cdf analyze.out > cdf.plot Use gnuplot, Excel to plot cdf.plot

Other uses Automated testing –Make sure your application behaves the same way with a new release of Linux / Windows GUI ``automation’’ –Automate GUI-intensive tasks that have no scriptable interface

Future directions Integrate some GUI toolkit knowledge for smarter matching (e.g. ignore tooltips) Better keyboard input handling –Machine learning –Details in paper If you want to help, let us know!

In conclusion … Need to measure interactive performance –Runtime benchmarks are not appropriate VNCplay –Portable, reliable session replay –Response time measurement

Questions?