David Halbig Rocky Mountain CMG Denver, Colorado Apdex (Application Performance Index):  A Rational Way to Report Delivered Service Quality to Users of the IT Resource? David Halbig Rocky Mountain CMG Denver, Colorado Thursday Feb 26, 2009

Agenda Are there metrics that are of primary importance? What kind of presentation should we use? Raw Data? Derived Data? Formatting? How do we determine thresholds?

What Metrics are Important? How busy is a given resource? What is the end-user perceiving?

What Metrics are Important? How busy is a given resource? What is the end-user perceiving? Delivered Service isn’t EVERYTHING!

What Metrics are Important? How busy is a given resource? What is the end-user perceiving? Delivered Service isn’t EVERYTHING! It’s the ONLY thing! (with apologies to Vince Lombardi)

What kind of presentation should we use?

What kind of presentation should we use?

What kind of presentation should we use?

What kind of presentation should we use? Problems with Raw Data Presentation Does not reflect user satisfaction Does not capture effects of multi-modal distribution The definition of ‘good’ varies with each application, making cross-comparisons difficult

©Netforecast.com - use by permission Apdex Defined Apdex is a numerical measure of user satisfaction with the performance of enterprise applications It defines a method that converts many measurements into one number Uniform 0-1 scale 0 = no users satisfied 1 = all users satisfied Standardized method that is a comparable metric across Applications, Measurement approaches, and Enterprises ©Netforecast.com - use by permission

©Netforecast.com - use by permission How Apdex Works Define T for the application T = Target time (satisfied-tolerating threshold) F = 4T (tolerating-frustrated threshold) Define a Report Group Extract data set from existing measurements Count number of samples in three zones Calculate the Apdex formula Display Apdex value showing T Optionally display value using quality colors Existing Task Response Time Measurement Samples 7 2 Good Fair Poor Unacceptable 0.00T 0.50T 1.00T 0.85T 0.94T 0.70T Excellent Report Group: Application User Group Time Period 3 Frustrated 5 6 4 Tolerating ApdexT= Total samples Satisfied 2 + F 0.91 [6] Tolerating Value T 1 T Satisfied ©Netforecast.com - use by permission

Apdex Example (Computation) T = 4 Seconds (important in setting up buckets, but not in math) # of tx representing ‘satisfied’ = 3000 # of tx representing ‘tolerating’ = 1000 # of tx representing ‘frustrated’ = 500 Total = 4500 Apdex score = 3000 + (1000/2) = 0.77 (out of possible 1.0) 4500 = ‘fair’ response time result However, if we compute ‘average’ response time, we get 2.7 second response time, which is ‘good/excellent’…. ( ((0.1*3000) +(4.1*1000) +(16.1*500))/4500) = 2.7) Hmmmm……

(Another) Example 52% Satisfied 42% Tolerating 6% Frustrated Major eCommerce site ($4B annual on-line sales) North American broadband users accessing the San Francisco data center 16% 14% 12% 52% Satisfied 10% This site had an average response time of 4 seconds, so it looked like all was well But: Apdex = 0.7310 = Fair Probability of Experiencing the Time 8% 6% 4% 42% Tolerating 2% 6% Frustrated 0% 20 40 60 80 100 120 Load Time of a Typical Business Page (sec) ©Netforecast.com - use by permission

Soooo… What should T be set to? Methods for determining T BOGGSAT External SLA mandates Type of work Heads-down data entry Research with certain # of items of interest/entry Competitive information submission/retrieval environment Creative / Analytical work Control Systems work (What role does VARIABILITY play? How would you adjust the Apdex methodology to penalize high variability?)

Counting Interest Elements One Simple check box One data entry field: enter part number Few Select among the following options Expected few lines: headers of recently arrived email Several Type your first name, last name, address, phone number Information on product, prices, shipping alternatives, etc. The user will typically only be interested in a few of these information fields, do not assume if you present 20, the user will read 20 Many Interesting report that is read Scrolling down the page for more content ©Netforecast.com - use by permission

Rating Repetitiveness Very High There are many short tasks to the process High There are a few tasks to the process Low Sometimes there are a few tasks, sometimes there is browsing Very Low The user is browsing, there is no actual process being performed ©Netforecast.com - use by permission

Satisfied-Tolerating Threshold User is satisfied if task completes by T seconds Number of Elements Viewed 1 2 3 4 Very Low 8 12 16 Low 6 9 High Very High Task Repetitiveness Source: NetForecast, Inc.

Apdex SLA SLT SLO SLA California Colorado Florida Minnesota New York 1.00 0.95 0.92 0.90 SLO 0.85 0.80 Apdex [4] 0.75 0.70 0.65 SLA 0.60 Weekdays 0.55 2/25 2/26 2/27 2/28 3/1 3/2 3/3 3/4 3/5 3/6 3/7 3/8 3/9 3/10 3/11 3/12 3/13 3/14 3/15 3/16 3/17

Resources Peter Sevcik NetForecast, Inc. peter@netforecast.com 955 Emerson Drive www.netforecast.com Charlottesville, VA 22901 434 249 1310 Apdex Alliance – www.apdex.org “Designing the User Interface”, Shneiderman / Plaisant “Instrumentation Dashboard Design”, Stephen Few

Questions?