1. Your Session Leader Martin Klubeck Educause ITMetrics CG Leader
Strategy & Planning Consultant
359 ITC, University of Notre Dame 46556
Educause ITMetrics CG Leader 1

2. Handout
What it is
How to use
_______ 1a

3. Agenda Presentation Practicum Metrics Overview Answer Key Report Card
Team Breakouts
Practicum
Hands-on
Group Presentations
Q & A / Wrap-up 2

4. Metrics Overview Why Use Metrics?
To Help You …
Gain support from above
Provide visibility
Improve processes
Make better “data driven” decisions 3
Metrics Overview

5. What Is A Metric?
Data
Measure
Information 4
Metrics Overview

6. Metrics… Are question driven Tell a complete story Include: Data
Measures
Information
Other Metrics 5
Metrics Overview

7. Leaves Limbs Trunk Roots
Branches
Measures
Leaves
Limbs
Data
Information
Trunk
Metric
Roots
The Question 6
Metrics Overview

8. Draw a line between the item and the correct definition
Exercise 1
Draw a line between the item and the correct definition
How healthy is my car?
Data
28 mpg
Measure 28
Information
28 mpg in the city, using regular unleaded
Metric
Root Question
Usually, get 31 mpg in the city, using regular unleaded E1
Metrics Overview

9. Exercise 1 How healthy is my car? Data 28 mpg Measure 28 Information
28 mpg in the city, using regular unleaded
Metric
Root Question
Usually, get 31 mpg in the city, using regular unleaded E1
Metrics Overview

10. How Not To Use Metrics To “support my case” To “motivate” the staff
To “manage” the staff or others
To evaluate individual performance 7
Metrics Overview

11. How To Use Metrics Explain how it will Investigate
Share – Close the loop
and WON’T be used 8
Metrics Overview

12. Who Will Use The Metric? Campus Community Management Service Providers
Owners and Workers
Leadership 9
Metrics Overview

13. Top Five Warning Signs The boss says – “I’ll know it when I see it.”1.
The boss says – “I’ll know it when I see it.”
“We’ve been collecting this data for five years and no one is using it.”
“Do we have any data on...?”
“They don’t trust the data.”
“Sounds interesting, let’s collect it.” 2. 3. 4. 5. 10
Metrics Overview

14. Drawing A Picture Focus on “how it looks”
A picture is worth a thousand words
Send a clear message 11
Metrics Overview

15. 12
Metrics Overview

16. 13
Metrics Overview

17. 14
Metrics Overview

18. The Complete Metric “Pallet”
Metric Name
Purpose
Metric Area/Category
Customer
Graphical Representation
Explanation
Metrics Analysis
Measures used to develop metric
Collection Schema
Schedule
Assumptions and Constraints
Related Metrics and Data Dependencies
Lessons Learned 15
Metrics Overview

19. The Primary Metric Colors
Purpose
Graphical representation
Explanation
Metrics analysis
Measures used to develop metric 16
Metrics Overview

20. Answer Key 17

21. Information Needs 18
Answer Key

22. Information Needs Value to Organization Management of Resources19
Answer Key

23. Information Needs Answer Key Effectiveness Value to Organization
Customer view
Value to
Organization
Efficiency
Business view
Information Needs
Human Resources
Worker view
Management
of Resources
Visibility
Management view 20
Answer Key

24. Effectiveness Value to Organization Efficiency Campus Needs
Delivery
Effectiveness
Customer Satisfaction
Customer view
Customer Service
Value to
Organization
Cost
Efficiency
Time
Campus view
Quality
Campus Needs
Employee Satisfaction
Training
Human Resources
Work Environment
Worker view
Reward & Recognition
Management
of Resources
Project/Program Status
Resource Allocation
Visibility
Strat. Planning & Goal Attainment
Management view
Communications
Priority Setting 21
Answer Key

25. Report Card OIT Operational Excellence22

26. OIT Operational Excellence
A Little Background - Impact on Staff Workload
Minimal impact, no new work created
Workload impact over life of project
Operational Gains (benefits)
Better processes
Better use of tracking tools
Better service 23
Report Card

27. OIT Operational Excellence
Goal: make it as simple as possible (but no simpler)
Flexibility (expectations, measures, weights)
If we err, err on the conservative side
Overcoming FUD is essential – how we will use and how we won’t use the measures 24
Report Card

28. OIT Operational Excellence
Effectiveness – Customer View
Triangulation
Delivery, Customer Satisfaction, and Usage
Security – a critical factor 25
Report Card

29. Exercise 2 Restaurant Scenario
Delivery
Availability
Speed
Accuracy
Usage
Customer Satisfaction
Security E2

30. OIT Operational Excellence
Some Keys to Stress:
Customer View
Indicators – with a compound message when compared with each other
Expectations vs. goals/thresholds 26
Report Card

31. OIT Operational Excellence
Primer on Expectations
Rework (take a guess)
Customer Satisfaction 1-3 vs. 5 (take a guess) 27
Report Card

32. OIT Operational Excellence
Example 1 28
Report Card

33. OIT Operational Excellence
Example 1 29
Report Card

34. OIT Operational Excellence
Example 1 30
Report Card

35. OIT Operational Excellence
Example 1 31
Report Card

36. OIT Operational Excellence32
Report Card

37. OIT Operational Excellence
Example 2 33
Report Card

38. OIT Operational Excellence
Example 2 34
Report Card

39. OIT Operational Excellence
Example 2 35
Report Card

40. OIT Operational Excellence
Example 2 36
Report Card

41. OIT Operational Excellence
Example 2 37
Report Card

42. OIT Operational Excellence
Expectations – a test
How do we deal with anomalies?
If we fail to meet expectations – Opp for Improvement
Current Month
Over the Fiscal Year?
If we exceed expectations 38
Report Card

43. OIT Operational Excellence
Expectations – ANSWERS
How do we deal with anomalies?
INVESTIGATE
Decide if want to expend resources to investigate
Do you need focus on the area? (O)
Are you expending too much focus/resources? (E)
Ask – How does this align with the other Measures? 39
Report Card

44. OIT Operational Excellence
Making the Report Card
How measures are chosen
Setting weights (delivery is an example)
Collecting data
Identifying expectations
Determining the “grade”
Rolling up the grades
Efficiency Metrics – later phase, after organization learns to use metrics properly 40
Report Card

45. Understanding the Performance of OIT
The metrics answer the questions: “How is the OIT doing?” and “How do you know?”
Security
The ability of an IT service or one of its components to perform its required function while maintaining confidentiality, integrity, and availability of data
Delivery
Delivery is essentially the Reliability of a system – The ability of a system or component to perform its required functions under stated conditions for a specified period of time.
Availability (there when you want it), Speed (during defined time intervals, within specified time constraints), Accuracy (w/in acceptable amount of error or fault)
The actual use of an IT service or one of its components, measured against the intended customer base
Usage
Customer Satisfaction
The customers’ perceived level of satisfaction with an IT service or one of its components
(Availability (w) + Speed (w) + Accuracy (w)) : Usage : Customer Sat : Security
w = weight 41
Report Card

46. Services requiring action
Report Card
Services
Delivery
Usage
Cust Sat
Security
Key Service 1 E E E M
E = Exceeds expectations
Key Service 2 E M M M
M = Meets expectations
Key Service 3 M M M M
Key Service 4 O O M M
O = Opportunity for Improvement
Key Service 5 E E E M
Key Service 6 M M M M
Key Service 7 O O O M E O E M
Key Service 8
Services requiring action
Required Action
Low Performers
Service 7
Action Plan
Disparities
Service 2
Investigation
Service 4
Service 8 42
Report Card

47. Opportunity for Improvement
Adding up the grades
Meets Expectations 10 9 8 7 6 5 4 3 2 1
Exceeds Expectations
Opportunity for Improvement
Translation value 2
Translation value 1
Actual Grade 43
Report Card

48. Adding up the grades 44
Report Card

49. Dashboard? 45
Report Card

50. Adding up the grades 46
Report Card

51. What It Looks Like
The final product 47

52. Team Breakouts Team Breakouts Are you a change agent?
Are you a manager or a customer?
What service do you want to measure? 48

53. Use the Implementation Guides – as many as you need
Practicum
Hands-on
Group presentations
Q & A / wrap-up
Use the Implementation Guides – as many as you need 49

54. Hands on Identify basic information about the service What is it?
Why is it? What is the purpose?
Who does it serve?
How many customers?
What type of customers?
Who is the Service Provider? 50
Practicum

55. Hands on Identify measures Revisit the Purpose of the service
Customer Satisfaction is easiest to start with
Perception of the customer
Pretty much have to ask
Usage
May be N/A if monopoly (like telephone) 51
Practicum

56. Hands on Identify measures Delivery Security Availability Speed
Accuracy
Security 52
Practicum

57. Hands on Expectations Meets Range (upper and lower bounds)
Opportunity for Improvement
Exceeds Expectations 53
Practicum

58. Questions? 54

59. Join our Constituent Group
Thank You!
Join our Constituent Group
Educause ITMetrics CG 55

60. Copyright 2009 Martin Klubeck
Metrics Report Card
Copyright Martin Klubeck
This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.