1. Measuring What Matters Educause 2007 Craig Blaha, Information Technology Services David Burns, McCombs School of Business Copyright Craig Blaha and David Burns, 2007. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

2. Abstract Determining what to measure, how to measure it, and to whom to report the results can be more of an art than a science. In this session, four major research universities will address how to communicate the importance of IT to administration, outreach, teaching and learning, and research. Project Overview Lessons Learned Moving Forward

3. Who We Are Brian Busby Collaborative Applications Manager University of Wisconsin-Madison Phil Devan Sr. Director Network Research Services The Pennsylvania State University Geoff Lakeman Director, Network Engineering & Technical Services University of Washington Jim Lowe Chief Information Security Officer University of Wisconsin-Madison Shanna Smith, PhD Division of Statistics and Scientific Computation University of Texas

4. What We Did Ambitious Plan –Account for all IT spending –Develop service catalogs –Identify key metrics Focus On Central IT Organizations 4 Stakeholders Of Higher Education –Administration, UW-Madison –Outreach, Penn State –Research, University of Washington –Teaching & Learning, University of Texas

5. Approach Interview key stakeholders outside of IT 1.From your perspective what are the goals, strategies or objectives you are striving to achieve to make the university a better learning and working environment (what matters)? 2.What measurements are you using to gauge the progress towards achieving these goals (how do you know you are succeeding)? 3.From your perspective, how does IT help you make your work successful (role of IT)? 4.What do you think IT should be measuring and why? …This would be where things went off the rails.

6. What Happened Discoveries –People hadn’t had a conversation like this with someone from IT… EVER! –Metrics don’t matter?!? Goals Barriers Role of IT

7. Goals Teaching and Learning –Reasoning, moral & ethical thinking –Computer competency –Problem solving –Teamwork and communication Outreach –Research, teaching, service –Leverage talents of university Research –Research grants & contracts, published work –Positive educational & economic impact –Collaboration Administration –Efficiency, transparency, integrity –Collaboration and communication –Savings

8. Barriers Teaching and Learning –Many faculty married to old learning models –Individualized, flexible learning model is difficult with one teacher and many students Administration –Resistance to change –External forces –Politics and minority stakeholders Research –Funding –Improving access to resources –Available infrastructure –Highly competitive Outreach –Funding –Marketing/perception –Geography –Little involvement locally

9. Role of IT Teaching and Learning –Accessibility –Resource-saving –Interaction & collaboration –On-the-spot information –Student perceptions Administration –It’s not the technology – it’s what we do with it –IT helps us provide services –Stakes are high - success of IT is tied to success of administrative organization Outreach –Enabler and enhancer Videoconferencing for problem solving Fundamental to World Campus –Much of it should be common good IT is fundamental Like a utility Research –Research grants depend on technology infrastructure –Sharing IT resources from around the world promotes big science

10. “Not in the least bit interested, metrics become the goal”

11. Themes –Communication –Collaborative Metrics –Alignment

12. Communication

13. “No news is good news” Better communications from IT –What’s going on? –Lay language –“IT has A real marketing problem” Listen –Understand the real mission Laying the groundwork –Establishing a relationship –Working towards goals, metrics, and alignment

14. Essential Questions What is the value derived from IT? What are the results that matter? –Measuring performance against desired outcomes –Converting expectations to service standards How are these results measured? –Types of performance measures Reliability measures Responsiveness measures Project measures Utilization/adoption Client satisfaction

15. A Path Forward

16. Collaborating On Metrics Leadership from IT required Starting the conversation about metrics Present a metric Discuss that metric regularly –Especially progress or problems See if it ends up being useful Wash, rinse, repeat. It doesn’t matter unless you measure it If it doesn’t matter – DON’T measure it!

17. Money Matters Technology Spending Enables IT Development or Implementation Enables Enhanced Mission Capability Worth The Cost? Measures of IT changes Measures of mission changes Itemized IT Expenses

18. Current Metrics Reliability Measures –Quality = no errors –Uptime Utilization/Adoption –Blackboard usage statistics Client Satisfaction –Blackboard satisfaction –Annual survey Responsiveness Measures –Desktop support Project Measures –Increased funding –Technology licensing –Budgets and timelines –Statements of work

19. Goals and Measures Goals –Operational excellence –Communication –Fiscal responsibility Recommended Metrics –Project based metrics are fine –Qualitative better than quantitative

20. Barriers Behavior beats metrics Metrics that don’t matter Defensive metrics Fear of accountability

21. Finding Alignment

22. Alignment Process Identifying Stakeholders Understanding Goals and Measures Mapping IT Services and Metrics Feedback and Refine

23. Alignment Process Identify Stakeholders –Planning that leaves out the middle Their Goals and Measures –What is important to them –How do they measure themselves Mapping IT Services and Metrics –What do they use –What measures do you have for those services Feedback and Refine –Create ongoing dialog and engagement

24. Key Points Focus on the mission Different strokes… Understand customer needs Metrics are collaborative Active listening is communication Strategic planning – mind the gap Sell IT!