Measurement, Evaluation, Analysis & Benchmarking Towards a CUCCIO Program Michael Ridley Chief Information Officer (CIO) and Chief Librarian University of Guelph

Agenda  Review current initiatives in evaluation & analysis  Solicit feedback, interest & directions  Next steps for CUCCIO

Strategic Objectives  Core descriptive data with longitudinal analysis & institutional comparisons  Benchmark against “best practice”  Accountability Reporting  Data driven decision making  Institutional/CUCCIO KPIs  Others?

Review of Current Initiatives  EDUCAUSE Core Data  CAUDIT: Core Data + Local Data + Benchmarking  TechQual+: client satisfaction

EDUCAUSE Core Data  Established program & analysis tools  Historical data  Stable questionnaire  Good US participation  Limited Canadian participation 2005: : 15

EDUCAUSE Core Data: Issues  Data integrity definitions (data consistency) Measured, estimated, guessed  Extendibility  Applicability to Canadian setting  Workload re responding: onerous?

CAUDIT Core Data Extensions  Additional elements  EDUCAUSE policy re data  Double collection & management  Increased relevancy

Apples & Oranges  IT is non-uniform across institutions Central/Distributed Cost allocation models Multi campuses/sites  Normalization possibilities? Ratios Limited comparator data

CAUDIT Benchmarking  Limited data elements (weighted) FTE Students FTE Faculty & Staff Research Income  Existing tools & techniques  Enabled identification of comparators or institutional benchmarks  Renewed program: need for focus

CAUDIT & CUCCIO  Political & social similarities  Alignment re higher education  Willingness to collaborate  Small organization undergoing change

TechQual+  Qualitative analysis of client/user expectation & satisfaction  ServQual+ & LibQual+  Leadership: Texas A&M; Pepperdine  Participation: Toronto, Guelph, UNB

TechQual+  30 individual types of technology services grouped in 6 clusters.  For each service: minimum expectation level desired service level perceived service level  “Zone of Tolerance”

TechQual+  Access & Connectivity  Support & User Experience  Computing & Classroom Technology  Information Systems  Publishing Tools  Governance & Planning

TechQual+  Canadian version?  East-West comparison more important than North-South?  Still a work in progress  Worthy of investment?

What is Important to CUCCIO?  What do we want to know?  Identify the data to be collected Descriptive & transactional data Inventory data User satisfaction Best or current practices Benchmarks KPIs

Proposals  EDUCAUSE Core Data 2007 CUCCIO members participate Agree on data definitions Create a CUCCIO HE IT Scorecard  Create a CUCCIO E&A SIG Examine CAUDIT experience Monitor TechQual+  E&A Workshop at CANHEIT 2008

E&A Workshop at CANHEIT  Professional development opportunity for management staff  Key principles and practices of effective E&A  CAUDIT & EDUCASE participation  Consolidate CUCCIO E&A Program

Measurement, Evaluation, Analysis & Benchmarking Towards a CUCCIO Program DISCUSSION