1 Metrics and Money: The Process and Politics of Accountability Stephen Daigle, Ph.D, California State University Michael Large, Ph.D, Social and Behavioral Research Institute, CSU San Marcos Patricia Cuocco, California State University Copyright Patricia M. Cuocco, Stephen L. Daigle, Michael Large and Allen Risley, 2003

2 Overview ITS and the Measures of Success (MOS) MOS Research Agenda Findings Questions (and, we hope, answers)

3 Integrated Technology Strategy Early 1990s – Presidential Interest in Using Technology As a Strategic Tool Driven By: – Immense Growth in Technology, –Antiquated Legacy Systems, –Increasing Demand for Access –Fiscal Constraints Mid 1990s – Develop Integrated Technology Strategy Framework –Iterative Process, Constituent Input and Approval

4 ITS – The Process 1996 Accepted by Board of Trustees Living Framework – Not Written Plan 2003 – Technology Decisions Still Governed by ITS

5 What Is ITS Outcome Driven –-Excellence in Learning and Teaching –Quality of the Student Experience –Administrative Quality and Productivity –Personal Productivity

6 ITS – the Icon

7 ITS – How It Fits Personal Productivity - Attained Through Infrastructure Initiative – Minimum Baseline at All CSU Campuses The Infrastructure Is Critical If ITS Benefits Are to Extend to All Students, Faculty and Staff The Infrastructure Requires Money ($250M) = PROBLEM

8 Getting the Infrastructure Legislature Challenged CSU to Be Creative CSU Was Too Creative Public/private Partnership Made Legislature Uncomfortable Californians Passed a Bond Presidents Decided Infrastructure Had Priority Over New Buildings

9 Getting to “Yes” * Bond Expenditures Require Legislative Approval Legislature Wanted to Tie Investment in Infrastructure to “Learning Outcomes” Not What ITS Is About Infrastructure Enables ITS Initiatives Which, in Turn, Enable Outcomes Apologies to Fisher, Ury and Patton

10 Negotiating Accountability Worked With Legislature – Agreed on What Could Be Measured Developed 10 Year Process for Measures of Success Framework Baseline Changes to the Baseline

11 Types Of Data Collected System Campus Individual (Student, Faculty, Staff) External

12 Scope Of Data Collection 23 Campuses; 1,000 Miles 400,000 Students 35,000 Faculty And Staff 10 Year Commitment Institutional Comparisons Across Time S, F, S Comparisons Across Time

13 Technology User Survey Samples Students: Campus, Class Level, Ethnicity; 23 X 5 X 9 Cells Faculty: Campus, Rank, Discipline; 23 X 4 X 8 Cells Staff: Campus, Job Classification; 23 X 7 Cells

14 CATI LOGISTICS ( COMPUTER ASSISTED TELEPHONE INTERVIEWING ) No Self-select As With Mail About 100 Questions; Minutes Importance Of Skip Facility Instant Database Creation

15 CATI (Continued) Standardized Context Provided (E.G., High Speed Network) Trained Interview Staff, Monitoring, Evaluation Refusals Less Than 2 Percent; Still Over Sample $75 To $100k Per Survey, But Systemwide

16 User Outcomes HardwareSoftwareNetworkSupportTraining ACCESS USE SATISFACTION

17 Examples of Metrics Institutional –Library Cost Avoidance –Smart Classrooms –Data Center Savings –Training and Support Spending Individual –Computer Ownership –Network Use –Help Desk Satisfaction –Use of Administrative Systems

18 Findings and Data

19 Findings HardwareSoftwareNetworkSupportTraining Access Use Satisfaction

20 Hardware Access –Students’ access to hardwareStudents’ access to hardware Use –Percent of Faculty Requiring Computer UsePercent of Faculty Requiring Computer Use Satisfaction –Faculty satisfaction with Teaching in Computer LabsFaculty satisfaction with Teaching in Computer Labs

21 Software Access –Faculty access to softwareFaculty access to software Use –Percent of Students Using Components of Their Student Information SystemPercent of Students Using Components of Their Student Information System Satisfaction –Staff Satisfaction with SoftwareStaff Satisfaction with Software

22 Network Access –Students’ access to wireless networkingStudents’ access to wireless networking Use –Staff use of network from off-campusStaff use of network from off-campus Satisfaction –Staff satisfaction with Staff satisfaction with

23 Student Access to a University-provided Computer

24 Percent of Faculty Requiring Students to Use a Computer

25 Faculty Satisfaction with Giving Instruction in a Computer Lab

26 Faculty Access to University-provided Software Needed

27 Percent of Students Using Their Campus Student Information System

28 Staff Satisfaction with University-provided Software Available

29 Percent of Students with Laptops and Wireless Capability

30 Staff Use of University Network from Off-campus

31 Staff Satisfaction with Services

32 Importance of Providing Electronic Access to Course Instruction Any Time and Place

33 Comparison between Faculty and Students in Perceived Importance of Providing “Any-time, Any-place” Instruction

34 Uses of the Data Accountability Description of the Population of Interest Change Over Time Subgroup Comparisons Cross-group Comparisons

35 Conclusion Expensive – but You Get What You Pay For: Valid Reliable High Level of Confidence Negotiate – Don’t Be Passive – Be Proactive Accountability Provides Cover Infrastructure Is a Utility and a Prerequisite Strategic Planning Is Dynamic – Change/add

36 Look For Yourself Collection/I_Reports_MOS/Measure_of _Success.shtml Collection/I_Reports_MOS/Measure_of _Success.shtml

37 Copies of the Presentation