HERDING JAVELINAS : CoursEval Implementation at a Large Multi-Campus University

INTRODUCTION Arizona State University, with more than 70,000 students in 15 colleges, has adopted CoursEval as our sole university-supported course evaluation system. Over the course of a single academic year, we expanded the use of CoursEval from two colleges to nearly all online and on-ground courses.

INTRODUCTION We will describe how we: achieved buy-in from administration, faculty, staff, and students; retired our legacy paper and online evaluation systems; balanced decentralized college-level ownership with centralized facilitation, and saw considerable cost-savings, with no major problems.

HERDING JAVELINAS Paper evaluations were used for in-person classes for most of ASU Strengths Units had control over survey design and content Familiar system, little training involved Weaknesses Required considerable consumption of paper, printing, and personnel time o ASU commitment to reduce our carbon footprint Less than ideal for data retention, historical tracking, etc.

HERDING JAVELINAS Online Evaluation Legacy System Strengths: Centralized administration Reporting convenient for units

Legacy System Weaknesses 1.Used separate systems for survey administration and analysis/reporting. 2.Single survey for all units - did not meet needs of some (and did not match survey in use for on-ground sections). This was an issue for promotion/tenure. 3.No comparative/historical data options.

Legacy System Weaknesses (contd) 4.Access control required SQL programming by Associate Director to update database. Time consuming and could not delegate to staff. 5.Not available for on-ground courses, despite considerable demand from units. 6.Limited to single administration per traditional semester. Did not accommodate non-traditional course schedules. 7.Data analysis and reporting were time-consuming. Reports not available for several weeks after close of survey.

Our Requirements: a.Online surveys for both online and in- person courses b.Units can use their own survey(s) c.Flexible administration schedules, controlled by units d.Results available immediately e.Release of reports (and report content) determined by units SEARCHING FOR A SOLUTION

Our Requirements (contd) : f.Instructors, chairs, & dept. support staff have access to results g.Reports available online for both current & future access/review h.Flexible reporting format i.Support comparisons across instructors & across time j.Support co-taught sections SEARCHING FOR A SOLUTION

We explored these options : a.Re-design homegrown legacy system b.Use one of the homegrown systems in use by ASUs Engineering or Business schools c.Identify a commercial system 10

DECISION: PURCHASE COMMERCIAL SYSTEM Engineering & Business systems could not be easily modified Re-design of legacy system would require considerable work by developers Mary Lou Fulton Teachers College used CoursEval for years with good results. They did demo for us, and we were impressed with features

PILOT TEST Attended Sept Courseval User Group meeting to determine whether CoursEval would meet ASUs needs Worked with Brian Hopewell during the meeting to negotiate a price for a limited pilot

Spring 2011 CoursEval available for all online and in- person sections Participation is still voluntary, but most made transition

TRANSITION SPRING 2010 Scanned 119,038 Surveys SPRING 2011 Scanned 49,336 Surveys

TRAINING Fall 2010 a.Trained one user in each college to create survey items & responses, and assemble surveys. After surveys closed, trained them to design and release reports. b.Training scheduled in one on one sessions - via WebEx, or in-person. c.Coached users through cleaning course/instructor data from PeopleSoft before our staff imported into CoursEval d.I handled entire administration: s, reminders, & questions from faculty & students

DISTRIBUTING RESPONSIBILITIES Winter 2010 I transferred responsibility for s to college users – sending them detailed instructions (including screenshots) I continued to manage PeopleSoft queries and data imports

Spring 2011 I was instructed to fully transfer responsibility to the units Several users realized that there was too much work for one person, and wanted me to train others in their areas DISTRIBUTING RESPONSIBILITIES

SPRING 2011 CoursEval agreed to send Michele to ASU for a two-day Train the Trainer workshop. Approximately 40 users from across the university registered for the workshop. Michele trained them on: Creating items & responses Building surveys Data imports Survey administration Reports

SPRING TRAINING We also taught the group how to run the PeopleSoft query for course, instructor, and student data, how to clean the data, and how to do imports. We obtained approval for me to send a common e- mail invitation to students to reduce the number of student s. Established the ASU CoursEval User Group, so that users from across the university can support one another.

InvitationsResponsesRate Spring ,99814,04147% Spring ,54375,70154%

SPRING 2011 We thought we had our pigs in a row! But then…

SPRING 2011 Shortly after the spring training, we lost several of our power users!

SUMMER 2011 More Challenges Some trained users left on vacation during open surveys More staff turnover Proposed Solutions Train more people? Return to centralized model?

FALL 2011 Stay Tuned … We are in discussions with ASU leadership about how to balance unit-level autonomy with centralized support and continuity.

Wanda K. Baker Associate Director for Assessment & Information Technology University Office of Evaluation & Educational Effectiveness Arizona State University Tempe, AZ