August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System Please silence cell phones. THANK YOU!

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System Challenges and Advantages of Online Class Evaluations Shohreh Bozorgmehri Manager, Instructional Web Technologies & Briandy Walden Web Developer, Instructional Web Technologies

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 3 TOPICS 1.BACKGROUND 2.CHALLENGES 3.TRANSITION 4.ADVANTAGES & RESULTS 5.FACTS & FIGURES 6.FUTURE 7.CONCLUSION 8.QUESTIONS

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 4 BACKGROUND Paper Online Version 1: Instructor-initiated Version 2: Department Administrative Tool for Evaluations (D.A.T.E.) Inspiration

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 5 BACKGROUND Paper Process Purchase Scantron forms Organize & distribute packets Fill in forms Collect & return forms Scan forms Analyze results Distribute results to instructors

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 6 BACKGROUND Academic Senate Role EEE final evaluations are based on the Campus-wide Teaching Evaluation Form (CTEF) approved by the academic senate committee, the Council on Student Experience (CSE). Senate proposal:

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 7 BACKGROUND Instructor initiated Campus-wide Teaching Evaluation Form (CTEF) Secure and anonymous Fixed one time announcement Automated analysis of results First Online Evaluations

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 8 BACKGROUND Online evaluations had initially been driven by faculty requests. With broader interest, the needs of entire Schools or Departments became the new driving force behind the direction of the online evaluation tool’s evolution.

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 9 BACKGROUND School of Engineering Engineering was the first school to use EEE’s tool school-wide Entire school switched to online Final Evaluations Simple Administrative Web page was developed School’s evaluations fully online by Fall 2003 Learned from experience and incorporated feedback into version 2

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 10 CHALLENGES Significant Culture change Concern of response rate decrease Students would become responsible for taking and submitting evaluations on own time and outside of class Duration changed from 15 minutes inside the classroom to what individual students chose on their own Instructors would need to diligently notify and remind students in order to receive desirable submissions Departments would need to communicate to faculty and/or directly enrolled students ENCOURAGEMENT & COMMUNICATION BECAME VITAL

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 11 CHALLENGES Evaluation requirements: CTEF format Anonymity Department Customizability Cooperation from academic units Willingness to take part in a potentially difficult transition Strong history of good support and cooperation helped convince Departments that their needs would be met, and any unexpected issues would be addressed Instructor & Student participation Notification of students encouragement to participate required for best results

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 12 TRANSITION PROCESS OVERVIEW TOOL FEATURES ACTIVATION ANNOUNCEMENT MONITORING RESULTS WEB INTERFACE STUDENT EXPERIENCE

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 13 TRANSITION Tool Features Departments steered development with requests: Optional addition of four custom questions Scantron-like results file Printable report with page breaks Response rate monitoring tools More department control over instructor options reminders for multiple evaluations combined into one Only non-respondents receive reminders

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 14 TRANSITION Activation Administrators Create and upload CSV files containing instructor IDs and course codes Specify the bounds of instructor control over evaluation windows Process

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 15 TRANSITION Announcement Administrators compose, schedule, and send announcements and/or reminders using a simple online form through the Web interface No limit to the number of announcements sent Only non-respondents do receive reminders Reminder s can have different subject-line or body each time Process

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 16 TRANSITION Monitoring Administrators monitor rate of evaluation submissions Constant monitoring helps department administrators decide when to send announcements or reminder s Process

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 17 TRANSITION Process Results Speed: Administrators, evaluated instructors, and teaching assistants view and download results files in desired format(s) after the “Registrar’s Final Grade Submission Deadline”. Flexible Formats: Comma Separated Value (CSV) Print-friendly report Scantron compatible

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 18 TRANSITION Error check Prevents user errors from adversely affecting the process Instructor abilities Preserves option to allow or disallow instructors from changing evaluation dates/times, or canceling evaluations Retains midterm evaluation options for all instructors, regardless of Department- activated final evaluations The D.A.T.E. Web Interface makes Department activation of final evaluations for multiple instructors easy and efficient. Web Interface

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 19 TRANSITION Online form Requires secure authentication Available anywhere, at any time during the evaluation window Quick and easy multiple choice questions Provides space for typed open-ended responses Incentives “ Respondent List” makes incentives possible Anonymity preserved Names cannot be associated with responses List only available if 5 or more responses received (minimum increasable at department discretion) Student Experience

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 20 ADVANTAGES & RESULTS Save Time: 90% savings common Save Money: $2000/yr for Scantron (PhySci) Add Customizability: Add 4 custom questions Unexpected side-effect: With online evaluations, students have begun to submit more thorough and detailed comments, presumably because they are under no time pressure, and are able to type, rather than handwrite, their comments. Because comments are already typed, they can be distributed and read more easily and quickly. Results are compiled and ready for distribution and review in matter of hours rather than months. Faculty are enabled to powerfully and immediately reflect the evaluation results to the upcoming class. Some departments more rapidly are able to renew or take away TAs or lecturers appointments. DEFINITE IMPACT ON QUALITY OF TEACHING

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 21 ADVANTAGES & RESULTS EEE’s Evaluations tool is the fastest growing tool in the EEE course management system toolbox

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 22 FACTS & FIGURES SurveyMonkey 2.QuestBack 3.Zoomerang 4.Paper Evaluations 5.D.A.T.E. Supported Unsupported Comparison

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 23 FACTS & FIGURES

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 24 FACTS & FIGURES Submissions Classes Engineering Physical Sciences Biological Sciences

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 25 FACTS & FIGURES * Projected Department participation for Fall, 2005

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 26 FACTS & FIGURES * Numbers in parentheses represent percentage of time saved per department

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 27 FUTURE More customizability New survey engine will power tools: Evaluation tool Survey tool Quiz tool More mobility Ability to share Evaluation forms with others Re-use frequently used questions & answers

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 28 CONCLUSION D.A.T.E.’s strength and key success come from the solid partnership our team has created and maintained with a wide range of senior academic faculty, committees, and campus units.

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS) University of California, Irvine Electronic Educational Environment (EEE) UCI’s Course Management System 29 QUESTIONS? For questions, please contact: Shohreh Bozorgmehri Briandy Walden or