1. www.sakaiproject.org 1 Course Evaluation Amitava ‘Babi’ Mitra, MIT Maneesha Aggarwal & Robert Cartolano, Columbia University William Plymale and Aaron Zeckoski, Virginia Tech

2. www.sakaiproject.org 2 Agenda Introduction Review Common Ground Features at Columbia and Virginia Tech Next Steps Q & A

3. www.sakaiproject.org 3 Introduction How did this begin ? –Late Feb 05 began exploring who was using/planning to use online course eval within SEPP –March 05, SEPP Course Eval Working Group set up What’s the WG been doing since then ? –Decided to use Columbia's existing feature set as a baseline to start with –CU and VT prepared a detailed functional layout and work flow, and posted it in April so that we can clearly show how the existing systems might operate. MIT, e.g., has studied this and responded with their own needs –CU and VT have made sandboxes available Why are we doing this ? –Getting to the common ground - the baseline that most of us can agree on

4. www.sakaiproject.org 4 What is Course Evaluation? Administration of survey to students across one or more courses –May span course, department, school, university –Hierarchy, permissions, access levels are important Provide results to University, School, Department, Faculty, Students to evaluate: –Instructors –Curriculum –Academic Program (Program Evaluation) –Department, School, University (accreditation)

5. www.sakaiproject.org 5 Course Evaluation Evaluation has historical aspects at each school There are different motivations for evaluation System must be flexible enough to deal with different needs, but… There are common features that we have identified among our working group members.

6. www.sakaiproject.org 6 Common Features --- from the WG’s discussions Anonymity - student privacy must be preserved Evaluation administrator - someone with appropriate authority must create and administer evaluation. Course Selection - must be very flexible –As little as 1 course… –Or the entire department, program, school, etc.

7. www.sakaiproject.org 7 More Common Features --- from the WG’s discussions Ability to combine core and specific questions –Course-specific, instructor-specific, etc. Requires extensive reporting capabilities –Administrator, Instructors, Students, Public accessible Requires workflow and automation to make it easy, reduce burden to administrators.

8. www.sakaiproject.org 8 Columbia - Current System –In production since Fall 2003 –Built into CourseWorks, campus CMS –Flexible approach needed to support multiple schools, multiple departments, etc. Carrot vs. Stick Open vs. Closed reports Many Reporting types; built-in reports and export data

9. www.sakaiproject.org 9 CU Current System (cont’d.) –E-mail tool very important; both auto and manual –Aggregate data across semesters to support accreditation review –Very popular, well-received by faculty, students, and administrators 87,825 evaluations administered to date 54,480 evaluations completed 157 evaluation templates created

10. www.sakaiproject.org 10 Columbia Live Demo

11. www.sakaiproject.org 11 --- Current System

12. www.sakaiproject.org 12 --- Current System Implemented - 1981 Different forms for different colleges –Business, Architecture, Engineering Statistics –500 class sections per term –230,000 sheets per year Response Rate: 78% during past 3 years

13. www.sakaiproject.org 13 SEPP Contribution – VT / CU Columbia University Partnership –SEPP Conference – Summer, 2004 – Denver –Robert Cartolano / Maneesha Aggarwal Virginia Tech Development –Aaron Zeckoski –Kapil Ahuja / Justin Gawrilow SEPP Course Eval Discussion Group VT / CU Evaluation System Summary VT Evaluation System – “sandbox” release –https://courseeval-dev.cc.vt.eduhttps://courseeval-dev.cc.vt.edu

14. www.sakaiproject.org 14 SEPP Contribution – VT / CU Virginia Tech extensions –Administrative hierarchy (Super, University, School, Dept, Instructor) –Dynamic reporting (flexible formatting, trend analysis, sub-setting) –Ability to incorporate existing data collected with VT’s current evaluation system. –Oracle database tuning

15. www.sakaiproject.org 15 --- Pilot System VT Pilot release – May 2005 –Columbia System with VT extensions –Very limited response to VT pilot study due to late release. –VT pilot study will resume Fall 2005

16. www.sakaiproject.org 16 --- Pilot System Faculty training will be incorporated into FDI Change to online evaluation system needs approval by Deans and Department Heads Online system will provide better access for Institutional Research and the administration Online system offers faculty flexibility in adding questions to their evaluations

17. www.sakaiproject.org 17 --- Demo

18. www.sakaiproject.org 18 --- Current scenario Institutional subject evaluation evaluates between 650-700 subjects each term, about 2/3 of which are undergraduate subjects and 1/3 graduate. This is out of approximately 1700 subjects with students enrolled and 3000 total subjects offered for credit each term. Evaluations are end-of-term only, on 2 paper forms separately for Science/Engineering subjects and for Humanities, Arts, and Social Science subjects, as chosen by the department. Quantitative data are captured and analyzed, written comments are available to each department from originals. Three departments, Electrical Engineering and Computer Science, Management, and Aeronautics and Astronautics (online) have their own forms and systems for subject evaluations.

19. www.sakaiproject.org 19 Needs --- Must haves Must haves for a new system (to replicate current system): Stand alone system (with the capability of being tied to the Course Management System) Anonymity, confidentiality, security Overall institutional administrator, plus levels of school/departmental administrators At least 2 different forms (1 for Science/Engineering subjects and 1 for Humanities, Arts, and Social Sciences subjects) with preference for many Some questions consistent across all schools with flexibility within school/dept./subject. Flexible questions would need to be administered at the local level but available to all. Rating both subject and instructors on the same form Access to individualized reports for each person evaluated Assigning roles to instructors (lecturer, recitation instructor, etc.) Integration with other MIT systems for data downloads (student, subject, and instructor information) and appropriate access (e.g., only to students registered in subject) A 7 point rating scale, with flexibility to change that if needed Blocking access to data/reports until after grades are turned in Analysis and reporting at a centralized point Student access to appropriate report information through a centralized site

20. www.sakaiproject.org 20 Needs --- Necessary Necessary in any new system: Searching data/reports on instructor name Assigning rank/status to instructors (prof./assoc. prof./assist. prof./lecturer/grad. student, etc.) Individual and other reports that include overall subject, department, school, and institutional data Longitudinal data/reports Analysis and reporting at local points in addition to centralized one, with capability to print subject/department/school level with 1 click Linking to evaluation results by department and by subject Linking to evaluation results from subject listings/selection (i.e., our on-line catalog) *Integration with course management system with necessary security Having students go to a single site for all their subjects to be evaluated

21. www.sakaiproject.org 21 Needs --- Nice to have Nice to have capability for: Evaluating sections of a subject (with the capacity for students to enter instructor's name) Instructors' photos together with their name Mid-term evaluations Evaluations that only faculty would have access to Capturing, editing, and distributing students' comments Having a single email sent to a student listing all subjects to be evaluated by that student

22. www.sakaiproject.org 22 What’s Next Create a functional specification for ‘Course Eval ver 1.0’ built upon Columbia base and Virginia Tech extensions Develop Java-based version based on Sakai framework Run as standalone or within Sakai Consider using SAMigo as assessment engine, but probably not in version Course Eval ver 1.0

23. www.sakaiproject.org 23 Open Discussion Questions? See Demonstrations at 5:30 today Join Evaluation Working Group –Contacts: Robert Cartolano rtc@columbia.edurtc@columbia.edu Amitava ‘Babi’ Mitra babi@mit.edubabi@mit.edu