1. 1 June 10-15, 2012 Growing Community; Growing Possibilities Switching to on-line evaluations for courses at UC Berkeley Daphne Ogle, Lead Design, UC Berkeley Aaron Zeckoski, Lead Developer, Unicon

2. 2 2012 Jasig Sakai Conference Speakers  Daphne Ogle  Berkeley  Design Lead  OAE since 2010, Fluid Project, Sakai since 2004  Aaron Zeckoski  Unicon  Development Lead  Evaluation System original lead (2006), Evals software since 2001, Sakai since 2004

3. 3 2012 Jasig Sakai Conference Unicon Profile

4. 4 2012 Jasig Sakai Conference Unicon technology & domain expertise

5. 5 2012 Jasig Sakai Conference Unicon service capabilities / offerings

6. 6 2012 Jasig Sakai Conference  Intro to UC Berkeley on-line course evaluation project  Demo: Updates to CLE Evaluation System for Spring pilot  Future work  What we learned  Questions & Answers Today’s topics

7. 7 2012 Jasig Sakai Conference Introduction to online evaluations project at UC Berkeley Overall project initiative

8. 8 2012 Jasig Sakai Conference Online evaluations for courses  New shared questions ◦ campus-wide ◦ instructional format-based (14 formats) ◦ piloted first  On-line ◦ pilot CLE Evaluation System ◦ development to meet Berkeley needs & overall system improvement ◦ Berkeley & Unicon collaboration

9. 9 2012 Jasig Sakai Conference Goals of initiative  Rapid access to useful information to help faculty improve their courses and their teaching effectiveness  Improvements to the quality & integrity of data that the campus uses to understand and recognize teaching contributions  21st century tools for staff to perform their jobs efficiently and free up time for direct service to students and faculty  High quality information for students about courses and instructors at Berkeley Cynthia Schrager, Assistant Vice Provost

10. 10 2012 Jasig Sakai Conference Project time line Summer 2010 Spring 2011 Summer 2011 Fall 2011 Spring 2012 Summer 2012 Fall 2012 Research new questions & instructional formats Evaluation systems comparative analysis Design, analysis & development CLE eval sys (in UCB branch) Campus buy-in Fall 2010 Pilot new questions & instructional formats Pilot CLE eval sys Assess pilot Design & development (intend to move to trunk) Larger 2nd pilot

11. 11 2012 Jasig Sakai Conference Spring Pilot Introduction to online evaluations project at UC Berkeley

12. 12 2012 Jasig Sakai Conference Goals of spring pilot  Establish baseline response rate  Assess CLE evaluation system ◦ technical longterm sustainability ◦ user experience  Build knowledge of CLE evaluation system  Build knowledge of infrastructure needs

13. 13 2012 Jasig Sakai Conference Spring pilot success criteria  Establish baseline response rate  Collect data on function & experience  User experience meets or exceeds previous  Transparency across campus  Elicit higher quality response  Gain understanding to inform Fall design & development effort

14. 14 2012 Jasig Sakai Conference Spring pilot makeup  8 departments  61 courses (primary & secondary)  2300+ enrollments

15. 15 2012 Jasig Sakai Conference Spring pilot Demo

16. 16 2012 Jasig Sakai Conference Spring pilot development  Significant time understanding system  Overall model & process to follow ◦ “uber” template ◦ self-service  Focus on student & instructor experience  System areas of focus ◦ Dashboard ◦ Evaluation presentation ◦ Administration page ◦ Notifications

17. 17 2012 Jasig Sakai Conference Dashboard - student

18. 18 2012 Jasig Sakai Conference Dashboard - administrator

19. 19 2012 Jasig Sakai Conference Evaluation

20. 20 2012 Jasig Sakai Conference Administration

21. 21 2012 Jasig Sakai Conference Notifications

22. 22 2012 Jasig Sakai Conference Bulk import / export configurations

23. 23 2012 Jasig Sakai Conference Live Demo http://berkeley-eval.unicon.net/portal

24. 24 2012 Jasig Sakai Conference Lessons learned Food for thought

25. 25 2012 Jasig Sakai Conference Lessons learned  Configurability + flexibility = Complexity ◦ documentation not always up to date ◦ allow significant time to “play around”  Unfinished functionality ◦ email notifications ◦ options in evaluation creation  Global tool ◦ acts different than other tools ◦ where will it be exposed?

26. 26 2012 Jasig Sakai Conference Lessons learned - cont’d  Allow a lot of time for QA ◦ see 1st bullet: lots of variables ◦ QA’ing up to the last minute  Clean data required ◦ setup if not using group provider ◦ site roles NOT always evaluation roles  Communicate! ◦ be transparent ◦ remind users it’s a pilot

27. 27 2012 Jasig Sakai Conference Lessons learned - cont’d  Learn from other implementors ◦ document models, workflows & implications ◦ community project?  Culture & legal requirements vary across campus ◦ viewing respondents ◦ roles, permissions, viewing each others evaluations

28. 28 2012 Jasig Sakai Conference Future plans Evaluation system at Berkeley

29. 29 2012 Jasig Sakai Conference What’s next (1 of 2) ?  Feedback sessions with users on Spring pilot  Needs analysis around results  Move our work back into trunk  Configuration option to hide respondents page  UX improvements ◦ results page ◦ multiple instructors (Berkeley-centric) ◦ add respondent link to dashboard ◦ make display options clear in context ◦ configuration page (information architecture)

30. 30 2012 Jasig Sakai Conference What else is next?  Investigation ◦ support for instructional formats (Berkeley-centric) ◦ give department admins access to results ◦ Evalsys tool added to My Workspace automatically ◦ group provider & SIS integration ◦ partial OAE integration

31. 31 2012 Jasig Sakai Conference Results (early idea)

32. 32 2012 Jasig Sakai Conference Results(early idea)

33. 33 2012 Jasig Sakai Conference Sneak peak: dashboard

34. 34 2012 Jasig Sakai Conference Sneak peak: scale creation

35. 35 2012 Jasig Sakai Conference Sneak peak: scale creation

36. 36 2012 Jasig Sakai Conference Resources  Berkeley Confluence ◦ https://confluence.media.berkeley.edu/confluence/display/EVAL/Home https://confluence.media.berkeley.edu/confluence/display/EVAL/Home  Sakai Confluence ◦ https://confluence.sakaiproject.org/display/EVALSYS/Home https://confluence.sakaiproject.org/display/EVALSYS/Home  Sakai Jira ◦ https://jira.sakaiproject.org/browse/EVALSYS https://jira.sakaiproject.org/browse/EVALSYS

37. 37 2012 Jasig Sakai Conference Thank you!  Questions, comments, discussion...