1. Evaluating Online Courses

2. What Is Evaluation? Per Michael Scriven (1991): Process of determining the merit, worth, or value of something (Process that) leads to evaluative conclusions good/bad works/ doesnt work high/low quality

3. Course Evaluation Is the course working as intended? Does the course meet its objectives? Is the online learning environment as effective as it can be? Is the course manageable? Are learners satisfied?

4. Why Evaluate? To improve instructional quality To motivate learners To motivate instructors * Scriven, M. (1991). Evaluation Thesaurus, 4 th Ed. Sage Publications: London

5. Conducted by Administration vs. Conducted by Instructor Administration-Led Focuses on learner satisfaction & broader issues of program Access, Course Management System University service – registration, instructor Minimal instructor feedback Instructor-Led Focuses on course components/design Student feedback & performance Instructor reflection

6. Dick and Carey Design Model Goals Needs. Anal. Per Obj. Revise Assmt. Instr. Inst. Strat. Inst. Mtls. Form Eval. Summ Eval. Lnr. & Context

7. Gerlach and Ely Design Model Specify Content Specify Object. Assess Entry Behavior Determine Strategies Org. Groups Allocate Time Allocate Space Select Resources Evaluate Perform. Analyze Feedback

8. In Reality, Evaluation is…. Done continuously once the course begins.

9. Recommendation Formative course evaluation conducted by the Instructor.

10. It needs to be… Informal and Reflective, as opposed to Reactive.

11. Interim vs. Terminal Interim Evaluating at various points in the course Not TOO many points 2-3 times per semester, mid-term at the minimum May affect future weeks of course Terminal Conducted at the end of the course Affects next course offering May get more input on interim evaluations than terminal evaluation

12. Evaluation Framework

13. AEIOU – Course Evaluation (Based on AEIOU Program Evaluation Method) A – Access – Was the course easily and consistently accessible? E – Effectiveness - How well was the course designed/organized? Objectives & instructions clear? Assessment matched objectives? Level of student performance? I – Interaction – Was there a good balance of interaction? Level of learner participation in discussions? Level of peer support, instructor support? O – Obstacles – What hindered the potential effectiveness of the course? U – Unanticipated Consequences – What happened that was not expected --- both from learner and instructor perspective?

14. Evaluation Considerations Environment Course structure Access reliability Instructional Strategy Objectives Activities Interaction/Communication Assessment

15. Evaluation Considerations Learner Performance Motivation Satisfaction Instructor Performance Motivation Satisfaction

16. Better to Design the Quality In Than to evaluate the problems out

17. Communication Tools for Evaluation Email Survey attached Quick/one question surveys Online surveys Quick feedback on a new technique – What did you think? Check the box.. Discussion board (evaluation summaries) Fax For handwriters

18. Encouraging Completion – Interim Make it as short/easy as possible Students need to trust interim wont affect their grade – stay away from instructor components (i.e. feedback) Evals could be submitted to peer/TA who summarizes and forwards to instructor or posts on course site Could be incremental questioning Every student to respond at least once

19. Encouraging Completion - Final One peer holds evals until after final grades are submitted Could be team leaders Make it the final assignment and student would have to turn it in (to peer/TA) before final grade was submitted

20. Summary Clear, easy, non-threatening – to maximize feedback Focus on key components of your course and key participants in the course Include YOUR thoughts as well as your students