1. Choosing Critical Indicators in Online Learning Evaluation Mark Hawkes, Dakota State University Merrill Chandler, University of Illinois American Evaluation Association Annual Conference, November 8, 2001

2. Presentation Objective Discuss online learning evaluation approaches in graduate programs at two universities Identify criteria/indicators suitable for the evaluation of online learning environments

3. Distance Learning Literature Evaluation....... No Yes (79%) (21%) Focus......... Training Education Impact on Learning....... No Yes (87%) (13%) (58%)(42%)

4. Instructor Student Internet Login Interface Content Resources Assessment Communication Modes Support Services System Resources Management and Organizational Information Online Learning Architecture

5. Learner Coordinator DeliveryAssessment Learning Resources Learner Records Catalog info QueryPreferences Performance Interaction Example (Metcalf, Snitzer, Austin, 2001)

6. Familiar Online Learning Evaluation Targets... interface design instructional design student satisfaction technology access faculty satisfaction economic viability departmental capacity interdepartmental collaboration

7. DSU’s Educational Technology Program Students: 36 Credit hour MS program 80% Education; 20% Business/industry 90% Online; 10% On campus Female 68%; Male 32% Project-based curriculum

8. DSU’s ET Environment Pervasive technological culture Consistency between program goals and the state/region-wide initiatives Campus-wide faculty support Institutional experience in Web-based instruction delivery Multi-delivery methods Client: teachers, teacher developers, trainers, technology coordinators, etc. Predominantly web-based delivery

9. An Evaluation Model... Illuminative Operation of Components And Subcomponents Integrative Holistic perspective on The learning experience Course & Program Design Components Infra- structure /System Work Flow InteractionImpact ProcessImpact Observing and Detecting Focused on Performance Functional Problems Outcomes

10. Infrastructure/System Input/output devices Network speed and connectivity Network design/Topology Technical support systems and maintenance

11. Course and Program Design Nature of the Design Situation Based Role of State and National Standards Sequencing/Instructional Strategies Assessment Motivation: Learning vs. Performance Visualization Tools and Media User Interface Course Management

12. Work Flow Use of discussion tools Software usage Message redundancy (audio, video, web pages, emails). Progression Do learners progress through their work tasks in a linear fashion? (novice-like) Nonlinear opportunistic fashion (expert- like)

14. Interaction Social and instructional Must account for all of the following relationships: InstructorLearner Learners Content Learners Technology Content Technology

15. Online Course Interaction Announcements Email Discussion Board Synchronous text chat Desktop Video File Loading Online assessment Audio/video clips Room-based Video

16. Impact Course performance Collaborative learning Retention/attrition (course and program) Professional relevance and utility Learner productivity

17. Evaluation Attributes Multi-sourced data (students, server log files, etc) Internal and external Performance based Comparison and criterion based

21. The breadth of this course was: 1234567 Not nearly enough The right amount Way too much Compared to a traditional course 1234567 A much narrower range About the same range A much wider range of of material was covered of material was covered material was covered Online: 4.61 Compared to traditional 4.65n=32

22. The depth of this course was: 1234567 Not nearly enough The right amount Way too much Compared to a traditional course 1234567 Material was covered in Material was covered in Material was covered in much less depth about the same depth much more depth Online: 4.48 Compared to traditional 4.42n=32

23. The extent of critical thinking required: 1234567 Not nearly enough The right amount Way too much Compared to a traditional course 1234567 Much less About the same Much more Online: 4.61 Compared to traditional 4.94n=32

24. The amount of effort put into the course: 1234567 Much less About the same Much more Compared to a traditional course 1234567 Much less About the same Much more Online: 5.65 Compared to traditional: 5.26n=32

25. U of I’s Curriculum Technology and Education Reform (CTER) Master of Education (Ed.M.) For practicing K-12 teachers and administrators A two-year program Eight online courses Project based

26. CTER... Program is in its fourth year CTER cohorts 1 & 2 have graduated CTER 3 cohort has 26 students CTER 4 has cohort 25 students Female 73%; Male 27% Many students have technology responsibilities for their schools or districts

27. CTER’s synchronous and asynchronous technologies WebBoard conferencing Streaming media using Real Player Audio narrated PowerPoint presentations Tapped In CTER Base iVisit RogerWilco Interactive Multimedia Paper

28. CTER evaluation Mostly formative Mixed methods Course evaluation Program evaluation Mini-case studies

29. Course Evaluation Instructor and Course Evaluation System (ICES) Piloting Evaluation Online (EON) CTER course survey using SurveyIt Instructor Technology use Support Exemplary student projects

30. Program Evaluation Program surveys Application skills Web browser skills Learner profile Student interviews Collection of student artifacts Mini-case studies

31. CTER studies identify five dimensions of effective learning: Relevant and challenging assignments Providing adequate and timely feedback through teacher-student interaction Flexibility in teaching and learning Constructing coordinated learning environments Constructing rich environments for student to student interaction

32. Indicators of CTER effectiveness Low dropout rate Student satisfaction Student learning transferred into practice

33. Typical Problems with Online Courses Facilitating and encouraging collaboration Time management Student proficiency with course tools Ambiguous directions Timeliness of feedback

34. Factors Beyond ID Control Student sophistication with technology tools System capacity Learner availability/accessibility Enthusiastic, responsive instructor Good learner support Motivated learners

35. How to Design and Effective Online Course? Follow basic ID principals Build a climate of disclosure and full participation Institute informal student evaluation and check-in mechanisms Active and intensive instructor participation Build in as much interactivity as possible Create visually interesting screens/pages Ensure instructions are very clear Multi-mode interaction is critical

36. Slides at: www.homepages.dsu.edu/hawkesm/