1. Evaluating Distance Learning: Feedback from Distressed Students Christopher Essex Kursat Cagiltay Indiana University AECT National Convention Denver, Colorado October 27, 2000

2. Our Background in Online DE Essex has worked for the Indiana University SOE Distance Education Program since 1994 Essex has taught online since Summer 1998 Cagiltay taught Turkish students online from USA

3. Outline Definition of Distress Study Background Course and Student Profile Method Findings: Positive and Negative Feedback Recommendations Limitations

4. Online Distance Education Student Distress Key References Hara, N. and R. Kling (in press). Students' distress with a web-based distance education course. Information, Communication and Society. (Earlier draft is on-line: Available: http://www.slis.indiana.edu/CSI/ wp99_01.html) Mendels, P. (Sept. 22, 1999). Study Finds Problems With Web Class. New York Times Online Edition. (On-line). Available: http://www.nytimes.com/library/tech/99/09/cyber/education/22education.html Gale, C. (2000, January). Online learning: A student perspective. Syllabus. pp. 52-53

5. Online Distance Education Student Distress Distress: situations that the students...find particularly troublesome. –Social Isolation –Overwhelming Email Communication –Lack of Instructor Feedback –Technical Problems –Ambiguous Instructions –(Hara and Kling)

6. Study Background The Course –Focused on the evaluation of Internet resources for use by K-12 educators and students Course activities –Required students to navigate through various websites, the course website along with a number of others, send and respond to email messages, and use a web-based conferencing tool for class discussion purposes Instructors role/pedagogy –Facilitator, guided students through PBL-based activities and moderated class discussion

7. Study Background Our role as evaluators: –The researchers conducted the study from a third-party perspective, observing impartially and not involved in making any changes to the program based on our evaluation.

8. Course Profile 100% online course, utilizing course website, email and asynchronous web-based conferencing Masters-level, three-credit hour course Designed and taught by advanced doctoral student Part of an online Masters degree program

9. Student Profile Of 11 registered students, 9 responded to at least one of various survey instruments. Seven students filled out all the instruments 100% graduate-level students Students were located in Indiana, Hawaii, Iowa, Tennessee, North Dakota, and Sweden 4 in on-campus degree program 1 in online masters degree program 10 enrolled for educational and business-related reasons--Only 1 registered for personal reasons

10. Methods Interviewed instructor Reviewed course content/activities Developed questions based on Kirkpatricks Four Levels of Evaluation Developed online survey instruments Integrated surveys into course website

11. Focus of Kirkpatrick Levels Level 1 (Student satisfaction) –Course materials –Course activities –Instructor performance –Overall rating for course Level 2 (Student learning) Level 3 (Transfer of learning) Level 4 (Cost/benefit impact)

12. Positive Student Feedback Level 1: Overall, students gave positive comments regarding this course. Level 2: The students reported a moderate level of learning. Level 3: Students expected professional benefits in the future from taking the course. Level 4: Students responded that the course cost them more money than on-campus course, but saved them time. The majority of the students felt that the cost/benefit ratio of the course was very favorable.

13. Negative Feedback: Distress Level 1: Many students were not very satisfied with their interactions with the instructor. Level 2: A small number of students were not well-prepared for the technological requirements of this course, which caused them frustration. Level 4: Students responded the course cost them more money than an on- campus course.

14. Negative Feedback: Distress Levels 1, 2, 3: –One student, in a moment of distress, made the following statement: "I am totally frustrated. I absolutely do not know how this class is organized and how to access the information I need. I hate Long Distance education and I never plan to do this ever again. It has made me rethink even using the Internet in my class at school. I hate this. I hate this. I hate this."

15. Distress: Comparison to Hara and Kling Distress: situations that the students...find particularly troublesome. –Social Isolation (not found) –Overwhelming Email Communication (not found) –Lack of Instructor Feedback (found) –Technical Problems (found) –Ambiguous Instructions (found) (Hara and Kling)

16. Recommendations for DE Instructors and Developers The instructor should review her practices in responding to students email and web-based conferencing posts, to ensure that she is providing sufficient and appropriate feedback. The instructor should specify the technological requirements in the syllabus, and arrange for technical support. The instructor should conduct a usability tests on the course requirements and other instructions to ensure that they are clear and non-ambiguous. The distance education program should provide an on- campus face-to-face orientation. For students that cannot be on-campus, a video tape should be provided.

17. Limitations Small sample size (though entire class) Education students may be atypical Lack of follow-up evaluation Lack of ability to measure Level 3 (Transfer of Learning)

18. Questions? Christopher Essex cessex@indiana.edu Kursat Cagiltay kursat@indiana.edu Presentation Online at: http://php.indiana.edu/~kursat/ evalde-aect-denver.ppt