1. Instructor: Ming-puu Chen Presenter : Ching-ting Lin 2007/1/9CSL Lab@ ICE.NTNU1 Considerations for Developing Evaluations of Online Courses Achtemeier, S. D., Morris L. V., & Finnegan C. L. (2003). Considerations for developing evaluations of online courses. Journal of Asynchronous Learning Networks, 7(1), 1-13.

2. 2007/1/9 CSL Lab ICE.NTNU 2 Contents Introduction Literature Review –Seven Principle for Good Practice –Assessment Essentials –Guiding Questions for Planning Assessment –Applying the Seven Principles to the Evaluation Student-faculty contact Cooperation among students Active learning Online Task Research Questions and Design Conclusion

3. 2007/1/9 CSL Lab ICE.NTNU 3 Introduction These questions lead the researchers –Investigate the definitions and principles of effective teaching and learning –Perform a content analysis of instruments –To develop considerations for design of evaluation instruments

4. 2007/1/9 CSL Lab ICE.NTNU 4 Literature Review Described “good” assessment Summarized course evaluation broadly Identified principles specifically for evaluation of online instruction

5. 2007/1/9 CSL Lab ICE.NTNU 5 Seven Principles for Good Practice In undergraduate education Supported by the American Association of Higher Education (AAHE) Good practice in teaching and learning must do the following: –Encourage student-faculty contact 師生互動 –Encourage cooperation among students 合作 –Encourage active learning 主動學習 –Give prompt feedback 立即回饋 –Emphasize time on task 完成任務時間 –Communicate high expectations 高度期待 –Respect diverse talents and ways of learning 因材施教

6. 2007/1/9 CSL Lab ICE.NTNU 6 Assessment Essentials Palomba and Banta address qualities that should lay the foundation for any assessment effort –Assessment should be preceded by explicitly stated outcomes. –Assessment should distinguish between formative and summative uses. –Assessment should have strong faculty buy-in. –Multiple methods should be used. –Assessment results should be shared and used. –The assessment itself should be assessed.

7. 2007/1/9 CSL Lab ICE.NTNU 7 Guiding Questions for Planning Assessment Joseph B. Cuseo in Assessment of the First-Year Experience –Why is the assessment undertaken? Is the focus student or faculty experience? Is it the design of the course? Is it the delivery? –What will change? –What outcomes are being assessed? At what level? –When should the assessment take place? –Where and how should the assessment take place? –Who should be involved?

8. 2007/1/9 CSL Lab ICE.NTNU 8 Applying the Seven Principles to the Evaluation Applying the Seven Principles to the Evaluation of Web Based Distance Education discovered that key strengths in –encouraging active learning, student- faculty contact, and diverse ways of learning –But needed improvement in encouraging cooperation among students and giving prompt feedback

9. 2007/1/9 CSL Lab ICE.NTNU 9 Student-faculty contact Three web-specific requirements of the faculty –Faculty need skill with asynchronous conferencing tools –Faculty clearly and adequately communicate their email response policy to students –Burden on faculty to detect and contact students who are falling behind

10. 2007/1/9 CSL Lab ICE.NTNU 10 Cooperation among students Faculty should begin with –structured activities that facilitate community among students –Develop assignments that require meaningful peer interaction

11. 2007/1/9 CSL Lab ICE.NTNU 11 Active learning Faculty should ask students to present their work electronically Prompt feedback –Faculty should give immediate acknowledgement feedback 立即的確認 –Prompt information feedback regarding the content of the student’s work

12. 2007/1/9 CSL Lab ICE.NTNU 12 Online Task To encourage students’ time on task –Faculty should give structured assignment deadlines –Assignments should require resources that are easily accessible to the online students To communicate high expectation –Faculty provide examples to students of exemplary online performance 網路行為 的模範

13. 2007/1/9 CSL Lab ICE.NTNU 13 Research Questions and Design To examine the above recommendations for teaching and learning online, thirteen evaluation instruments were received Answers were sought to the following research questions: –To what degree do actual evaluation instruments try to assess whether the Seven Principles for Good Practice in Undergraduate Education are taking place? –To what degree are other principles of effective teaching identified in the literature being evaluated by these instruments? –What other issues are being addressed regarding the evaluation of courses? –What other issues are being addressed regarding the evaluation of faculty?

14. 2007/1/9 CSL Lab ICE.NTNU 14 Research Results Table 1. Numbers of institutions whose evaluation instruments assess for the Seven Principles of Good Practice in Undergraduate Education. Table 2. Numbers of institutions whose evaluation instruments assess for the questions suggested in Principles of Effective Teaching in the Online Classroom. Table 3. Numbers of institutions whose evaluation instruments assess these additional questions regarding courses Table 4. Numbers of institutions whose evaluation instruments assess these additional questions regarding instructors

15. 2007/1/9 CSL Lab ICE.NTNU 15 Table 1. Numbers of institutions whose evaluation instruments assess for the Seven Principles of Good Practice in Undergraduate Education.

16. 2007/1/9 CSL Lab ICE.NTNU 16 Table 2. Questions Following a review of the Principles of Effective Teaching in the Online Classroom 1.Were the course goals, learning objectives and outcomes made clear to you at the beginning of the course? 2.Did you have the necessary technological equipment and skills required for this course? 3.Was there adequate technical support if you encountered difficulties? 4.Was the format and page design of the online course easy to use? 5.Were there sufficient instructions given for you to complete all assignments? 6.Did you feel hindered in your online course experience any way? Please describe. 7.Were standards for evaluation of assignments made clear? 8.Did you receive prompt feedback on your completed assignments? 9.Did you participate in online conversations with your instructor during the course? 10.Did you participate in online conversations with your classmates during the course? 11.What learning activities most influenced your learning in this course?

17. 2007/1/9 CSL Lab ICE.NTNU 17 Table 2. Numbers of institutions whose evaluation instruments assess for the questions suggested in Principles of Effective Teaching in the Online Classroom. Important Questions to Ask Online (4) 2-year (2) 4-year (3) Master’s (2) Doctoral (1) Research (1) Percent (of 13) #1 Course goals clearly articulated 22111162% #2 Student has skills & equipment necessary 2123% #4 Format & page design easy to use 31138% #5 Sufficient instructions given for all assignments 18% #11 Satisfaction with learning activities 41211177%

18. 2007/1/9 CSL Lab ICE.NTNU 18 Table 3. Numbers of institutions whose evaluation instruments assess these additional questions regarding courses

19. 2007/1/9 CSL Lab ICE.NTNU 19 Table 4. Numbers of institutions whose evaluation instruments assess these additional questions regarding instructors Other Questions Asked Regarding the Faculty Online (4) 2-year (2) 4-year (3) Master’s (2) Doctoral (1) Research (1) Percent (of 13) 1. Knowledge of subject 323277% 2. Availability for help 423285% 3. Good job answering questions 2212162% 4. Enthusiasm 1212154% 5. Overall rating of the faculty 221146%

20. 2007/1/9 CSL Lab ICE.NTNU 20 Conclusions Seven principles + suggests eleven other questions pertinent –Only 8 of the 18 principles were assessed by those evaluation instruments –Notably missing were questions about cooperation among students and active learning –While one instrument asked about prompt feedback in general, none asked about prompt feedback on completed assignments

21. 2007/1/9 CSL Lab ICE.NTNU 21 END Any advice is appreciated.