Copyright 2006 © Lycurgus Group, Inc. Your Logo Here Evaluation Equals Excellence Presented by: Donna DiMatteo, Ph.D. DLA 2006 Conference 2:00 PM on Tuesday, June 6, 2006 Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Agenda Why evaluate? What is the importance of evaluation in distance education? Evaluation methods Describing what is involved with a systematic process of measuring learning outcomes and institutional effectiveness Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Why Evaluate? It is essential that there is a clearly delineated method of evaluating performance and to assess if goals are being achieved Great vehicle to communicate quality standards Evaluation is required for satisfaction, retention and accreditation reasons. Copyright 2006 © Lycurgus Group, Inc.

Necessity of Evaluation Determination if course objectives are being met Identification of whether students and faculty are satisfied with the course structure Provides the means of celebrating what is going well Identification of changes that need to be made with courses and programs Copyright 2006 © Lycurgus Group, Inc.

Key Requirements in Evaluation Should be tailored to individual needs Creating an evaluation system that is linked to measuring quality standards and best practices in distance education Identifying strengths and areas of improvement Using a systematic process of measuring learning outcomes and institutional effectiveness. Involving key stakeholders in evaluating performance Process is continuous Copyright 2006 © Lycurgus Group, Inc.

for Systematic Evaluation Evaluation Methods Traditional Methods for Systematic Evaluation Basic Paradigms: Business Decision Support Quality Instructional Design Formative Feedback Summative Feedback Copyright 2006 © Lycurgus Group, Inc.

Evaluation and Effective Learning Recent research has indicated how important both formative and summative evaluations are to effective learning Copyright 2006 © Lycurgus Group, Inc.

Traditional Evaluation Methods Formative=Ongoing Institutional Effectiveness Formative evaluations can be accomplished through: obtaining feedback from students and faculty on their satisfaction with online learning. Improvement of programs Evaluate strengths and areas of concern Summative=Completion Course Effectiveness Synthesis of data to make decisions on course improvements Summative evaluations Are important to be able to improve the course design to facilitate learning in a more effective way. Evaluate course objectives, experience and facilitation style Improvement of programs Form (process) and summative (outcomes) Formative evaluation: Is an on-going process to be considered at all stages of instruction. Will enable the instructor to improve the course as he/she proceeds. Facilitates course and content adaptation. Will identify major gaps in the instructional plan or the need for minor adjustments. Summative evaluation: Assesses overall effectiveness of the finished product or course. Can be a springboard in developing a revision plan. Can be a baseline of information for designing a new plan, program, or course. Will not help current students since it is conducted upon course completion. Some questions that educators may want to ask students when collecting summative data include: List five weaknesses of the course. List three (or five) strengths of the course. Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Proactive Evaluation Focus on Quality Standards Formative/Summative Evaluation Formative/Summative Evaluation Best Practices in Online Learning Kirkpatrick’s Evaluation Model Copyright 2006 © Lycurgus Group, Inc. $16,200/year

Copyright 2006 © Lycurgus Group, Inc. Proactive Evaluation Ensures that all critical areas are addressed in an evaluation process. Planning and design activities will not only ensure that all aspects of creative online learning activities and environments are addressed, but that subsequent formative and summative evaluation will be more directed and meaningful Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Proactive Evaluation Proactive evaluation is the foundation for the methodology that will be proposed Thus, an evaluation system that provides a comprehensive and systematic view of institutional effectiveness supports this model and includes feedback from multiple sources. Copyright 2006 © Lycurgus Group, Inc.

Kirkpatrick’s Four-Level Model Evaluation model focused on results Level 1: Evaluate reaction: Instruction/satisfaction Level 2: Evaluate learning: KSAs Level 3: Evaluate behavior: Application Level 4: Evaluate results: Impact on organization (Schankman, 2004) Level 5 added: ROI Missing: Fails to consider necessary improvements Considering the importance of motivation Level 1: Evaluate reaction to instruction/Satisfaction-survey Level 2: Evaluate learning-survey Level 3: Evaluate behavior-survey Were performance objectives achieved? Level 4: Evaluate results/impact on organization (Schankman, 2004)-Interview/performance data Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Quality Quality standards are important to consider in an evaluation plan Quality framework: enables institutions to set goals, measure progress in reaching goals and implement required changes The focus on quality and best practices are the basis of our evaluation system. Applied to each individual institution uniquely by identifying needs/goals Following organizations identified quality standards: American Council of Education, American Distance Education Consortium, Sloan-C, Institute for Higher Education Copyright 2006 © Lycurgus Group, Inc.

Instructional Effectiveness Process Taken from a systematic view of change and development, quality initiatives, best practices and proactive evaluation, a systematic evaluation model will be described In order to promote a comprehensive and proactive evaluation system, students, faculty and staff should be involved in providing feedback. Key areas have been identified to be important for student success. Identified in connection with research on best practices in online learning. Developing this type of evaluation system will ensure that the feedback obtained will be useful for both faculty and school administrators. Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Methods Can use a variety of methods from surveys, interviews and focus groups Focus groups can be informal way to identify what is working and what can be improved Problem is that viewpoint is from a few participants Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Factors to Evaluate A number of different factors have been identified for evaluation purposes Based on best practices/quality in online instruction Copyright 2006 © Lycurgus Group, Inc.

How Factors are Identified Expectations of success are to be communicated Online learning should encourage interactivity between faculty and students as well as communication among students. Being learner-focused Best Practices: Course design should be engaging Courses should be geared to different styles of learning Active learning and critical thinking both must be encouraged It is imperative that instructors give prompt and substantive feedback. Copyright 2006 © Lycurgus Group, Inc.

Evaluation System: Based on Quality standards in Online Education Learning Effectiveness Student Satisfaction Faculty Satisfaction Cost Effectiveness Main Measure: End of Course Survey Also, can use focus groups and interviews Student Survey on Institutional Effectiveness Faculty Survey of online learning experience/Instructional Effectiveness *Goal is to cut or control costs *Reduce teacher turnover, lower training costs and improve technologies *Increase retention *Number of students obtaining degrees -Good to develop dashboard of results as a communication tool -important to utilize internal and external benchmarks n Copyright 2006 © Lycurgus Group, Inc.

Factors to Evaluate End of Course Survey The factors that should be evaluated in a comprehensive system include: Course content Course outcomes/objectives Technology Resources/Student support Interactivity (with peers and instructor) Feedback Barriers to learning Class engagement/Motivation Instructional approaches/Addresses different learning styles General Satisfaction Most valuable aspects of course Areas of improvement Copyright 2006 © Lycurgus Group, Inc.

Factors to Evaluate Faculty Survey Course Feedback Evaluation of learning/university Faculty Concerns What is going well What can be improved upon Experience with Students Incentives/Development opportunities Faculty Time/Workload Copyright 2006 © Lycurgus Group, Inc.

Factors to Evaluate Institutional Effectiveness University commitment to students and quality education Fairness in Evaluation/Adequacy of Feedback Support/Resources Received Instructor/Student interaction Overall experience with classes and university services Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Survey Items These factors should then be broken into specify survey items that measure each dimension. Also, different surveys should be developed for student, staff and instructor feedback. Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Online Surveys Surveys delivered via email Ease in administration and analysis Branching of data is not apparent to participants Web-based deployment Ensures confidentiality Provides quick turnaround of information Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Other Important Data Retention Data Feedback from graduates Document examination (handbooks, orientation, strategy, mission/vision) History of changes made to courses Copyright 2006 © Lycurgus Group, Inc.

Lycurgus Instructional Effectiveness Process Step 1: Determine goals, objectives and standards of excellence. Step 2: Identify internal benchmarks. (These benchmarks are used to compare to future evaluations and to compare with other schools.) Step 3: Collect data from stakeholders. Step 4: Report results and make recommendations. Step 5: Identify top areas for planned improvements Step 6: Implement changes. Step 7: Reevaluate. Copyright 2006 © Lycurgus Group, Inc.

Lycurgus Instructional Effectiveness Process Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Summary Utilizing this comprehensive evaluation process will ensure that key critical areas will be identified for improvement and planned change can take place. This is important for student and faculty satisfaction as well as overall institutional effectiveness. Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Q & A Contact: Donna DiMatteo, Ph.D. Lycurgus Group 310-543-0909 Donna DiMatteo, Ph.D. is an Online Instructor and also Managing Director of Consulting Services for The Lycurgus Group, Inc. 1016 Vincent Street, Redondo Beach, CA 90277, Email: ddimatteo@lycurgusgroup.com www.lycurgusgroup.com Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. References Cheung, D. (1998) Developing a student evaluation instrument for distance teaching. Distance Education, 19 (1), 23-43. DeVellis, R. F. (2003). Scale development: Theory and applications. Thousand Oaks, California: Sage Publications. Dominguez, P.S. & Ridley, D. (Sept. 1999) Reassessing the assessment of distance education courses. T.H.E. Journal, 27 (2), 70-74. Kaufman, R., Guerra, I., Platt, W.A. (2006). Practical evaluation for educators. Thousand Oaks, California: Sage Publications. Lockee, B., Moore, M., & Burton, J. (2002). Measuring success: Evaluation strategies for distance education. Educause Quarterly, 1, 20-26. Palloff, R. M. & Pratt, K. (2003). The virtual student: A profile and guide to working with online learners. San Francisco, California: Jossey-Bass. Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. References Reisetter, M. & Boris, G. (Winter 2004). What works: Student perceptions of effective elements in online learning. Quarterly Review of Distance Education, 5 (4), 277-293. Schankman, L.H. (2004). Holistic Evaluation of an Academic Online Program. Retrieved May 3, 2006, from http://library.mansfield.edu/larry/evalplan.pdf Sims, R., Dobbs, G., & Hand, T. (Oct. 2002) Enhancing quality in online learning: Scaffolding planning and design through proactive evaluation, Distance Education, 23 (2), 135-149. Stewart, B.L., Waight, C.L., Norwood, M.M., & Ezell, S.D. (Summer 2004) Formative and summative evaluation of online courses. Quarterly Review of Distance Education, 5 (2), 101-110. Strong, J. H. ( 2006). Evaluating teaching. Thousand Oaks, California: Sage Publications. Copyright 2006 © Lycurgus Group, Inc.