1. Copyright 2006 © Lycurgus Group, Inc.
Your Logo Here
Evaluation Equals Excellence
Presented by:
Donna DiMatteo, Ph.D.
DLA 2006 Conference
2:00 PM on Tuesday, June 6, 2006
Copyright 2006 © Lycurgus Group, Inc.

2. Copyright 2006 © Lycurgus Group, Inc.
Agenda
Why evaluate?
What is the importance of evaluation in distance education?
Evaluation methods
Describing what is involved with a systematic process of measuring learning outcomes and institutional effectiveness
Copyright 2006 © Lycurgus Group, Inc.

3. Copyright 2006 © Lycurgus Group, Inc.
Why Evaluate?
It is essential that there is a clearly delineated method of evaluating performance and to assess if goals are being achieved
Great vehicle to communicate quality standards
Evaluation is required for satisfaction, retention and accreditation reasons.
Copyright 2006 © Lycurgus Group, Inc.

4. Necessity of Evaluation
Determination if course objectives are being met
Identification of whether students and faculty are satisfied with the course structure
Provides the means of celebrating what is going well
Identification of changes that need to be made with courses and programs
Copyright 2006 © Lycurgus Group, Inc.

5. Key Requirements in Evaluation
Should be tailored to individual needs
Creating an evaluation system that is linked to measuring quality standards and best practices in distance education
Identifying strengths and areas of improvement
Using a systematic process of measuring learning outcomes and institutional effectiveness.
Involving key stakeholders in evaluating performance
Process is continuous
Copyright 2006 © Lycurgus Group, Inc.

6. for Systematic Evaluation
Evaluation Methods
Traditional Methods
for Systematic Evaluation
Basic Paradigms:
Business Decision Support
Quality
Instructional Design
Formative
Feedback
Summative
Feedback
Copyright 2006 © Lycurgus Group, Inc.

7. Evaluation and Effective Learning
Recent research has indicated how important both formative and summative evaluations are to effective learning
Copyright 2006 © Lycurgus Group, Inc.

8. Traditional Evaluation Methods
Formative=Ongoing
Institutional Effectiveness
Formative evaluations can be accomplished through:
obtaining feedback from students and faculty on their satisfaction with online learning.
Improvement of programs
Evaluate strengths and areas of concern
Summative=Completion
Course Effectiveness
Synthesis of data to make decisions on course improvements
Summative evaluations
Are important to be able to improve the course design to facilitate learning in a more effective way.
Evaluate course objectives, experience and facilitation style
Improvement of programs
Form (process) and summative (outcomes)
Formative evaluation:
Is an on-going process to be considered at all stages of instruction.
Will enable the instructor to improve the course as he/she proceeds.
Facilitates course and content adaptation.
Will identify major gaps in the instructional plan or the need for minor adjustments.
Summative evaluation:
Assesses overall effectiveness of the finished product or course.
Can be a springboard in developing a revision plan.
Can be a baseline of information for designing a new plan, program, or course.
Will not help current students since it is conducted upon course completion.
Some questions that educators may want to ask students when collecting summative data include:
List five weaknesses of the course.
List three (or five) strengths of the course.
Copyright 2006 © Lycurgus Group, Inc.

9. Copyright 2006 © Lycurgus Group, Inc.
Proactive Evaluation
Focus on Quality Standards
Formative/Summative Evaluation
Formative/Summative Evaluation
Best Practices in Online Learning
Kirkpatrick’s Evaluation Model
Copyright 2006 © Lycurgus Group, Inc.
$16,200/year

10. Copyright 2006 © Lycurgus Group, Inc.
Proactive Evaluation
Ensures that all critical areas are addressed in an evaluation process.
Planning and design activities will not only ensure that all aspects of creative online learning activities and environments are addressed, but that subsequent formative and summative evaluation will be more directed and meaningful
Copyright 2006 © Lycurgus Group, Inc.

11. Copyright 2006 © Lycurgus Group, Inc.
Proactive Evaluation
Proactive evaluation is the foundation for the methodology that will be proposed
Thus, an evaluation system that provides a comprehensive and systematic view of institutional effectiveness supports this model and includes feedback from multiple sources.
Copyright 2006 © Lycurgus Group, Inc.

12. Kirkpatrick’s Four-Level Model
Evaluation model focused on results
Level 1: Evaluate reaction: Instruction/satisfaction
Level 2: Evaluate learning: KSAs
Level 3: Evaluate behavior: Application
Level 4: Evaluate results: Impact on organization (Schankman, 2004)
Level 5 added: ROI
Missing:
Fails to consider necessary improvements
Considering the importance of motivation
Level 1: Evaluate reaction to instruction/Satisfaction-survey
Level 2: Evaluate learning-survey
Level 3: Evaluate behavior-survey
Were performance objectives achieved?
Level 4: Evaluate results/impact on organization (Schankman, 2004)-Interview/performance data
Copyright 2006 © Lycurgus Group, Inc.

13. Copyright 2006 © Lycurgus Group, Inc.
Quality
Quality standards are important to consider in an evaluation plan
Quality framework: enables institutions to set goals, measure progress in reaching goals and implement required changes
The focus on quality and best practices are the basis of our evaluation system.
Applied to each individual institution uniquely by identifying needs/goals
Following organizations identified quality standards:
American Council of Education, American Distance Education Consortium, Sloan-C, Institute for Higher Education
Copyright 2006 © Lycurgus Group, Inc.

14. Instructional Effectiveness Process
Taken from a systematic view of change and development, quality initiatives, best practices and proactive evaluation, a systematic evaluation model will be described
In order to promote a comprehensive and proactive evaluation system, students, faculty and staff should be involved in providing feedback.
Key areas have been identified to be important for student success.
Identified in connection with research on best practices in online learning.
Developing this type of evaluation system will ensure that the feedback obtained will be useful for both faculty and school administrators.
Copyright 2006 © Lycurgus Group, Inc.

15. Copyright 2006 © Lycurgus Group, Inc.
Methods
Can use a variety of methods from surveys, interviews and focus groups
Focus groups can be informal way to identify what is working and what can be improved
Problem is that viewpoint is from a few participants
Copyright 2006 © Lycurgus Group, Inc.

16. Copyright 2006 © Lycurgus Group, Inc.
Factors to Evaluate
A number of different factors have been identified for evaluation purposes
Based on best practices/quality in online instruction
Copyright 2006 © Lycurgus Group, Inc.

17. How Factors are Identified
Expectations of success are to be communicated
Online learning should encourage interactivity between faculty and students as well as communication among students.
Being learner-focused
Best Practices:
Course design should be engaging
Courses should be geared to different styles of learning
Active learning and critical thinking both must be encouraged
It is imperative that instructors give prompt and substantive feedback.
Copyright 2006 © Lycurgus Group, Inc.

18. Evaluation System: Based on Quality standards in Online Education
Learning Effectiveness
Student Satisfaction
Faculty
Satisfaction
Cost Effectiveness
Main Measure: End of Course Survey
Also, can use focus groups and interviews
Student Survey on Institutional Effectiveness
Faculty Survey of online learning experience/Instructional Effectiveness
*Goal is to cut or control costs
*Reduce teacher turnover, lower training costs and improve technologies
*Increase retention
*Number of students obtaining degrees
-Good to develop dashboard of results as a communication tool
-important to utilize internal and external benchmarks n
Copyright 2006 © Lycurgus Group, Inc.

19. Factors to Evaluate End of Course Survey
The factors that should be evaluated in a comprehensive system include:
Course content
Course outcomes/objectives
Technology
Resources/Student support
Interactivity (with peers and instructor)
Feedback
Barriers to learning
Class engagement/Motivation
Instructional approaches/Addresses different learning styles
General Satisfaction
Most valuable aspects of course
Areas of improvement
Copyright 2006 © Lycurgus Group, Inc.

20. Factors to Evaluate Faculty Survey
Course Feedback
Evaluation of learning/university
Faculty Concerns
What is going well
What can be improved upon
Experience with Students
Incentives/Development opportunities
Faculty Time/Workload
Copyright 2006 © Lycurgus Group, Inc.

21. Factors to Evaluate Institutional Effectiveness
University commitment to students and quality education
Fairness in Evaluation/Adequacy of Feedback
Support/Resources Received
Instructor/Student interaction
Overall experience with classes and university services
Copyright 2006 © Lycurgus Group, Inc.

22. Copyright 2006 © Lycurgus Group, Inc.
Survey Items
These factors should then be broken into specify survey items that measure each dimension.
Also, different surveys should be developed for student, staff and instructor feedback.
Copyright 2006 © Lycurgus Group, Inc.

23. Copyright 2006 © Lycurgus Group, Inc.
Online Surveys
Surveys delivered via
Ease in administration and analysis
Branching of data is not apparent to participants
Web-based deployment
Ensures confidentiality
Provides quick turnaround of information
Copyright 2006 © Lycurgus Group, Inc.

24. Copyright 2006 © Lycurgus Group, Inc.
Other Important Data
Retention Data
Feedback from graduates
Document examination (handbooks, orientation, strategy, mission/vision)
History of changes made to courses
Copyright 2006 © Lycurgus Group, Inc.

25. Lycurgus Instructional Effectiveness Process
Step 1: Determine goals, objectives and standards of excellence.
Step 2: Identify internal benchmarks. (These benchmarks are used to compare to future evaluations and to compare with other schools.)
Step 3: Collect data from stakeholders.
Step 4: Report results and make recommendations.
Step 5: Identify top areas for planned improvements
Step 6: Implement changes.
Step 7: Reevaluate.
Copyright 2006 © Lycurgus Group, Inc.

26. Lycurgus Instructional Effectiveness Process
Copyright 2006 © Lycurgus Group, Inc.

27. Copyright 2006 © Lycurgus Group, Inc.
Summary
Utilizing this comprehensive evaluation process will ensure that key critical areas will be identified for improvement and planned change can take place.
This is important for student and faculty satisfaction as well as overall institutional effectiveness.
Copyright 2006 © Lycurgus Group, Inc.

28. Copyright 2006 © Lycurgus Group, Inc.
Q & A
Contact:
Donna DiMatteo, Ph.D.
Lycurgus Group
Donna DiMatteo, Ph.D. is an Online Instructor and also Managing Director of Consulting Services for The Lycurgus Group, Inc Vincent Street, Redondo Beach, CA 90277,
Copyright 2006 © Lycurgus Group, Inc.

29. Copyright 2006 © Lycurgus Group, Inc.
References
Cheung, D. (1998) Developing a student evaluation instrument for distance teaching. Distance Education, 19 (1),
DeVellis, R. F. (2003). Scale development: Theory and applications. Thousand Oaks, California: Sage Publications.
Dominguez, P.S. & Ridley, D. (Sept. 1999) Reassessing the assessment of distance education courses. T.H.E. Journal, 27 (2),
Kaufman, R., Guerra, I., Platt, W.A. (2006). Practical evaluation for educators. Thousand Oaks, California: Sage Publications.
Lockee, B., Moore, M., & Burton, J. (2002). Measuring success: Evaluation strategies for distance education. Educause Quarterly, 1,
Palloff, R. M. & Pratt, K. (2003). The virtual student: A profile and guide to working with online learners. San Francisco, California: Jossey-Bass.
Copyright 2006 © Lycurgus Group, Inc.

30. Copyright 2006 © Lycurgus Group, Inc.
References
Reisetter, M. & Boris, G. (Winter 2004). What works: Student perceptions of effective elements in online learning. Quarterly Review of Distance Education, 5 (4),
Schankman, L.H. (2004). Holistic Evaluation of an Academic Online Program. Retrieved May 3, 2006, from
Sims, R., Dobbs, G., & Hand, T. (Oct. 2002) Enhancing quality in online learning: Scaffolding planning and design through proactive evaluation, Distance Education, 23 (2),
Stewart, B.L., Waight, C.L., Norwood, M.M., & Ezell, S.D. (Summer 2004) Formative and summative evaluation of online courses. Quarterly Review of Distance Education, 5 (2),
Strong, J. H. ( 2006). Evaluating teaching. Thousand Oaks, California: Sage Publications.
Copyright 2006 © Lycurgus Group, Inc.