The Lycurgus Group: Instructional Effectiveness Survey System LEAP System 9:00 AM on Wednesday, June 7, 2006

Copyright 2006 © Lycurgus Group, Inc. The Lycurgus Group, Inc. An organizational effectiveness and development consulting firm that uses scientific and research approaches to assist organizations in the areas of organizational assessment, change, development and online learning. Copyright 2006 © Lycurgus Group, Inc.

What you can expect from us We will... Design tailored programs to measure and manage Student, Faculty and Staff experience and effectiveness Deploy hundreds of surveys Conduct focus groups/interviews Develop faculty training and development programs Conduct industry studies/identify benchmarks Measure financial impact Determine instructional design needs Partner with your organization to achieve distance education excellence! Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Services Provided Our services are focused in the following areas: change management organizational and leadership development training and development e-learning organizational metrics performance management survey assessment and research Copyright 2006 © Lycurgus Group, Inc.

One small step for education, one giant LEAP for Introducing the LEAP System™ One small step for education, one giant LEAP for distance learning excellence Copyright 2006 © Lycurgus Group, Inc.

Lycurgus Evaluation Assessment Process The LEAP System™ The LEAP System™ is a systematic approach in assessment, change implementation and evaluation. Assessments measure learning outcomes and institutional effectiveness. Data is gathered and used to identify planned change efforts and needed improvements. All key stakeholders are included in the feedback process. The evaluation system is linked to measuring quality standards and best practices in distance education. The 12 key factors are based on research leveraging organizational effectiveness and best practices in distance education. Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Proactive Evaluation Focus on Quality Standards Formative/Summative Evaluation Formative/Summative Evaluation Best Practices in Online Learning Kirkpatrick’s Evaluation Model Copyright 2006 © Lycurgus Group, Inc. $16,200/year

Traditional Evaluation Methods Formative=Ongoing Institutional Effectiveness Formative evaluations can be accomplished through: This can be accomplished by obtaining feedback from students and faculty on their satisfaction with online learning. Improvement of programs Evaluate strengths/weaknesses and areas of concern Summative=Completion Course Effectiveness Synthesis of data to make decisions on course improvements Summative evaluations It is important to be able to improve the course design to facilitate learning in a more effective way. Evaluate course objectives, experience and facilitation style Improvement of programs Form (process) and summative (outcomes) Formative evaluation: Is an on-going process to be considered at all stages of instruction. Will enable the instructor to improve the course as he/she proceeds. Facilitates course and content adaptation. Will identify major gaps in the instructional plan or the need for minor adjustments. Summative evaluation: Assesses overall effectiveness of the finished product or course. Can be a springboard in developing a revision plan. Can be a baseline of information for designing a new plan, program, or course. Will not help current students since it is conducted upon course completion. Some questions that educators may want to ask students when collecting summative data include: List five weaknesses of the course. List three (or five) strengths of the course. Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Quality Framework Quality framework: enables institutions to set goals, measure progress in reaching goals and implement required changes The focus on quality and best practices are the basis of our evaluation system. Applied to each individual institution uniquely by identifying needs/goals Following organizations identified quality standards: American Council of Education, American Distance Education Consortium, Sloan-C, Institute for Higher Education Sloan Consortium: A consortium of institutions and organizations committed to quality online education Copyright 2006 © Lycurgus Group, Inc.

Kirkpatrick’s Four-Level Model Evaluation model focused on results Level 1: Evaluate reaction to instruction/satisfaction Level 2: Evaluate learning Level 3: Evaluate behavior Were performance objectives achieved? Level 4: Evaluate results/impact on organization (Schankman, 2004) Level 5 added: ROI Missing: Fails to consider necessary improvements Considering the importance of motivation Level 1: Evaluate reaction to instruction/Satisfaction-survey Level 2: Evaluate learning-survey Level 3: Evaluate behavior-survey Were performance objectives achieved? Level 4: Evaluate results/impact on organization (Schankman, 2004)-Interview/performance data Copyright 2006 © Lycurgus Group, Inc.

Evaluation System: Based on Quality standards in Online Education Learning Effectiveness Student Satisfaction Faculty Satisfaction Cost Effectiveness Main Measure: End of Course Survey Also, can use focus groups and interviews Student Survey on Institutional Effectiveness Faculty Survey of online learning experience/Instructional Effectiveness *Goal is to cut or control costs *Reduce teacher turnover, lower training costs and improve technologies *Increase retention *Number of students obtaining degrees -Good to develop dashboard of results as a communication tool -important to utilize internal and external benchmarks n Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Leap System™ Steps Step 1: Determine goals, objectives and standards of excellence. Step 2: Identify internal benchmarks. (These benchmarks are used to compare to future evaluations and to compare with other schools.) Step 3: Collect data from all stakeholders. Assessment includes surveys as well as interviews and focus groups when applicable. Step 4: Report results and make recommendations. Step 5: Identify top areas for planned improvements. Step 6: Implement changes. Step 7: Reevaluate. Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. 1. Determine Goals/Objectives Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Goals/Objectives It is important to determine: Why you are doing the evaluation? What do you hope to accomplish? What is success? Student performance, satisfaction, retention What will you do with the data? Who do you want to share the results with? Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. 2. Identify Internal Benchmarks Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Internal Benchmarks Important to identify key internal measures Such as student and faculty satisfaction and retention Will there be longitudinal comparisons made? Comparisons conducted within classes/departments/curriculum? What are internal messages of success? Or causes for alarm? Also, valuable to identify comparisons against established external benchmarks when applicable Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. 3.Collect Data from All Stakeholders Copyright 2006 © Lycurgus Group, Inc.

Data Collection Options Identify who will be involved in data collection process: Staff, faculty and/or students These are key stakeholders Also, graduates can provide informative data Identify best methods of data collection Different surveys, focus groups, or interviews, document evaluation Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Survey Assessments The nucleus of the system LEAP System™ Faculty/Staff & Student Effectiveness Survey & End of course Survey Web-Based Platform – Quickly gather critical data about your organization. Proprietary 12 Key Factors – Tested best practices. Researched and designed over the last 10 years. Detailed reports based on Data – Have the knowledge to make crucial decisions. Recommendations made on Key Areas – Know how to make improvements. Web-based Custom Dashboards – Know critical information about your organization anytime and anywhere in the world. Copyright 2006 © Lycurgus Group, Inc.

Survey Assessment Benefits Can involve many individuals in the feedback process in a short timeframe Able to get detailed feedback Ease in administration Utilize web to contact participants Ensures confidentiality of feedback obtained Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. 4. Report Results/Make Recommendations Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Reports Different levels of reports can be provided that will allow for different levels of action planning Strengths can be identified Can see overall picture of institutional performance Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. 5. Identify Top Areas for Planned Improvements Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Planned Improvements Top areas of concern can be discovered Focus of top action plans Use data to identify planned improvements, timeline and who will be involved in making needed changes Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. 6. Implement Changes Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Changes to Implement Utilizing an organizational effectiveness approach Identified from all data & participants Examples of changes made working with other schools include: Implementation of professional development activities Construction of new student orientation materials Changes to online courses/programs Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. 7. Reevaluate Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Evaluation Revisited Important to evaluate to see if improvements have occurred, if new changes need to be made or if new issues have developed Also, important to continually celebrate accomplishments! Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Other Key Services Training of Online Instructors Development of Materials to assist with new students new to online learning Online Course Design Selection/Recruitment of Adjunct Professors Copyright 2006 © Lycurgus Group, Inc.

Copyright 2006 © Lycurgus Group, Inc. Contact Us to discuss how we can help you! We can present solutions for your most difficult problems. www.lycurgusgroup.com info@lycurgusgroup.com ddimatteo@lycurgusgroup.com jgibson@lycurgusgroup.com Phone: (310) 543-0909 Fax: (310) 543-0050 Copyright 2006 © Lycurgus Group, Inc.