A Holistic Approach to Assessing eLearning A Holistic Approach to Assessing eLearning Dr. Tom Wilkinson, Director The Institute for Distance and Distributed Learning Virginia Tech Appalachian College Association Summit 2004 Abingdon, Virginia

Virginia Tech …. excellence and access Nearly 60,000 credit and non-credit eLearning enrollments since Su98. 1,672 eLearning course offerings since Su different eLearning courses offered in past 5 years 21 master degrees/certificate/ re- certification/licensure programs 85% of academic departments participating 89% of online learners were more confident they could reach their academic goals 88% rated the quality of instruction good or excellent Growth in eLearning course offerings

Created in 1999, the Institute for Distance and Distributed Learning provides leadership, coordination, management and support to the distance and distributed learning (eLearning) activities of Virginia Tech. The Institute takes a holistic approach to distance learning where the students total educational experience is considered. Electronically extend the campus Provide an Open Campus Research & Development International Education Specialized & Contractual Virginia Tech’s... Institute for Distance & Distributed Learning

Academic Administrative Support Student & Faculty Academic A Holistic Approach to eLearning … the whole of the experience is greater than the sum of its parts

Information Systems University Engagement … of distance and distributed learning at Virginia Tech Bursar Faculty Library Alumni Enrollment Services CEUT/ CIL Research University Outreach Budget & Finance Adminis trative Services Graduate School IDDL Colleges & Departments

Academic Stakeholders A Holistic Model of eLearning … the whole of the experience is greater than the sum of its parts

Information Systems Colleges & Departments University Engagement … of distance and distributed learning at Virginia Tech Bursar Faculty Library Alumni Enrollment Services CEUT/ CIL Research University Outreach Budget & Finance Adminis trative Services Graduate School Access Cost Systems Faculty StudentLearning Stakeholders

89% increase in eLearning enrollments 37% increase in academic departments involvement 20% of eLearning faculty are honored faculty 151% increase in eLearning courses 120% increase in degree & certificate programs 100% increase in high enrollment online courses 34% increase in centers & sites served 95% of barriers eliminated or significantly reduced Nearly $2 million in IDDL Enterprise Fund eligible revenue Award winning programs & the Chronicle article eLearning at Virginia Tech …

Assessment of Online Learners 91% were satisfied or very satisfied 96% rated services about the same, better or much better than with traditional courses 89% were more confident they could reach their academic goals 89% said they could not have taken this course in a traditional format 92% completion rate of “virtual” students …over 1,500 respondents

SO WHAT… does that tell us about the quality of our efforts?

Why Assess eLearning?

Who Assesses eLearning?

What about eLearning do we assess?

When do we assess eLearning?

What is Quality eLearning? “Quality is hard to define, but I know it when we see it.”

Holistic Approach to eLearning Quality The whole of the experience is greater than the sum of its parts Academic Stakeholders Institute for Distance and Distributed Learning Stakeholders

What are the Key Indicators of Quality? Accrediting agencies (SACS) State/Federal Agencies (SCHEV, DoE) Best Practices (WCET, ACHE) Sloan-C “Quality Pillars” Institutional Requirements Operational Requirements

eLearning Quality Matrix Key Quality Indicator Documentation –KQI Source –Quality Metric –Process/Practice –Documentation –Assessment Schedule –Score Card

The KQIs ….Key Quality Indicators Distance and Distributed Learning at Virginia Tech Key Quality Indicators CriteriaProcess Documentation Assessment Schedule Metrics Scor e 1. Institutional Indicators 1a. The program is consistent with the institution’s role and mission - SACS criteria - SACS Best Practices - ACHE Guidelines - WCET Principles Review for consistency at yearly IDDL strategic planning retreats - University Strategic Plan - Academic Affairs Strategic Plan - DDL Strategic Plan May each year IDDL Mission and Strategic Plan is consistent with University Mission and Strategic Plan 3 1b. Compliance with SACS notification accrediting requirements - SACS Criteria - SACS Best Practices - WCET Principles Review of program offerings with SACS notification file - VT SACS notification process. - Grad. School program list. - SACS notification file. June each year SACS notified of all appropriate distance learning program changes. 3 1c. University’s budgets and policy statements reflect its commitment to Learners - SACS Accreditation - SACS Best Practices - WCET Principles - ACHE Guidelines Review university budget for DDL inclusion and IDDL budget for adequacy; Review policy statements for DDL - University Strategic Plan - DDL strategic Plan - University Budget - IDDL budget allocation - IDDL Enterprise Fund report July each year IDDL budget on university base budget; 2 1d. Adequacy of technical & physical plant facilities including appropriate staffing and technical assistance - SACS Accreditation - SACS Best Practices - WCET Principles - ACHE Guidelines Conduct student & faculty course and support service evaluations; Review technology plan and budget - student evaluations - faculty evaluations - support staff evaluations End of each term At least 85% of all respondents satisfied with technology and technical assistance; Budget plan provides for technology updating; Staffing structure is appropriate 3

Example 1 Key Quality IndicatorsCriteria Source (s)Process 1. Institution/Organization Indicators 1a. The program is consistent with the institution’s role and mission. SACS criteria SACS Best Practices ACHE Guidelines WCET Principles Review for consistency at yearly IDDL strategic planning retreats

Example 1 (continued) Documentation Assessment Schedule Metric (s)Score University Strategic Plan Academic Affairs Strategic Plan DDL Strategic Plan IDDL Strategic Plan May / June each year Program is consistent with University Mission and Strategic Plan 3 Score: 3 = meets or exceeds Key Quality Indicator 2 = meets most of KQI 1 = meets some of KQI 0 = does not meet KQI

Example 2 Key Quality IndicatorsCriteria Source (s)Process 4. Student Indicators 4a. Appropriate services are available for students of electronically offered programs. SACS criteria SACS QEP SACS Best Practices ACHE Guidelines WCET Principles Identify appropriate online student services through: consultation with university student affairs division student surveys utilization reports evaluation of services

Example 2 (continued) Documentation Assessment Schedule Metric (s)Score Student Services for Distance Learners thesis results Online Wellness Resource Center dissertation results Hierarchy of Student Services needs for distance learners (independent study) Course evaluations OWRC utilization results Yearly identification of service needs (May/June) Student satisfaction & service utilization each semester Online student services provided if 10% or more of students indicate the importance of service. 90% or more of distance learners rate services good or excellent. 3

Contextually Designed Models … addressing changes within the system Pedagogical Considerations - course management - opportunity for interaction and engagement - extending research - real-world application - assessment and evaluation - content life-cycle Economic Considerations - business plan - costing break points - program maintenance including course re-design/development - faculty and department incentives - revenue distribution - return on investment Administrative Considerations - course logistics: application, registration, section assignment - infrastructure requirements - roles and responsibilities - faculty availability: qualifications, recruitment, hiring, orientation, training - policy adaptation/creation

Interactivity … in eLearning Number of Students Fully Interactive Self-paced Amount of interactivity with faculty & other students High Low High

Program Development Process … key system elements Analysis and Planning - need identification and assessment - congruency with mission/strategic plan - market analysis - SWOT analysis - business plan - approval process Development and Delivery - faculty development - program and instructional design - prototype development - online course development/ production - scalable T & L technology deployment - deployment of support services - program delivery - ongoing assessment and continuous improvement Evaluation and Assessment - learning outcomes - 5 key holistic factors -education, access, communication, technology and satisfaction - quality of the online experience - program effectiveness and impact - program return on investment (ROI)

Identifying a Niche … finding your spot in the eLearning marketplace Important Factors for Individuals Interest in Taking Online Courses 98% - transferability 97% - online access to library 96% - interaction with instructor 94% - reputation of the institution 94% - technical support offered 93% - cost 93% - allowed to work at own pace Market Analysis Adults aged 18 – 50 Possess some college 4 states in eastern U.S.A.

Course Development Model … a scalable system PHASE 1 Planning & Development PHASE 2 Component & Module Production PHASE 3 Component & Module Evaluation Material Development Content Formatting Course Management Tool Streaming Production ADA Compliance Audio Graphics Learning Outcomes Design and Flow Instructional Delivery Production Values LEARNER Course Feedback DELIVERY PRINT CD-ROM INTERNET SERVER  Distance Learning  Instructional Design  Online Education  Course Transformation Considerations  Identify Delivery Methods  Course Re-design Plan  Component Design Plan  Production Plan DESIGN, DEVELOPMENT Master Faculty and IDDL Staff LEARNER INSTRUCTOR

Online Course Delivery Model … a scalable system MASTER FACULTY Plan Course Design Content Maintain Content Establish Assessment Metrics Instructor of Record IDDL Programmatic Support Course Transformation Develop Course Media Coordinate Delivery DL INSTRUCTOR DL INSTRUCTOR DL INSTRUCTOR DELIVERY N N N DESIGN, DEVELOPMENT, DELIVERYMANAGEMENT Course Feedback LEARNERS INSTRUCTOR Monitor Student Progress Manage Logistical Tasks Guide, Mentor, Coach Lecture & Provide Feedback Help Design Learning

Scalability … Cost Model Comparison (Technology Costing Methodology – WCET) Number of Students $ Course-Related Costs Classroom Mediated High Low High Student-Related costs

For more information please contact: Dr. Tom Wilkinson – Director, Institute for Distance and Distributed Learning Virginia Tech