Measuring the Effectiveness of Virtual School Programs: Research and Beyond July 10, 2007

2 Introductions and Program Overviews VHS Global Consortium – Liz Pape IDLA -Idaho Digital Learning Academy – Donna Vakili eHigh School, Cobb County, GA – Becky Nunnally

3 What does an effective virtual school program look like? What are the indicators for measurement and how are they defined? How are virtual school programs held accountable? How do we communicate if we all define ourselves by different standards and definitions?

4 EVALUATION is integral to a continuous quality improvement process. What do we evaluate?  Online Course (Becky)  Online Instruction (Donna)  Student Learning (Liz)  Online Program ( All)

5 “For many students, online access has changed the way they see the world and the way they work and play. Consideration of these student issues is now critical in designing and delivering quality online instruction.” Standards for Quality Online Courses, SREB, November, 2006 Why do we evaluate an online course?  To improve the course  To improve student learning

6 How do we evaluate an online course?  Clearly defined objectives aligned to standards  Linking learning objectives to specific learning activities, course materials, assessment and measurement  Opportunities for communication and interaction including student to student, student to instructor and student to content  Highly engaging activities that meet the needs of all learning styles and engage the student  Developed in teams and meets standards and quality measures by peers.

7 Ex. eHigh School Course Evaluation Process  Technical design  Curriculum alignment  Rigor, depth, and breadth  Student performance  Student participation and interaction  Ongoing

8 Why do we evaluate online course instruction?  To improve course design  To improve delivery of course content  To improve course facilitation and management  To meet the needs of our customers !

9 How do we evaluate online course instruction? Focuses on whether the objectives of the course were met & informs practice.  Pre-/Mid/ Post-course surveys to all stakeholders  Learner reflections  Peer and Administrator review  Instructor self-evaluation

10 Ex: IDLA Teacher Evaluation: External (Fall ‘04-Fall ‘05) Student Surveys (N=715) – 87% of students had positive and rewarding interactions with their online teacher. – 88% said that the online teacher was very supportive and helped them learn. – 89% said that the teacher was good at moderating course discussions. – 86% felt that the teacher met their needs. – 85% felt that the teacher was prompt and helpful with feedback. – 84% felt encouraged to express themselves throughout the course. – 84% felt satisfied with the extra online help received from the teacher. –Areas of Improvement: Declining trend in experience with teachers (up 11% for Summer ’06), Increase prof dev of teachers in moderating online discussions, more teacher interaction

11 Ex: IDLA Teacher Evaluation: External (Spring 2006) Administrators (N=152) –Overall, 86% of administrators rate IDLA as good or excellent. –85% of administrators rate responsiveness of IDLA teachers to needs of their school and students as good or excellent. –64% rate the quality of the IDLA teachers as average to above avg (30% Don’t Know) –Areas of improvement: Improve student progress reporting, clear lines of communication, and schools know their responsibilities.

12 Ex: IDLA Teacher Evaluation: Internal Methods Independent Contractors –Non-Renewable –Teaching Expectations Bonus: Teaching Rubric –Administrative Requirements (25%) –Professional Development (20%) –Communication and Intervention (45%) –Student Evaluations (10%)

13 Ex: Evaluating Online Course Instruction: Informs Practice at IDLA Mandated Online Site Coordinator Course. Instituted Teacher Rubric. Systemized and clarified role of online principal and evaluation. Established systems and consistency for reporting such as phone logs and progress reports. Guided Online and Summer Conference Professional Development for Faculty. Focused Emphasis on Quality & Customer Service. Increased Value for External Evaluations.

14 Why do we evaluate student learning?  Mastery of course content  Application of course content to the real world

15 How do we evaluate student learning?  Pre-/Mid/Post-course surveys  Learner reflections  Peer review  Instructor self-evaluation  End-of-course assessments

16 Ex: VHS Student Learning Measures AP pass rates: 70% AP exam take rates:79% Course completion rates: 79% Credit recovery rate (summer):88% Teacher/SC/Principal/Super/Student Survey input

17 Ex: VHS AP exam take rate with # of Students

18 Ex: VHS Course Completion Rate

19 Why do we evaluate an online program?  Accountability  Stewardship of funds  Improve student achievement  Meet the needs of the learning community  Plan for the future

20 How do we evaluate an online program? Program Utilization  Enrollment (seats)  Number of students served  Number of students served with no other opportunity  Course/Teacher Evaluations  By students  By parents  Counselor and Administrator Surveys and Interviews

21 Ex. VHS Growth Goal: Number of Students

22 Ex: VHS Professional Development Measures – Hard Data Points, Research and Survey Data Professional development grad rate: 89% 1 st -semester teacher success rate: 94% % of teachers remaining with FA: 4% # of external evaluations Satisfied with VHS professional development –99% principals Applying VHS pedagogy to f2f teaching –79% teachers

23 Ex: VHS Performance Metrics-Program and Services Quality Membership retention rates: 87% Seat utilization: 83% # of external evaluations Satisfaction with electronic communications: –Teachers: 99%, SCs: 96% Website –Teachers: 97%, SCs: 98% Newsletter/Annual Report –Superintendents: 86%/68% –Principals: 78%/79% –Teachers: 79%/54% –SCs: 84%/54%

24 Ex: VHS Quality Benchmark Indicators

25 eHigh School Program Measures of Effectiveness  Student success - completions - passing - exceeds/meets EOCT  Meets local school needs - scheduling conflicts - graduation requirements  Meets student needs - flexibility - acceleration - credit recovery - hospital/homebound - family obligations

26 Tools of Engagement Whole Course Delivery Blended Learning Blackboard Live Classroom Blogs/Wikis Podcasts Polar Watches Turnitin.com AP Courses Work Study Programs Career Education High School Middle School Faculty Collaboration Mathematics Language Arts Science Social Studies Health & PE Business Meeting our 21 st Century Students Online Cobb County School District Marietta, Georgia ISTE NETS·S Standards 1. Creativity & Innovation 2. Communications & Collaboration 3. Research and Information Fluency 4. Critical Thinking, Problem-Solving & Decision-Making 5. Digital Citizenship 6. Technology Operations & Concepts

27 Ex. Virtual Charter Schools: Quality/Evaluation Similar to Supplemental: – Inputs: Sourcing, Structure, Alignment – User Input: Surveys, Feedback, “Popularity” – Results: Successful Completion Different from Supplementals: – Total Student Progress Monitoring (and Intervention with Multiple Tools) – NCLB Accountability – Test Results – Systemic Improvement – Courses, Curriculum, Platform, Instruction

28 Measuring the Effectiveness of Virtual School Programs: Research & Beyond ISSUES  Indicators for measurement  Definition of indicators  Timeframe for evaluation  Example: Completion Rates

29 Measure of Effectiveness? Policy Changes –Establishment of State Virtual Schools –Michigan Requirement for High School Graduation Funding –Changes to State Law and funding formulas –Examples: IDLA, Georgia, and Florida Virtual

30 And still more questions Are we measuring what we should be measuring? What else should we measure? Who should be measuring? How do we publish results of measures? Where is the Consumer Reports for virtual schools and online courses?

31 Will the growing national investment in K–12 online learning, particularly implementing and expanding the use of online high school courses provide a basically sound, high-quality educational experience for online students? Or will this new investment in K–12 online learning be yet another waste of scarce national educational resources on today’s newest educational technologies that is destined to be abandoned in some virtual tomorrow?” (A Synthesis of New Research on K–12 Online Learning)

32 References eSchool Idaho Digital Learning Academy Keeping Pace with K12 Online Learning: A Review of State Level Policy and Practice North American Council for Online Learning (NACOL) A Synthesis of New Research on K–12 Online Learning Virtual High School (VHS)