The Online Academy Evaluation Plan

Need for Evaluation 1.Inform and direct current and future development 2.Examine the usability of the checklist 1. Organization and self-regulation 3.Determine effectiveness of navigation

4.Review course modules for the following: Correlation to SOL Clarity of instruction Engagement and motivation of instruction Content presented at the appropriate level Adequacy of the resources, skills, and content-adequate for challenge Pedagogical approach and format Pedagogical re-engineering Need for Evaluation

 Analyze instructional strengths (pedagogical re- engineering)  Determine the effectiveness of the design and transfer of the materials from classroom to online.  Evaluate the usability and the benefits of the checklist to the learner  Locate ease of use problems Purpose/Goals

 The evaluation strategy is to determine the effectiveness of the initial modules of the seven courses being developed for TOA. Using expert and one-to-one reviews.  The results of this evaluation will be used to inform and direct the current and future design and development of TOA. Strategy

Expert review a.Find a way to determine whether the site adheres to Clark and Mayer criteria and Gardner criteria. b.Recruit experts. c.Prepare questions. d.Design data collection tools. e.Set up testing area. f.Conduct evaluation. g.Analyze results. h.Write evaluation. Objectives

One-to-One review a.Recruit students. b.Prepare evaluation questions. c.Design data collection tools. d.Write script. e.Set up testing area. f.Conduct evaluation. g.Analyze results. h.Write evaluation. Objectives

Social/Cultural Factors  Priscilla Norton, TOA Project Director  The Policy Board  School districts  Subject Matter Experts (SMEs)  Students

 multi-level evaluation to explore as many perspectives as possible  evaluate at Levels 1, 2, and 3 of Kirkpatrick’s Levels of Evaluation.  the level of formality of the evaluations will vary from informal to formal Levels of Evaluation

 use observations during the one-to-one evaluations with the students  conduct interviews for the expert reviews  follow a script with a list of prepared questions, allowing for the introduction of additional questions or probing questions when needed.  evaluations will be recorded on either audiotape or videotape.  we will document observations and interviews in field notes. Data Collection

Analyze data by reviewing field notes and transcripts to look for common themes, patterns, and trends. Data Analysis

Logistics  Identified potential evaluators  Assigned team members to conduct evaluation (2 to each evaluation, one person interviews while the other records)  Identified tentative location and time  Written interview questions  Developed a protocol and script for each type of evaluation.