2. Evaluating CAI Lessons Dr. Ennis-Cole CECS 5130

3. Lesson Objectives Have the expected outcomes been realized ? Were the desired outcomes a result of the lesson ?

4. Purpose of Evaluation Purpose of Evaluation 1. Confirmation of Decisions 2. Lesson Improvement “An ongoing process used to determine whether lesson objectives have been met, identify reasons for the observed performance, and identify areas of modification.” p. 299 Hanafin & Peck

5. Evaluation Techniques Informal - advice from experts, ideas from users, observations, interviews (simplistic- anecdotal data) Formal - planned evaluation models, formal analysis, attitudinal questionnaires, data collection & collation (formal tabulation - empirical evidence)

6. Levels of Evaluation FormativeSummative

7. Formative - identification of lesson features that need modification - ongoing process lesson improvement is the goal

8. Levels of Evaluation Summative - signature authority not used for modification, used for purchase or adoption decisions

9. The Formative Evaluation - - - - - - - - - - Issues addressed: Design Logic Selection of Vocabulary Clarity of Graphics & Branching Issues Interrogation of Student/User Responses

10. Stages of Formative Evaluation b 1. One-to-One: Initial lesson design & developmentInitial lesson design & development Informal proceduresInformal procedures Identify potential problemsIdentify potential problems b 1a. Assets of One-to-One Evaluations: Insight gained before coding or implementationInsight gained before coding or implementation Identification of problemsIdentification of problems Expansion of lesson features or redirectionExpansion of lesson features or redirection

11. Stages of Formative Evaluations b 2. Small-Group Evaluation: Conducted at the lesson’s completionConducted at the lesson’s completion Determines effectivenessDetermines effectiveness Collection & Interpretation of Student Performance DataCollection & Interpretation of Student Performance Data b 2a. Assets of Small-Group Evaluations: Use Informal techniques & dataUse Informal techniques & data Identify areas of improvementIdentify areas of improvement Generalized results from small-group interactionGeneralized results from small-group interaction

12. Stages of Formative Evaluations b 3. Field Testing: Conducted in semi or actual settingsConducted in semi or actual settings Occurs when lessons are “final quality”Occurs when lessons are “final quality” Conducted to confirm lesson effectivenessConducted to confirm lesson effectiveness b 3a. Assets of Field Testing: “Real Life” Simulations of Lesson Use“Real Life” Simulations of Lesson Use Rely heavily on both formal and informal techniquesRely heavily on both formal and informal techniques Minimal Changes can be madeMinimal Changes can be made

13. Evaluating CAI Instructional Adequacy Cosmetic Appeal Program Adequacy Curriculum Adequacy

14. Instructional Adequacy b Directions b Content b Navigation b Text Presentation b Logical Organization

15. Instructional Adequacy Instructional Adequacy b Does the lesson: have good procedures and activities ?have good procedures and activities ? amplify important terms & concepts ?amplify important terms & concepts ? emphasize important sections & content ?emphasize important sections & content ? promote meaningful user interaction ?promote meaningful user interaction ? match the learner with the appropriate learning task ?match the learner with the appropriate learning task ?

16. Instructional Adequacy Instructional Adequacy b Does the lesson: stress retention & transfer ?stress retention & transfer ? pace base on the learner’s characteristics ?pace base on the learner’s characteristics ? motivate the learner ?motivate the learner ? correctly track and record learner performance ?correctly track and record learner performance ? provide learner control ?provide learner control ?

17. Cosmetic Adequacy Cosmetic Adequacy b Is the lesson: appealing in a visual sense?appealing in a visual sense? using technology effectively ?using technology effectively ? presenting a consistent and cohesive picture ?presenting a consistent and cohesive picture ? supporting learning via “special effects ?”supporting learning via “special effects ?” free from grammatical and mechanical errors ?free from grammatical and mechanical errors ? effectively communicating objectives, outcomes, and procedural information ?effectively communicating objectives, outcomes, and procedural information ?

18. Cosmetic Adequacy Cosmetic Adequacy b Is the lesson: well structured ?well structured ? presenting a psychologically safe learning environment ?presenting a psychologically safe learning environment ? supporting the student’s learning ?supporting the student’s learning ?

19. Program Adequacy b Is the lesson: free from conceptual loops ?free from conceptual loops ? free from technical details ?free from technical details ? performing as intended ?performing as intended ? processing efficiently ?processing efficiently ? secure ?secure ? accurate ?accurate ? managing resources wisely ?managing resources wisely ?

20. Curriculum Adequacy b Does the lesson: relate to other lessons & activities ?relate to other lessons & activities ? use consistent procedures ?use consistent procedures ? incorporate teacher & learner preferences ?incorporate teacher & learner preferences ? facilitate the development of other lessons ?facilitate the development of other lessons ? become obsolete quickly ?become obsolete quickly ?

21. Curriculum Adequacy b Does the lesson: match user expectations ?match user expectations ? augment or supplement the curriculum ?augment or supplement the curriculum ? permit completion within the allotted time ?permit completion within the allotted time ? require “special” hardware or additional software ?require “special” hardware or additional software ? provide flexibility ?provide flexibility ?

22. “Complete” Software Evaluations “Complete” Software Evaluations b Courseware Evaluation Form: Background of the UserBackground of the User Lesson DescriptionLesson Description Hardware RequirementsHardware Requirements Evaluator’s Name & DateEvaluator’s Name & Date Criteria Above-mentionedCriteria Above-mentioned Summary RecommendationSummary Recommendation

23. “Web” Activity: “Web” Activity: b Search for: Software ReportsSoftware Reports PC Clearinghouse Software DirectoryPC Clearinghouse Software Directory Educational Products Information Exchange (EPIE)Educational Products Information Exchange (EPIE) Swift’s Educational Software DirectorySwift’s Educational Software Directory

24. “Web” Activity: “Web” Activity: b What are the advantages of commercial software reviews ? b How can these be used ? b What did you find on the WWW ?

25. The End The End b “That’s all folks.”