© E. Kowch iD Instructional Design Evaluation, Assessment & Design: A Discussion (EDER 673 L.91 ) From Calgary With Asst. Professor Eugene G. Kowch march 27 (An Synchronous Meeting using WebCT discussion Thread and WWW Course Home Page Material)

© E. Kowch iD Assessment Instruments: Assessing learners Criterion Referenced Pretest: Placement & entry level skills Readiness Questions Placement Questions Posttest: helps us identify what did not work for this learner To make these tests, we need a lot of info: List of task based behavior objectives or Type of learning (verbal, etc, ) and Type of conditions for learning (Gagne) and events for learning Bad criteria test subjects: Writing a paragraph…affective evaluations…..

© E. Kowch iD Placement Learner Readiness Assessment (to determine entry performance) Readiness Concern Do the students have the Prerequisite skills? Pretest: Method of Inquiry for Placement no yes Provide readiness experiences Proceed with The instruction Placement Concern Have the students already Achieved the intended Outcomes? Advance Students To a higher level no yes Discuss: Math vs. Social Studies (criterion?) From Driscoll, M. (2000). Psychology for Instruction, 2nd Ed.

© E. Kowch iD Formative and Summative Assessments: ( of your Design) Formative methodology (can be done in order): 1. Education Context: 1 to 1 evaluation (designer & learner) 1. Questionnaires, interviews - check learning time, outcomes 2. Focus group 1. Nets out the instructor 2. Check the effectiveness of instruction (viability, attitudes of instr.) 3. Field Trial 1. Observe a full instruction event with the 2x modified course materials and design… test for achievement.

© E. Kowch iD Formative and Summative Assessments: ( of your Design) Summative Assessment methodology 1. Expert Judgment (assess the instructor) 1. Congruence; Can the instructor meet the needs of the organization? 2. Content: Are the materials complete, updated? 3. Instructor followed the design? 4. Utility of instructor? 5. Design: Are the principles of learning, instruction and motivation in the materials? Instruction? 2. Field Trial 1. Outcomes Analysis: impact to learner on job, in school 2. Management: Instructor, supervisor performance test

© E. Kowch iD Summative Assessment (to monitor learning progress) Provide additional learning experiences no yes Assign grades to certify mastery Discuss Have the students achieved The intended outcomes Of instruction? Summative Learner Assmt. Method Evaluate the effectiveness of the instruction

© E. Kowch iD Assessment Planning (Magliaro from notes) Assessment: Learner performance Evaluation: judging learner decisions Program Evaluation: Appraising program success Assessment Possibilities Portfolios Physical Electronic Communicate task and knowledge performance Time consuming for all! Projects Demonstrate action / processes / competencies (training) Address problem solving (education) Require relevant subject matter!

© E. Kowch iD Assessment Planning (Magliaro from notes) 5 Step assessment process 1. Identify goals and objectives from Needs Assmt 2. Match goals & objectives with assmt tools 3. Choose instructional method 4. Design tasks and scoring methods (match) 5. Think through the implications…. Guidelines: 1. Start small 2. Develop rubrics 3. Communicate purpose of assessment 4. Involve participants 5. Keep going - recursive design 6. Look for blips - unfairness, odd results

© E. Kowch iD PostTest: Method of Inquiry for Placement The instructional role of Formative Assessment (to monitor learning progress) Provide Group or Individual remediation no yes DIAGNOSTIC ASSESSMENT (to study persistent difficulties Provide feedback to reinforce learning Proceed with The planned instruction Discuss During Instruction Are the students achieving the intended Learning outcomes?

© E. Kowch iD Recall: The “one” and the “many”

© E. Kowch iD The Distribution Curve: Defining Normal Norm Relative ranking, describes what cannot be done And what can be done..

© E. Kowch iD Great questions: "Can and should instructional designers place importance on process instead of product? If yes, is this desirable and reversible? Discuss" Designers Ask..

© E. Kowch iD Great questions: Should and can evaluation be both internal and external? Should and can the evaluator personalize his or her findings and put those results into a cultural perspective? Discuss." Designers Ask..

© E. Kowch iD 1) There is the sense that "outside the company" designers are not as biased as "backyard" designers. I wonder whether this is in fact true, or whether their biases are simply different. The "outside" designers could have biases that are harmful or just as blinding as "in house" people's. Designers Ask..

© E. Kowch iD Evaluation/Assessment There seems to be a belief that objectivity is somehow better than subjectivity, and I wonder about that too. Sometimes, a subjective approach may be called for. For example, in evaluating a Native education program, would it be effective to use a white person? Maybe. Then again, maybe not. Maybe the white person would have to spend so much time and effort learning first about the culture that her work on actual evaluation would become miniscule in comparison. Designers Ask..

© E. Kowch iD