Computer-based Assessment Paul Horwitz The Concord Consortium Presentation to DR-K12 PI Meeting, November 10, 2009

For assessment purposes, is the computer a process innovation or a product innovation? Inventions that started as process innovations and became new products Inventions that started as process innovations and became new products  The telephone  The refrigerator

Computer-assisted assessment as a way to increase productivity Deliver the identical test online Deliver the identical test online Embed questions within visualizations Embed questions within visualizations Random assignment of items Random assignment of items Adaptive testing Adaptive testing

Assessment techniques that depend on the computer Performance assessment based on simulations of real systems and processes Performance assessment based on simulations of real systems and processes Assessment based on students’ manipulations of causal models Assessment based on students’ manipulations of causal models Inferences based on tracking of problem- solving activities Inferences based on tracking of problem- solving activities

Examples of performance assessment: 1. The SPARKS Project Current ATE Project Current ATE Project College level: introductory electronics College level: introductory electronics Building self-paced assessments that give students useful feedback and offer an opportunity to practice skills Building self-paced assessments that give students useful feedback and offer an opportunity to practice skills Optionally, assessments generate reports for instructors as well. Optionally, assessments generate reports for instructors as well.

Examples of performance assessment: 2. “Evolution Readiness” Current DR-K12 project Current DR-K12 project Fourth grade science dealing with evolution as emerging from natural selection Fourth grade science dealing with evolution as emerging from natural selection Students perform experiments with virtual plant and (soon) animal communities Students perform experiments with virtual plant and (soon) animal communities Actions are logged, analyzed, and used to report on students’ inquiry skills and content knowledge Actions are logged, analyzed, and used to report on students’ inquiry skills and content knowledge

Challenges to innovation in assessment Technological barriers Technological barriers  Models and simulations take time to build  Required infrastructure complex and hard to maintain Psychometric barriers Psychometric barriers  How do you score these things?  e.g., what constitutes an “item”?  How do you determine validity and reliability?

Importance of innovation in assessment Our understanding of learning has advanced faster than our ability to assess learning Our understanding of learning has advanced faster than our ability to assess learning Schools live or die by the results of assessments that don’t reflect current learning sciences research Schools live or die by the results of assessments that don’t reflect current learning sciences research We “teach to the test” so we don’t teach what we ought to teach We “teach to the test” so we don’t teach what we ought to teach