1 Science, Learning, and Assessment: (Eats, Shoots, and Leaves) Choices for Comprehensive Assessment Design Eva L. Baker UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing (CRESST) U.S. Department of Education Boston, MA June 19, 2004

2 Learning from the Past  Knowledge generated by assessments needs to be usable and useful; present system is inchoate  Once-a-year results underrepresents learning and deforms instruction  Uncertain alignment, unless goals are restricted  Lack procedures and measures to document classroom practice  Early growth likely not sustainable  Coherence needed for assessments for classroom and external purposes  Transfer of learning a major deficiency

3 Assessment Design—Option 1 © 1965 Fantasy Records

4 Option 2: Coherent System Sharing Deep Structure  Focused on science learning  Shared by all assessments  Evidence-based  Framework needed beyond standards  Cognitive demands, content models, and expert scoring models  Technology-assisted design, administration, scoring, data interpretation, reporting

5 Purposes of Comprehensive Assessment System Components  External tests for public reporting and comparisons  Curriculum-embedded assessments for measuring instructional progress and diagnosis, and ultimately for accountability  Certification, end-of-course performance measures  Evidence of classroom practices and reform  Transfer tasks across conditions, situations, topics to avoid narrowing, inflation, support efficiency

6 Component Options  External exams: accountability, evaluation, certification, transfer—a mix of individual-level and matrix-sampled tests  Classroom assessment: fixed sequence, item pools, technology-based authoring systems, teacher-made assessments  Models and measures of classroom practice: assignments, work, assessments to document progress; staff development certifications  Reports: individual, classroom, school, district on individual, matrixed and transfer tasks

7 Research Findings and Futures  Complex assessments can be developed based on principles, reusable models, and templates; technology is expanding  Scalable rules for implementation  Validity evidence weak, especially for instructional sensitivity  Need: technical quality analyses for diagnosis, simulations, plus external uses

8 Cognitive Families for Science Assessment Science Content Access & Understanding Problem Solving Teamwork and Collaboration Metacognition Learning to Learn Communication Learning

9 Cognitive Model-Based Assessment  Cross-curricular cognitive families define core constructs  Embedded in science content, like linguistic rules in domains of language understanding systems  Derives from work in cognitive science, measurement, writing assessment, and AI development  Evidence that it works across subject matters, types of learning, and students

10 From Research to Assessment Design Domain- Independent Cognitive Principles:Reusable Architecture Topics Topic C B A TEMPLATE Learning/Assess Model DI+DS+Format RESEARCH FINDINGS COGNITIVE DEMANDS Research Findings SUBJECT MATTER SPECIFIC FINDINGS KWSK CONDITIONS TOPICS OF TASKS Expert scoring models

11 Deep Architecture for Science Assessment Design  Learning Research  Domain-independent cognitive demands  Domain-specific learning models  Expert scoring models  English language requirements  Content  Standards  Content ontologies or access rules, integration  Utilities  Reusable—transfer & instruction  Effectiveness  Validity—sensitive to instruction  Communication & review  Comprehensive

12 Feasibility  Quality assessments and cost containment— sharable objects, technology tools  Teacher support—certification  Dangers of ad hoc task structures  Selection of format relative to learning goals  Validity studies planned at time of design  Early sanctions not demonstrated

13 Technology Helps  Beyond the screen, computer-based testing of open-ended tasks, sensors  Graphic representations as outcomes  Automated open-ended scoring—towards propositional rather than regression solutions  Authoring systems with smarts—same system for teachers and external tests  Simulation architecture and narrative structures—new approaches to examining test quality

14 Achieve Deep Alignment of System

15 Description: Extra comfort for senior dogs. A popular orthopedic pet bed, made extra thick for aging dogs. A full 2" of medical grade convoluted foam supports bones and joints. copyright 2004 DK Cavanaugh

16 CRESST Web Site