Principled Assessment Designs for Inquiry (PADI) Geneva Haertel, SRI International Robert Mislevy, University of Maryland Partners: Mark Wilson & Cathleen Kennedy, University of California-Berkeley Nancy Butler Songer, University of Michigan, BioKIDS Project Kathy Long, University of California-Berkeley, Lawrence Hall of Science, FOSS Project Larry Hamel, Codeguild Inc.

PADI Goals Conceptual frameworks for: Object model for task design Design Patterns for assessing inquiry Task-design Assessment delivery Object model for task design Software tools to aid the design process “Scoring engine” to sort out evidence Worked-through applications Globe, FOSS, BioKIDS

Method Student Model(s) Evidence Models Stat model Evidence Rules Task Models Feature s 1. xxxxx 2. 3. What complex of knowledge, skills, or other attributes should be assessed, presumably because they are tied to explicit or implicit objectives of instruction or are otherwise valued by society? What behaviors or performances should reveal those constructs? What tasks or situations should elicit those behaviors? (Messick, 1992)

OBJECT MODELING AND SYSTEMS DEVELOPMENT Strand Accomplishments Drafted use-cases and initial UML models Prototyped web application to support example-based modeling Entered examples and iterated models for design patterns, templates, student models, etc. Expanded measurement model to support MRCML scoring engine Created GradeBook sample application Abstracted EMO framework for CSCW 2004 paper and NSF Science of Design proposal Implementing wizards for assessment design

Object Models Design System GradeBook Wizards This is a high-level view of the elements of the system, represented in an Object Model. Design patterns are one of many objects in the system. Each object is further made up of component objects and relationships.

MODELING Strand Purpose Problem Approach Accomplishments Designs to assess students’ creation, use, and understanding of scientific models Problem While there has been rapid growth in the development of modeling software and tools for instructional purposes, few of these modeling tools are accompanied by assessments that provide high-quality evidence of student understanding Approach Prepare literature review of modeling tools and how student use of modeling tools and activities have been assessed in recent years Create design patterns for creation and use of models Organize AERA Interactive Symposium Accomplishments AERA Interactive symposium, Modeling, Inquiry, and Assessment: Developing a Principled Assessment Argument for Computer-Based Modeling Tools Modeling design patterns in the PADI design system Draft review of modeling research and implementation in education and recommendations for principled assessment designs

Examples of Modeling Tools STELLA model of market growth ModelIt interface for air quality model NetLogo model for Connected Chemistry ThinkerTools model

FOSS Strand Purpose To use the PADI Design System to develop new science inquiry assessments with the following features and capabilities: Accommodates alternative student models to measure student learning in different ways from the same assessment activities. Provides immediate feedback to students and teachers when assessments are online. Uses assessment feedback to help students monitor and direct their learning activities.

The Online Assessment Integrates Instructional and Assessment Purposes The FOSS Online Assessment The Online Assessment Integrates Instructional and Assessment Purposes This item can be used in “Practice Mode” to generate tutorial hints, or in “Quick Check” mode to generate a proficiency estimate for the student at this time point.

NAEP/TIMSS Strand Purpose Problem Method To understand the surface and deep characteristics of NAEP and TIMSS items through the lens of the PADI system Problem What are the characteristic and variable features of NAEP and TIMSS items that measure inquiry? Method Reverse engineer NAEP and TIMSS items into PADI templates

NAEP Floating Pencil Template Student Model Variables Task Model Variables - Content: Density / relative density - Content: Errors of measurement - Inquiry: Tools and techniques used to gather data - Inquiry: Evidence, content knowledge, and logical argument used to predict or explain - Level of scaffolding (inquiry, content) Format of student response (multiple choice, short constructed response, extended constructed response) Level of student explanation (none, choice of response, student-generated) Data representation (drawing, table, graph)

SCORING ENGINE Strand Purpose To develop a stand-alone module that can be called by an assessment application to compute student proficiencies for: Multivariate student models Multivariate observations Application-managed model specifications: Student model parameters Item parameters Scoring parameters Estimation parameters Proficiency type (posterior distribution, maximum likelihood estimate, plausible values)

Student Responses Database PADI Design System Database The Scoring Engine Servlet An Assessment Application Calls the Scoring Engine via HTTP to Estimate Student Proficiencies Student Responses Database   PADI Design System Database  Assessment Application Scoring Engine via HTTP via HTTP or local Extracts measurement model details from the PADI Design System. Requests student proficiency estimates from the Scoring Engine.

EPISTEMIC FORMS AND GAMES Strand Epistemic Forms & Games (Collins & Ferguson 1993) Epistemic forms - target structures that guide inquiry Epistemic games - procedures for “filling out” a form PADI as Complex of Epistemic Forms & Games Types of inquiries guided by PADI “What are potential constituent elements of assessment tasks that can provide evidence about (such and such) skill?” “Given this conceptual assessment argument, what operational assessment can elicit, identify, and synthesize evidence in a way that instantiates the conceptual argument?” If TOMS = PADI’s epistemic forms, then what are PADI’s games and in what ways are they epistemic?

Method of Investigating PADI’s Epistemic Nature What exactly are PADI’s games? Some rules and constraints are “built in” Investigate and catalog constraints and strategies that have been discovered by PADI’s first users through interviews How do PADI’s design principles make its games epistemic? Evidence Centered Design Object Oriented Design

GLOBE Strand Purpose Why GLOBE Is a Good Use-Case for PADI To develop GLOBE template(s) with the PADI system To use GLOBE template as an example for PADI Wizard Why GLOBE Is a Good Use-Case for PADI GLOBE assessments address multiple aspects of science inquiry and content knowledge GLOBE generic template and samples of GLOBE assessments created by SRI facilitate development of the GLOBE template for PADI Global Learning and Observations to Benefit the Environment (GLOBE) is a worldwide hands-on, primary and secondary school-based education and science program (www.globe.gov).

GLOBE Generic Template [Insert GLOBE data or graphs here] Present problem requiring use of GLOBE data archives. Planning Investigations: Ask students to pose relevant questions. Analyzing and Interpreting GLOBE Data: Ask questions about data in the table. Conducting Investigations: Assure data quality. Analyzing and Interpreting Data: Ask questions about the relationships between variables. Analyzing and Interpreting Data: Ask to generate new representations to analyze trends. Analyzing and Interpreting Data: Ask for interpretation of specific features of graphs. Planning Investigations: Ask to set up another problem using new data set. Communicate Findings: Ask to summarize and report findings.

GLOBE PADI Template Activity 1: Posing Research Questions Student Model Evidence Model Task Model Student Model Variables Measurement Model Work Product Materials & Presentation Task Model Variables GLOBE Data Ability to Plan GLOBE Investigations GLOBE Activity Scaffolding Student-Developed Research Question Based on GLOBE Data Observable Variable: Score on Posing Research Questions Problem Situation Description Number of Questions Requested Prompt to Pose Questions Ability to Understand GLOBE Concept Sample Question TMV Sample Question Specification of Required Components TMV Specification of Required Components

BioKIDS Strand Purpose Longitudinal studies focusing on 5th-7th graders’ critical thinking in science Audience is diverse, high poverty urban children Students analyze data, look for patterns, build explanations of phenomena Students learn basic science concepts and complex reasoning in concert

inquiry scaffold provided Content-Inquiry Matrix for “Formulating Scientific Explanations from Evidence” complexity of science content required to perform task amount of inquiry scaffold provided

Student Performance on Posttest Inquiry Reasoning By Complexity Type