1. Using the PADI Design System to Examine the Features of a NAEP Performance Assessment Kathleen C. Haynie Kathleen Haynie Consulting Andrea A. Lash, Geneva D. Haertel, Edys S. Quellmalz, Angela Haydel DeBarger SRI International Kathleen C. Haynie Kathleen Haynie Consulting Andrea A. Lash, Geneva D. Haertel, Edys S. Quellmalz, Angela Haydel DeBarger SRI International AERA April 2005

2. Background  Use of 21 st century collaborative workplace tools is on the rise  Technologically rich “dispersed” collaboration requires (Olson & Olson, 2000):  Common ground  Coupling of work  Technological readiness  Collaborative readiness 2

3. PADI Project  Seeks to improve assessment of inquiry in science learning  Networked collaboration  Development and use of Web-based tool 3

4. Central Process The central process for project strands working with the PADI Design System can be stated as follows: The interaction between assessment materials (e.g., assessment tasks) and the Evidence-Centered Design framework (Mislevy, Steinberg, & Almond, 2002) is mediated by various tools, representational forms, and representations utilized by collaborative workgroups. 4

5. 5 PADI Assessment Design

6.  Goals:  To reverse engineer a performance task from a large-scale, national assessment  To use the PADI System as an analytical tool for understanding task features  In reverse engineering and analyzing a performance assessment task via PADI, what types of knowledge were created? 6 NAEP Performance Assessment

7. 1. Selection of a Set of Items (July – August) 2. Exploration of the Floating Pencil Task (August – December) 3. Development of a Task Specification (December – March) 7 Reverse Engineering & Analysis Process

8.  We selected a set of items for analysis  NSES inquiry standards, PADI Design Patterns  Defined our Student Model  Exploration of multiple student models  Choice of NAEP framework  Refined the Student Model 8 The Student Model

9.  We explored evaluation & measurement issues via the PADI representational forms  Evaluative Sub-Model based on NAEP rubric  Multiple levels of conditional dependencies  MRCML psychometric model  Clarified our Measurement Model  Based closely on NAEP practices  Elements of PADI 9 The Evidence Model

10.  We explored attributes of the Floating Pencil task  Content-lean, inquiry-constrained  Use of lab materials; High verbal demand  Constructed Task Model  Sought to define a ‘Family of Tasks’ 10 The Task Model

11.  Process: Floating Pencil team engaged in collaboration using technological workplace tools  Results:  Analysis of the Floating Pencil task  A ‘trace’ on PADI: Floating Pencil Task Specification  The impact of our work on PADI Design System  Next Steps:  Task family and TMVs  Empirical data analysis  Technical report on Floating Pencil work  Consideration of alternative student models 11 So What?