1. ECD in the Scenario-Based GED ® Science Test Kevin McCarthy Dennis Fulkerson Science Content Specialists CCSSO NCSA June 29, 2012 Minneapolis This material is based on work supported by the National Science Foundation under grant DRL-0733172 (Application of Evidence-Centered Design in Large-Scale Assessment). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

2. NSF-Funded Research & GED ® Science Study the incorporation of evidence-centered test design (ECD) in technology-based large-scale assessment Goal: Determine how ECD can enhance the quality of technology- based large scale assessment and make development more efficient Partners: 2ECD in GED® Science

3. Incorporating ECD into GED ® Science Purpose To develop valid scenarios and items that allow us to evaluate evidence of test-taker’s performance regarding the GED ® Science Practices, a subset of the GED ® Science Assessment Targets Method Use design patterns during item and scenario development to help writers develop a coherent concept that explicitly references: – The knowledge, skills, and abilities that we are assessing – The types of observations that provide evidence about the KSAs – The features of a task or situation that provides that evidence 3ECD in GED® Science

4. Design Pattern in Assessment Design patterns are ECD tools used to guide the development of test scenarios and associated items A type of advance organizer for scenario/item writing Design patterns consist of a series of attributes that reflect assessment argument structure They lay out a design space for item writers Choices, connections, examples Aspects to consider Methods of increasing or decreasing cognitive difficulty

5. Benefits of Design Patterns ECD in GED® Science5 Connect the assessment argument to the items Can (in the long run) improve efficiency and validity Help scenario/item writers strengthen the alignment between their scenarios/items and the knowledge and skills they are trying to assess Support idea generation Inform the specific aspects of scenarios/items

6. Design Pattern Attributes Focal Knowledge, Skills, & Abilities (KSAs) The primary KSAs targeted by the design patterns Associated with a Science Practice Additional KSAs Other KSAs that may be required for successful performance on assessment tasks/items Potential Work Products Some possible things one could see students doing in response to an item that would give evidence about the KSAs Potential Observations Features of the things students say, do, or make on an item that constitute the evidence ECD in GED® Science6

7. Design Pattern Attributes Characteristic Features Aspects of assessment situations/items that are likely to evoke the desired evidence Variable Features Aspects of assessment situations/items that can be varied in order to shift difficulty or emphasis Narrative Structure Type of storyline used as a framework for scenario context ECD in GED® Science7

8. Design Patterns in GED ® Scenario Writing Each scenario and scenario-based item should be aligned to/informed by a design pattern when possible Multiple design patterns can be used to write items in a scenario Design patterns are housed in SRI’s online PADI Design System “Principled Assessment Design for Inquiry” (PADI) An 11-step process was developed to guide item writers through the use of PADI, design patterns Activation chart was designed to connect design patterns to GED ® Science Practices 8ECD in GED® Science

9. An 11-Step Process for Using Design Patterns Steps 1 & 2: Become familiar with design patterns, PADI, assignment Step 3: Brainstorm and research a suitable scenario topic idea Step 4: Refer to activation chart, connect design pattern KSAs to GED ® Science Practices Step 5: Use design pattern attributes to develop item ideas Steps 6 & 7: Modify scenario idea based on item ideas and design patterns, ensure alignment of scenario and items Steps 8-11: Document decisions, obtain approval, write items, and submit, revise, resubmit assignment 9ECD in GED® Science

10. Design Pattern Activation Chart Connects design pattern Focal KSAs to Science Practice indicators 10ECD in GED® Science

11. 11 Results, Reflections, Lessons Learned The 11-step process simplified and directed the use of design patterns The activation chart is a valuable tool for new writers, allowing them to focus on the writing rather than finding an appropriate design pattern Writers find the design pattern content and attributes to be moderately helpful in writing scenarios and items Increased usefulness may be gained as writers become more familiar with design patterns and PADI

12. Additional Information Technical reports, presentations, and project information http://ecd.sri.com http://padi.sri.com