The Application of ECD to the Redesign of Advanced Placement Exams CCSSO June 2012 Maureen Ewing, Kristen Huff, Amy Hendrickson, Pamela Kaliski

Why ECD for AP? Need targets of measurement and curricular requirements No guesswork on part of the test user, test maker or test taker about what is valued in the domain Lack of ambiguity and use of task models helps ensure stable measurement Enhances validity argument

Three Critical ECD Phases

Experts with Disciplinary Knowledge Unpacked the Content omains B IG IDEAS ( LEVEL 1) E NDURING UNDERSTANDINGS ( LEVEL 2) S UPPORTING UNDERSTANDINGS ( LEVEL 3)

Experts with Disciplinary Knowledge Unpacked the Domains I DENTIFY S KILLS OR REASONING PROCESSES A RTICULATE DEFINITION OF SKILL ( I. E., OBSERVABLE EVIDENCE )

Inputs to Domain Analysis National and state standards Latest research on student learning and assessment Curriculum Surveys Other relevant documents Expert judgment of panelists

Three Critical (re)Design Phases

Writing Claims and Evidence Claims start with the phrase The student can to reinforce the focus on what the student knows and can do Claims are comprised of content and/or skills that were identified in the domain analysis Evidence starts with the phrase the work is characterized by… Evidence includes nouns and adjectives (no verbs)

If its unobservable, it isnt evidence Examples of non-evidence: a deep awareness of the importance of theorems understanding of differences and similarities between derivatives and integrals What would a student have to do, say, or write in order to prove to you that he/she has a deep awareness of the importance of theorems? The answer to this question is the evidence. As evidence is written, necessary edits to the claims may be uncovered; iteration is key!

Achievement Level Descriptions – ALDs Describe what students in each performance category should reasonably know and be able to do Relate directly to the knowledge, skills, and abilities a student should know and be able to demonstrate to be classified into a particular score category Distinguish clearly from one performance category (e.g., AP Grade 2) to the next (e.g., AP Grade 3)

Draft claims and evidence Select exemplars Draft ALDs Assignment of ALs Revise c/e and assignments Revise ALDs

Three Critical (re)Design Phases

Achievement Level 3

Task Models: Take-home messages 1)Claims and evidence are fodder for item Stimuli Prompt Response options Rubrics 2)Many like items can be produced from same claim / evidence pair Construct relevance as the target of measurement is the claim/evidence Item generation; Comparability

Form Assembly Specifications - Considerations Multiple inputs Domain Model Experts ratings of importance of content and skills Psychometric criteria Structure of the domain Claims: skills-based versus integration of skills and content Content relationships Skill relationships Content and skill relationships

Challenges of ECD Iterative nature of work Sometimes viewed as doing it wrong the first time, an annoyance, but it IS the process Resource and time intensive (6 years and counting), especially development of task models based on iterative pilots Steep learning curve – difficult to think in terms of observable evidence Defining appropriate grain size for each piece

Lessons Learned and Ongoing Work We need to develop an evaluation process that specifically addresses the iterations between C&E, ALDs, etc Documents, captures, thinks about the iteration - this is the real work of figuring out the construct about how students learn as well as ECD We need to figure out the grain size issues and how everything fits together