I NTERNATIONAL B ENCHMARKING S TUDY — C ONTENT A LIGNMENT Mary J. Pitoniak, Nancy Glazer, Luis Saldivia Educational Testing Service June 22, 2015 National Conference on Student Assessment Council of Chief State School Officers 1

Overview of Presentation Purpose of study Methodology Results: o General characteristics of the assessments o Alignment at item and blueprint/framework levels 2

Purpose of Study In addition to providing results that allow states to compare themselves to each other, PARCC is considering linking results to several international measures: o Programme for International Student Assessment (PISA) o Trends in International Mathematics and Science Study (TIMSS) o Progress in International Reading Literacy Study (PIRLS) 3

Assessments Compared 4 Comparison # Assessment PARCC (2015)PISA (2015)TIMSS (2015)PIRLS (2016) ELA/Reading 1Grade 4 2Grade 1015-year-olds Mathematics 3Grade 4 4Grade 8 5Integrated Mathematics I 15-year-olds

Design of Study ETS staff served as facilitators and content coordinators Representative team of three subject matter experts (SMEs) compared each pair of assessments o PARCC o ETS o International Reviews done virtually, with web meetings to discuss ratings Used released international items and operational PARCC items 5

Tasks Undertaken Mapped PARCC Evidence Statements to International Framework Dimensions Reviewed PARCC Items and Their Mapping to Evidence Statements Reviewed International Items and Their Mapping to Framework Dimensions Mapped International Items to PARCC Evidence Statements Mapped PARCC Items to International Dimensions Revisited Blueprint–Framework Ratings Done Previously Provided rating of alignment between international assessment performance levels and PARCC’s level 4 Provided rating of overall alignment 6

Purpose of Assessment o PARCC  Increase the rates at which students graduate from high school prepared for success in college and the workplace o International Assessments  Allow for policy makers to: –Gauge the knowledge and skills of students in their own countries in comparison with those in other countries –Set policy targets against measurable goals achieved by other education systems –Learn from policies and practices applied elsewhere 7 Results—General Characteristics

FeaturePARCCInternational Population All students in participating states Grade-based and continuous Sample of students in participating countries Grade-based (PIRLS and TIMSS, Gr 4 & 8) or age-based (PISA, 15 y.o.), at intervals ScoresIndividual scoresGroup scores InclusionMany accommodationsLimited accommodations Content basis Common Core State Standards (CCSS) Developed using Evidence- Centered Design (ECD) PIRLS & TIMSS: Curricula PISA: Knowledge/skills needed to solve real-world problems Administration mode Computer, with limited paper-and-pencil PIRLS & TIMSS: Paper PISA: 2015—Computer Prior—Paper (applies to released items used for this study) 8 General Observations (continued)

Results—Alignment Caveats about results o Overall rates of mapping shown in the following slides can be misleading since a large proportion of the units may map to just a few of the other assessment’s dimensions in proportions very different from those targeted in the other assessment’s framework o Also, structure of blueprints and frameworks are very different, so mapping may be easier in one direction than the other:  Few and broad (international assessments)—easier to map PARCC items into these frameworks  Many and specific (PARCC)—more difficult to map international items into PARCC blueprints 9

Results—Alignment Caveats about results (continued) o Because of these issues, detailed mapping results should be reviewed before drawing conclusions o For time reasons, and because we think they are most informative, we will be focusing on results from blueprint/framework and item/framework mappings, not the SMEs’ ratings of PLD or overall alignment 10

11 Numbers of PARCC Evidence Statements and PARCC/International Items Comparison PARCC Evidence StatementsPARCC Items International Test Items PARCC ELA Grade 4/ PIRLS Grade PARCC ELA Grade 10/ PISA Reading PARCC Math Grade 4/ TIMSS Grade PARCC Math Grade 4/ TIMSS Grade PARCC Integrated Math I/ PISA Math

12 ELA: PARCC Grade 4–PIRLS PARCCPIRLS Subclaims Reading Literature Reading Information Purposes Literary Experience Acquire and Use Information Evidence Statements Indicate what students must demonstrate in support of the claims Describe what one can point to in a student’s work that substantiates that the standard has been mastered Processes Comprehension processes allow students to demonstrate a range of abilities and skills in constructing meaning from written texts o Focus on and retrieve explicitly stated information o Make straightforward inferences o Interpret and integrate ideas and information o Evaluate and critique content and textual elements

13 ELA: PARCC Grade 4–PIRLS (continued) ComparisonNotes on MatchPassages PARCC Blueprint → PIRLS Framework 24% of Evidence Statements rated as No Match to PIRLS framework 68% of ES mapped to 2 PIRLS processes (Focus/retrieve and Interpret/integrate) -- PARCC Items and Passages → PIRLS Framework 15% of items rated as No Match to PIRLS framework 48% of items mapped to 1 PIRLS process (Focus/retrieve) 9 of 13 PARCC texts rated not likely to be on PIRLS PIRLS Items and Passages → PARCC Blueprint All PIRLS items were mapped to PARCC blueprint However, items align to a limited set (6) of PARCC standards 2/3 of the alignment is to standards that are most broad and general in scope No items mapped to 14 standards All 6 PIRLS texts rated likely to be on PARCC See pie chart

14 Distribution Across PIRLS Processes

15 ELA: PARCC Grade 10–PISA PARCCPISA Subclaims Reading Literature Reading Information Situations (Reading Contexts) Personal Public Occupational Educational Evidence Statements Indicate what students must demonstrate in support of the claims Describe what one can point to in a student’s work that substantiates that the standard has been mastered Aspects Mental strategies, approaches, or purposes used by readers o Access and retrieve o Integrate and interpret o Reflect and evaluate

16 ELA: PARCC Grade 10–PISA (continued) ComparisonNotes on MatchPassages PARCC Blueprint → PISA Framework 12% of PARCC Evidence Statements rated as No Match to PISA framework 52% mapped to 1 PISA aspect Integrate/interpret) -- PARCC Items and Passages → PISA Framework 1% of items rated as No Match to PISA framework 76% of items mapped to 1 PIRLS aspect (Integrate/interpret) 9 of 11 PARCC texts rated not likely to be on PISA PISA Items and Passages → PARCC Blueprint 20% of items rated as No Match to PARCC blueprint Mapped items spread in small percentages across 13 of the 38 standards; no items mapped to other 25 standards; only 1 item mapped to a vocabulary standard 15 of 17 PISA texts rated not likely to be on PARCC See pie chart

17 Distribution Across PISA Reading Aspects

18 Math: PARCC Grade 4–TIMSS PARCCTIMSS, Grade 4 Content Domains Operations and Algebraic Thinking Numbers and Operations in Base 10 Numbers and Operations-Fractions Measurement and Data Geometry Content Domains Number Geometric Shapes and Measures Data Display Evidence Statements Indicate what students must demonstrate in support of the claims Describe what one can point to in a student’s work that substantiates that the standard has been mastered Topic Area and Topics Each contain domain has two levels within it: o Topic Area  Topic

19 Math: PARCC Grade 4–TIMSS (continued) ComparisonNotes on Match PARCC Blueprint → TIMSS Framework 16% of PARCC Evidence Statements rated as No Match to TIMSS framework 76% mapped to 1 TIMSS content domain (Number) PARCC Items → TIMSS Framework 3% of PARCC items rated as No Match to TIMSS framework 79% mapped to 1 TIMSS content domain (Number) TIMSS Items → PARCC Blueprint 44% of TIMSS items rated as No Match to PARCC blueprint 26% of items mapped to Operations and Algebraic Thinking Remaining items spread across other PARCC domains, with percentages ranging from 5% to 8% See pie chart

20 Distribution Across TIMSS Grade 4 Content Domains

21 Math: PARCC Grade 8–TIMSS PARCCTIMSS, Grade 8 Content Domains The Number Systems Expressions and Equations Functions Geometry Statistics and Probability Content Domains Number Algebra Geometry Data and Chance Evidence Statements Indicate what students must demonstrate in support of the claims Describe what one can point to in a student’s work that substantiates that the standard has been mastered Topic Area and Topics Each contain domain has two levels within it: o Topic Area  Topic

22 Math: PARCC Grade 8–TIMSS (continued) ComparisonNotes on Match PARCC Blueprint → TIMSS Framework 22% of PARCC Evidence Statements rated as No Match to TIMSS framework 41% mapped to 1 TIMSS content domain (Algebra) PARCC Items → TIMSS Framework 27% of PARCC items rated as No Match to TIMSS framework 48% mapped to 1 TIMSS content domain (Algebra) TIMSS Items → PARCC Blueprint 83% of TIMSS items rated as No Match to PARCC blueprint Remaining items spread across PARCC domains, with percentages ranging from 1% to 6% Many PISA items judged as No Match are covered in other grades in CCSS-Mathematics and thus in other PARCC assessments See pie chart

23 Distribution Across TIMSS Grade 8 Content Domains

24 Math: PARCC Integrated Mathematics 1–PISA PARCCPISA Conceptual Categories Algebra Functions Geometry Statistics and Probability Content Domains Change and Relationships Space and Shape Quantity Uncertainty and Data Evidence Statements Indicate what students must demonstrate in support of the claims Describe what one can point to in a student’s work that substantiates that the standard has been mastered Content Topics Illustrative of content included in the assessment Cut across content domains

25 Math: PARCC Math 1–PISA (continued) ComparisonNotes on Match PARCC Blueprint → PISA Framework All PARCC Evidence Statements to mapped to PISA framework 79% of items were mapped to 1 PISA content category(Change and relationships) PARCC Items → PISA Framework 2% of PARCC items rated as No Match to PISA framework 83% of items were mapped to 1 PISA content category (Change and relationships) PISA Items → PARCC Blueprint 55% of PISA items rated as No Match to PARCC blueprint Remaining items spread across 4 of the 5 PARCC domains, with percentages ranging from 2% to 18% Many PISA items judged as No Match are covered in other grades in CCSS-Mathematics and thus in other PARCC assessments See pie chart

26 Distribution Across PISA Mathematics Content Categories

27 Conclusions For each of the assessment comparisons, the alignment could best be described as “Partial” PARCC items could often be mapped to international assessment frameworks; however: o This is often due to the “few and broad” nature of those frameworks o The proportion of PARCC items mapped to each domain is usually very discrepant from the targets given in the international framework Many international items could not be mapped to PARCC blueprints o PARCC blueprints are “many and specific” in terms of evidence statements o Particularly for mathematics, content was covered in other grades

28 Conclusions (continued) Based on these results, any inferences made about the relationship of results of the two assessments should be undertaken with caution

Thank you! Are there any questions? 29