Gary W. Phillips American Institutes for Research CCSSO 2014 National Conference on Student Assessment (NCSA) New Orleans June 25-27, 2014 Multi State Alternate Assessment Collaborative: Measuring Growth in Alternate Assessments

Alternate Assessment State Collaborative

Three State Collaboratives National Center and State Collaborative (NCSC) hosted by the National Center on Education Outcomes (NCEO) at the University of Minnesota involving 24 states. Dynamic Learning Maps (DLM) hosted by the Center for Educational Testing and Evaluation (CETE) at the University of Kansas involving 13 states. Multistate Alternate Assessment Collaborative hosted by the American Institutes for Research (AIR) in Washington, DC involving 6 states. Reading, Writing (combine for ELA), Mathematics, Science & Social Studies 3 Multistate Alternate Assessment Collaborative

State (first Year)Grades Common Core State Standards Peer Review Approval Vertical Scale FT Designs -independent -operational -embeddedGrowth Psychometric Model NM (2007) R,M Gr. 3-4, 5-6, 7-8, S Gr. 3-4, 6-8, SS Gr Extensions to CCSS for R, M Yes IFT & EFTAcross grades 1 parameter Partial Credit SC (2007) R,M Gr. 3-5, 6-8, HS S Gr. 3-5, 6-8 Bio Gr. 10 SS Gr. 3-5, 6-8 Extensions to CCSS for R, M Yes IFT & EFTAcross grades 1 parameter Partial Credit DE (2011) R,M Gr. 2, 3-5, 6-8, 9-10 S Gr. 5, 8, 10 SS Gr. 4, 7, 9 Extensions to CCSS for R, M To be Submitted YesR,M IFT & EFT S,SS OFT & EFT Across grades fall to spring 1 parameter Partial Credit HI (2013) R,M Gr. 3-5, 6-8, 10 S Gr. 4, 8, 10 Extensions to CCSS for R, M To be Submitted YesOFT & EFTAcross grades 1 parameter Partial Credit OH (2013) R,M Gr. 3-5, 6-8, HS S Gr. 5, 8, HS SS Gr. HS Extensions to CCSS for R, M To be Submitted YesOFT & EFTAcross grades 1 parameter Partial Credit WY (2015) R, M, S Gr. 3-5, 6-8, HSExtensions to CCSS for R, M To be Submitted YesOFT & EFTAcross grades 1 parameter Partial Credit 4 Multistate Alternate Assessment Collaborative

Advantages of an Adaptive Alternate Assessment

Advantages of Adaptive Alternate Assessment Standardized administration which allows the scores from the test to be comparable from year-to-year Test difficulty is adapted to student ability Administered and scored by teachers High reliability of the scores Aligned to Common Core State Standards (in most states) Cheaper than portfolio assessments Less administration time (about one hour per content area) 6 Multistate Alternate Assessment Collaborative

Advantages of Adaptive Alternate Assessment Meets the same APA/AERA/NCME technical requirements as assessments of the general population Growth models iPad pilot 7 Multistate Alternate Assessment Collaborative

Administration

Multistate Alternate Assessment Collaborative 9 Individually administered to students by the Test Administrator Takes about one hour per subject A second rater independently scores the test for a sample of students (used to establish inter-rater reliability)

Calibration and Vertical Scale?

Typical Linking Design & Vertical Scale Grade Band Unique Tasks Linking Tasks Total for Operational Test Tasks in Each Grade-band Assessment Test Booklet —12 6 – – 59—12 Total Number of Tasks All Grades 24630

How do we Estimate Student Ability?

Student Ability Estimation 13 Partial Credit Rasch Model Pattern Scoring Multistate Alternate Assessment Collaborative

How are the Scores Reported?

Vertical Scale

Multistate Alternate Assessment Collaborative 20

Vertical Scale Multistate Alternate Assessment Collaborative 21

Growth

Multistate Alternate Assessment Collaborative 23

Multistate Alternate Assessment Collaborative 24

Multistate Alternate Assessment Collaborative 25

Multistate Alternate Assessment Collaborative 26

How do we Establish Performance Standards?

Standard Setting 28 Bookmark Procedure (or ID Matching) Workshop panel of broadly representative stakeholders Ordered Item Booklet (OIB) covers Extended Common Core State Standards Response Probability for ordering items Two Rounds Impact data Achievement Level Descriptors (ALDs) Benchmarking (if requested by the Department of Education) Vertical articulation Multiple Proficiency levels Multistate Alternate Assessment Collaborative

29 Standard Setting Ordered Item Booklet. Most Difficult Item Easiest Item Ordered Item Booklet

30 Standard Setting Bookmark Procedure. Most Difficult Item Easiest Item Ordered Item Booklet

31 Standard Setting I.D. Matching Procedure. Most Difficult Item Easiest Item Ordered Item Booklet Consistent with the PLD for “Meets Proficiency” Consistent with the PLD for “Approaches Proficiency” Threshold region

Standard Setting Articulation Multistate Alternate Assessment Collaborative 32

Standard Setting Articulation Multistate Alternate Assessment Collaborative 33

Standard Setting Impact Data Multistate Alternate Assessment Collaborative 34

Standard Setting Impact Data Multistate Alternate Assessment Collaborative 35 OIB PagePercentageOIB PagePercentageOIB PagePercentage

Standard Setting Benchmarking Against Previous Standard Multistate Alternate Assessment Collaborative 36 OIB PagePercentageOIB PagePercentageOIB PagePercentage * *If this page were selected during the standard setting it would represent a level of proficiency comparable to the previous Proficient Standard. Since we do not want to lower standards as we transition to the new assessment, panelists should not set the Proficient Standard lower than this page.

Standard Setting Benchmarking Against General Education Standard Multistate Alternate Assessment Collaborative 37 OIB PagePercentageOIB PagePercentageOIB PagePercentage * *We want the standard for the alternate assessment to be as challenging for the alternate assessment population as the standard for the general assessment is for the general population. The page indicated above would represent such a standard for the alternate assessment.