1. Gary W. Phillips American Institutes for Research CCSSO 2014 National Conference on Student Assessment (NCSA) New Orleans June 25-27, 2014 Multi State Alternate Assessment Collaborative: Measuring Growth in Alternate Assessments

2. Alternate Assessment State Collaborative

3. Three State Collaboratives National Center and State Collaborative (NCSC) hosted by the National Center on Education Outcomes (NCEO) at the University of Minnesota involving 24 states. Dynamic Learning Maps (DLM) hosted by the Center for Educational Testing and Evaluation (CETE) at the University of Kansas involving 13 states. Multistate Alternate Assessment Collaborative hosted by the American Institutes for Research (AIR) in Washington, DC involving 6 states. Reading, Writing (combine for ELA), Mathematics, Science & Social Studies 3 Multistate Alternate Assessment Collaborative

4. State (first Year)Grades Common Core State Standards Peer Review Approval Vertical Scale FT Designs -independent -operational -embeddedGrowth Psychometric Model NM (2007) R,M Gr. 3-4, 5-6, 7-8, 11-12 S Gr. 3-4, 6-8, 11-12 SS Gr. 11-12 Extensions to CCSS for R, M Yes IFT & EFTAcross grades 1 parameter Partial Credit SC (2007) R,M Gr. 3-5, 6-8, HS S Gr. 3-5, 6-8 Bio Gr. 10 SS Gr. 3-5, 6-8 Extensions to CCSS for R, M Yes IFT & EFTAcross grades 1 parameter Partial Credit DE (2011) R,M Gr. 2, 3-5, 6-8, 9-10 S Gr. 5, 8, 10 SS Gr. 4, 7, 9 Extensions to CCSS for R, M To be Submitted YesR,M IFT & EFT S,SS OFT & EFT Across grades fall to spring 1 parameter Partial Credit HI (2013) R,M Gr. 3-5, 6-8, 10 S Gr. 4, 8, 10 Extensions to CCSS for R, M To be Submitted YesOFT & EFTAcross grades 1 parameter Partial Credit OH (2013) R,M Gr. 3-5, 6-8, HS S Gr. 5, 8, HS SS Gr. HS Extensions to CCSS for R, M To be Submitted YesOFT & EFTAcross grades 1 parameter Partial Credit WY (2015) R, M, S Gr. 3-5, 6-8, HSExtensions to CCSS for R, M To be Submitted YesOFT & EFTAcross grades 1 parameter Partial Credit 4 Multistate Alternate Assessment Collaborative

5. Advantages of an Adaptive Alternate Assessment

6. Advantages of Adaptive Alternate Assessment Standardized administration which allows the scores from the test to be comparable from year-to-year Test difficulty is adapted to student ability Administered and scored by teachers High reliability of the scores Aligned to Common Core State Standards (in most states) Cheaper than portfolio assessments Less administration time (about one hour per content area) 6 Multistate Alternate Assessment Collaborative

7. Advantages of Adaptive Alternate Assessment Meets the same APA/AERA/NCME technical requirements as assessments of the general population Growth models iPad pilot 7 Multistate Alternate Assessment Collaborative

8. Administration

9. Multistate Alternate Assessment Collaborative 9 Individually administered to students by the Test Administrator Takes about one hour per subject A second rater independently scores the test for a sample of students (used to establish inter-rater reliability)

10. Calibration and Vertical Scale?

11. Typical Linking Design & Vertical Scale Grade Band Unique Tasks Linking Tasks Total for Operational Test Tasks in Each Grade-band Assessment Test Booklet 109 3 —12 6 – 86 3 12 3 – 59—12 Total Number of Tasks All Grades 24630

12. How do we Estimate Student Ability?

13. Student Ability Estimation 13 Partial Credit Rasch Model Pattern Scoring Multistate Alternate Assessment Collaborative

14. How are the Scores Reported?

19. Vertical Scale

20. Multistate Alternate Assessment Collaborative 20

21. Vertical Scale Multistate Alternate Assessment Collaborative 21

22. Growth

23. Multistate Alternate Assessment Collaborative 23

24. Multistate Alternate Assessment Collaborative 24

25. Multistate Alternate Assessment Collaborative 25

26. Multistate Alternate Assessment Collaborative 26

27. How do we Establish Performance Standards?

28. Standard Setting 28 Bookmark Procedure (or ID Matching) Workshop panel of broadly representative stakeholders Ordered Item Booklet (OIB) covers Extended Common Core State Standards Response Probability for ordering items Two Rounds Impact data Achievement Level Descriptors (ALDs) Benchmarking (if requested by the Department of Education) Vertical articulation Multiple Proficiency levels Multistate Alternate Assessment Collaborative

29. 29 Standard Setting Ordered Item Booklet. Most Difficult Item 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 Easiest Item Ordered Item Booklet

30. 30 Standard Setting Bookmark Procedure. Most Difficult Item 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 Easiest Item Ordered Item Booklet

31. 31 Standard Setting I.D. Matching Procedure. Most Difficult Item 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 Easiest Item Ordered Item Booklet Consistent with the PLD for “Meets Proficiency” Consistent with the PLD for “Approaches Proficiency” Threshold region

32. Standard Setting Articulation Multistate Alternate Assessment Collaborative 32

33. Standard Setting Articulation Multistate Alternate Assessment Collaborative 33

34. Standard Setting Impact Data Multistate Alternate Assessment Collaborative 34

35. Standard Setting Impact Data Multistate Alternate Assessment Collaborative 35 OIB PagePercentageOIB PagePercentageOIB PagePercentage 19923604530 29624584629 39425574727 49426564827 59227554925 69128545024 78829525123 88630515223 98431485323 108332485422 118133465521 128034445620 137835435719 147736425817 157537415916 7338396015 177139376113 186940376213 196841356312 20664235649 21644332657 22634431663

36. Standard Setting Benchmarking Against Previous Standard Multistate Alternate Assessment Collaborative 36 OIB PagePercentageOIB PagePercentageOIB PagePercentage 19923604530 29624584629 39425574727 49426564827 59227554925 69128545024 78829525123 88630515223 98431485323 108332485422 118133465521 128034445620 137835435719 147736425817 157537415916 7338396015 17* 7139376113 186940376213 196841356312 20664235649 21644332657 22634431663 *If this page were selected during the standard setting it would represent a level of proficiency comparable to the previous Proficient Standard. Since we do not want to lower standards as we transition to the new assessment, panelists should not set the Proficient Standard lower than this page.

37. Standard Setting Benchmarking Against General Education Standard Multistate Alternate Assessment Collaborative 37 OIB PagePercentageOIB PagePercentageOIB PagePercentage 19923604530 29624584629 39425574727 49426564827 59227554925 69128545024 78829525123 88630515223 98431485323 108332485422 118133465521 128034445620 137835435719 1477 36* 425817 157537415916 7338396015 177139376113 186940376213 196841356312 20664235649 21644332657 22634431663 *We want the standard for the alternate assessment to be as challenging for the alternate assessment population as the standard for the general assessment is for the general population. The page indicated above would represent such a standard for the alternate assessment.