1. 2% Modified Achievement Assessments Overview (C-5) Ohio Department of Education Office of Assessment September 2008 09-05-08 1

2. Federal Rule on Modified Achievement Standards Known as 2% Modified Assessment Allowances o Modified assessment of academic content standards o Proficient and above scores from such an assessment may be used in AYP accountability calculations capped at 2% of total test population Similar to the 1% alternate achievement standards for students with cognitive disabilities, but with some significant differences 2

3. Final Federal Rule (April 2007) The intended population for this assessment is the students on IEPs who are persistently low performing. Modified academic achievement standards must be based on a state’s grade-level academic content standards for the grade in which an eligible student with disabilities is enrolled. A state’s academic content standards are not what is modified. The expectations for whether a student has mastered those standards, however, may be less difficult than grade-level academic achievement standards. (pp. 17748, 17749) 3

4. Who the Students Are The student’s progress to date in response to appropriate instruction…, is such that, even if significant growth occurs, the IEP team is reasonably certain that the student will not achieve grade- level proficiency within the year covered by the student’s IEP. Section 200.1(e)(2)(ii) 4

5. Cook and Elliot, CCSSO Large-Scale Session, 2007 5

6. 6

7. Three-State Consortium Minnesota, Ohio and Oregon are working with AIR on the awarded three-year GSEG grant. 7

8. The Goals for the Consortium Identify the appropriate IEP student group Identify the appropriate test question design, the test format and an administration procedure Develop 2% Modified Assessments for grades 5–8 in reading and mathematics Set modified achievement standards Provide professional development for the assessment Define eligibility guidelines for participation in the assessment Monitor implementation 8

9. Each state takes away modifications that will work in that state. Each state will move forward with the AA-MAS, using its own item-review process. Each state will determine its own blueprint for the AA-MAS. Each state will conduct its own standard setting. Each state will conduct its own alignment study. State Outcomes from the Consortium 9

10. Consortium Outcome for Ohio Modify existing and write new items for the general education assessment for use on the AA-MAS Use the general education assessment blueprints for the AA-MAS Use the currently established review process for the items used on the AA-MAS Conduct alignment studies for each AA-MAS Conduct a standard setting resulting in modified achievement standards based on grade-level academic standards Ohio Actions: 10

11. Task 1. Conduct initial and subsequent meetings Task 2. Define the student population, conduct data analysis, conduct focus groups, develop student profiles Task 3. Draft Performance Level Descriptors Task 4. Develop Eligibility Guidelines and Decision Guide Task 5. Begin professional development plans for o Eligibility guidelines o Standards-based IEPs Task 6. Select modification strategies, conduct literature reviews, define modification strategies, modify items and forms Task 7. Conduct pilot tests for item modifications First Year Tasks 11

12. Second Year Tasks Task 1. Conduct subsequent meetings Task 3. Create initial definitions of modified achievement standards, refine PLD, draft definitions of modified achievement standards Task 5. Implement professional development for o Eligibility guidelines o Standards-based IEPs Task 7. Conduct pilot test of item modifications Task 9. Produce standard setting plans 12

13. Third Year Tasks Task 1. Conduct subsequent meetings Task 5. Implement professional development plans for o Instructional strategies o Monitoring Task 7. Field test modifications to items, forms and administration as well as validation of field tests Task 8. Produce item maps for operational test 13

14. Research Marion, Scott. A Technical Design and Documentation Workbook for Assessments Based on Modified Achievement Standards. KS, SD, OK, and MD Department of Education Web sites. Higgins, J., Russell, M., & Hoffmann, T. (2004). Examining the Effect of Computer- Based Passage Presentation on Reading Test Performance. Miranda, H., Russell, M., & Hoffmann, T. (2004). Examining the Feasibility and Effect of a Computer-Based Read-Aloud Accommodation on Mathematics Test Performance. Famularo, L. and Russell, M. (2007). Examining the utility of a prototype assessment for assessing students in the gaps. Dolan, R., Murray, E., & Burling, K. (2007). Providing students with choice: An exploratory study on the application of universal design principles to large-scale assessment of students with learning disabilities and English language learners. The Assessment and Accountability Comprehensive Center: Assessments based on modified academic achievement standards: critical considerations and implications for implementation. National Centers on Educational Outcomes. (2007). The assessment and accountability comprehensive center: special populations strand. National Centers on Educational Outcomes. (2007). A Seven-Step Process to Creating Standards-based IEPs. 14

15. Initial Data Mining Help define the student population Examine questions that work with the target population (still undefined) Analyze the data to help with the following: 15

16. Initial Data Mining Procedures Persistently low performing Persistently low performing with IEPs Other students with IEPs General education students Data from the three states’ general education administration were divided into four sets: 16

17. Initial Data Mining—Research Questions What items discriminate well for both IEP populations? What are the characteristics of the items that discriminate well for the low- performing group? What is the breakdown by race/ethnicity/gender/SES for the low- performing group? 17

18. Initial Data Mining—Research Questions What are the disability classifications in the persistently low-performing groups? What happens when we include those students who top out of the AA-SWD? What are the performances of students allowed different amounts of time in the regular classroom rather than disability classifications? 18

19. Stakeholder Input Focus groups o Collect input from the special education community who work with the target student population o Aggregate information across states Public survey o Eligibility guidelines o Decision flowchart 19

20. Goals for the Focus Group Begin defining the target population o Brief review of what the data show about low-performing students o Professional judgment about who these students are Review and suggest modification to test questions o Review questions that work well with the persistently low- performing students o Suggest modifications to test questions Review and suggest formats for test booklets Suggest modifications to administration procedures 20

21. Requirements for Targeted Population Have IEPs Identified as persistently low performing on the regular statewide assessments Receive on-grade-level instruction Students must conform to the following minimum requirements: 21

22. Eligibility Decision Flowchart (Handout) 22

23. Stakeholder Input — Online Survey Results of the survey: Who responded Requirements for eligibility Training needs 23

24. Purpose: lower barriers that students with disabilities face in demonstrating achievement of grade-level skills and knowledge Two strategies combine to meet this goal o The assessment incorporates modifications targeting specific characteristics commonly found in the target population o Second, performance on the assessment will be judged against modified achievement standards Ohio Fall 2008 Pilot Goal 24

25. Grade 7 Fall 2008 Pilot (Cont.) Two groups tested: target and general education students Each student takes Reading and Mathematics tests One consumable test booklet Seven forms, each with 42 multiple-choice questions Week of October 20, 2008, for Reading Week of October 27, 2008, for Mathematics 25

26. Item modifications: 1. Simplify language where appropriate 2. Inserted question within a box at appropriate place in passage 3. Present summary questions at the beginning and end of passages 4. Use bold print for key words or phrases Grade 7 Fall 2008 Reading Pilot 26

27. Item modifications: 1. Simplify language where appropriate 2. Include relevant pictures, tables and graphics to replace text (this is different from enlarging or shrinking graphics) 3. Enlarge and/or shrink graphics 4. Bold key words 5. Use scaffolding: Break multi-step items into individual steps, each with questions Grade 7 Fall 2008 Mathematics Pilot 27

28. Caveats Not every item will be modified in every way. In cases where the modification is not possible, the unmodified version of the item for the block will be used. Examples might include: o An item with a one-step process for the solution may not be modified using scaffolding. o An item with no original graphic may not be modified by using enlarged or simplified graphics. 28

29. Analysis Questions for Fall 2008 Pilot Target group will omit fewer modified items Target group will omit more unmodified items Do the modifications improve access? 29

30. Analysis Questions for Grade 7 Fall 2008 Reading and Mathematics Pilot Target group will guess less on modified items Target group will guess more on unmodified items Does performance exceed chance? 30

31. Analysis Questions for Fall 2008 Pilot Does performance improve and is discrepancy in performance reduced? Target group will perform better on modified items than on the unmodified items. Regular education students will perform about the same on both modified and unmodified items. Marginal maximum likelihood will be used for group comparisons rather than aggregating individual maximum likelihood estimates for each student. Multi-group IRT model that allows calibration around multiple groups that is a partial credit extension of the Bock & Zimowski, 1997. 31

32. Grade 7 Spring 2009 Field Test Will be similar to the fall 2008 pilot Interested in participating? Contact Barry Lowry: barry.lowry@ode.state.oh.us District name, IRN, school, and contact information 32

33. 1. Include relevant picture 2. Scaffolding: priming—asking understanding questions 3. Scaffolding: cueing with focusing questions 4. Scaffolding: graphic organizers 5. Scaffolding: provide thoughtful questions through passages focusing on summative items Spring 2009 Pilot—Reading Modifications 33 Item modifications:

34. 1.Include relevant pictures, tables, graphics that replace text 2. Scaffolding: priming — asking understanding questions 3. Group questions by content strand Spring 2009 Pilot — Mathematics Modifications 34

35. Thank You. Questions? 35