Illustration of a Validity Argument for Two Alternate Assessment Approaches Presentation at the OSEP Project Directors’ Conference Steve Ferrara American Institutes for Research August 1, 2006

2 Goal Illustrate planning for the validation process for large-scale assessments using standards- based alternate assessments from two states Use selected examples from the paper Standards and Assessment Approaches for Students with Disabilities Using a Validity Argument

3 Over-arching concept Provide evidence to support intended inferences about students Consideration of assessment design (i.e., tasks, administration conditions, scoring) Plans for collecting procedural and empirical evidence Important specific principles and recommendations in the paper

4 Examples from the paper Massachusetts alternate assessment portfolio (Weiner, 2002) Oregon performance tasks (Tindal et al., 2003) Mathematics, grades 3-5 Intended inferences The assessment adequately reflects the domain of knowledge and skills for the construct The assessment accurately identifies students’ level of proficiency in mathematics

5 Procedural evidence Test design and development process Quality of the items and tasks Assemblage of items/tasks/evidence into an assessment Administration and scoring process

6 Empirical evidence Alignment between the alternate content standards and the assessment items/tasks/evidence (and linkage to grade level/band standards) Item/task functioning Reliability of scoring and test score interpretations Internal relations among items and tasks Response processes External relations with other measures

7 Target math standards Massachusetts standards Grades three and four standards that focus on number sense (seven objectives) and operations (three objectives) Oregon mathematics standards Numbers, Computation and Operations—Grades four to five

8 MA: Possible assessment strategies and portfolio products Addressing access skill(s) (skills embedded in academic instruction) Alice participates in this activity by assembling money envelopes paired with pictures. Alice works with a classmate who counts the money needed for each item and helps Alice place the correct amount into its corresponding envelope. Alice exchanges these envelopes when making a purchase.

9 Possible portfolio products (cont.) Teacher note describing the work accomplished by Alice and her classmate Data collected on Alice’s ability to assemble money envelopes and exchange correct envelopes when making a purchase Videotape of Alice making a purchase Alice’s choice of money envelopes selected for her portfolio

10 Oregon item Standard: Read, write, order, model, and compare whole numbers up to 1,000,000, common fractions, and decimals up to hundredths. Practice Item 24: Find the missing number in the pattern ___ 20.8 (A) 7.8 (B) 10.4 (C) 13.0 (D) 15.6 Alternate Assessment Task 11: Order Numbers Present the number cards in this order: 3, 1, 8, 6. Say: Place these numbers in order from smallest to largest.

11 Summary (MA) Assemble money envelopes with a classmate, make purchases Teacher observations of Alice working, videotape of making purchases (OR) Order numbers from smallest to largest

12 Test development process MassachusettsOregon Does the actual evidence described in the possible assessment strategy fully reflect this construct? Have the tasks been adequately developed and assembled into an alternate assessment?

13 Test administration and scoring MassachusettsOregon How well conducted are the test administration and scoring procedures? Are teachers sufficiently trained in administering and scoring the tests (especially because responses may be scored as partially correct and not just correct or incorrect)? Does the student independently complete work or is the teacher part of this process? If so, to what extent?

14 Alignment and construct representation MassachusettsOregon Are enough pieces of evidence present to represent the domain and avoid construct under- representation? Are enough tasks present to represent the domain and avoid construct under- representation? How closely is the alternate assessment aligned to the state content standards in categorical concurrence, depth of knowledge, range of knowledge, and balance of representation?

15 Rater accuracy and score reliability MassachusettsOregon How accurate is the scoring by trained professional scorers? How accurate is the scoring by trained teachers? How consistently and accurately do scores categorize students into performance categories? Which facets of the assessment process influence scores most (e.g., tasks, raters, administration conditions, occasions)?

16 Conclusion The same types of validity questions apply for all (alternate) assessment approaches How the questions are posed and the evidence relevant to those questions may differ Intended inferences, corresponding validity questions, and evidence: Identify during the conceptualization, design, and development process Pursue during development and as part of implementation

17 References Tindal, G., McDonald, Tedesco, M., Glasgow, A., & Almond, P., Crawford, L., Hollenbeck, K. (2003). Alternate assessments in reading and math: Development and validation for students with significant disabilities. Exceptional Children, 69(4), 481–494. Wiener, D. (2002). Massachusetts: One state's approach to setting performance levels on the alternate assessment. (Synthesis Report 48). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved Dec. 8, 2005 from