1. 11 2010 Winter Education Conference Consequential Validity Using Item- and Standard-Level Residuals to Inform Instruction

2. 2 Consequential Validity Districts must adopt instructional materials that are aligned to the Alaska Grade Level Expectations: –Districts have autonomy to select instructional material. –What can a district do to evaluate the effectiveness of instructional materials with the SBA?

3. 3 Analysis This presentation was designed to illustrate how the use of residual analysis could help districts and schools answer questions about the effectiveness of instruction. Commonly, the percent of students in each achievement level and the mean and standard deviation of scale scores are the only information available to describe student performance.

4. 4 Topics What is a Residual? How Residuals Control for Achievement Levels Interpreting Residual Plots Item-Level Residual Plots Standard-Level Residual Plots Comparing Cohort Standard Residuals Text Book Residual Plots

5. 5 Introduction The description of the results of a test for a district or school, or gender, ethnic, or native language group can go far beyond reporting the mean and the standard deviation for the group. Looking at subgroups of items or students can often reveal interesting patterns that have important curricular implications. This type of analysis is called residual analysis. This demonstration looks at the residuals for selected districts and common textbook for two grades and is only a small sample of the types of questions that can be investigated.

6. 6 What is a Residual? A residual is the difference between what you observe and what you expect. For each person/item interaction you can observe a 0 (wrong) or a 1 (correct). (Only multiple-choice items are used in this presentation.) Each students achievement is based on the total number of correct answers on the test. Each items difficulty is based on the total number of correct responses statewide.

7. 7 What is a Residual? The students achievement level and the items difficulty combine to produce the expected score for that student/item interaction. The expected score will be between 0 and 1. This is easily interpreted as the percentage of times that the student would answer questions of this difficulty correctly.

8. 8 What is a Residual? If the student and the item are at the same location on the continuum (construct), the student will have a 50% chance of answering the item correctly. If the student and the item are at the same location, the residual will be +0.5, if the answer is correct, or - 0.5, if the answer is incorrect. This will produce a residual matrix for every student/item interaction (There are about one-half million per test/grade combination.). For Grade 7 Mathematics this would be a 56 column (items) by about 9500 row (students) matrix.

9. 9 What is a Residual? The beauty of this matrix is that it allows you to sum any meaningful collection of rows or columns. For instance, you could combine all of the students into their respective districts. This would give you district-level results. Or you could combine all of the items that constitute the Estimation and Computation standard in Grade 7 Mathematics. This would give you a standard-level result.

10. 10 How Residuals Control for Achievement Levels Because each residual is computed using the students achievement level, the resulting residuals are free from the distribution of student achievement. This means that comparisons across combinations of rows, like classrooms, buildings, districts, or demographic factors such as gender, ethnicity, or native language will be independent of the achievement level for that group. This property allows educators to use the residual matrix to answer important questions about the effectiveness of the curriculum at the standard level.

11. 11 How Residuals Control for Achievement Levels Person District School ESL i1 i2 i3 i4 P1 1 1 N +.3 +.4 +.5 -.4 P2 1 2 Y +.1 +.2 +.3 +.4 P3 1 2 N +.5 -.4 +.7 -.2 P4 2 1 N +.2 -.7 +.4 -.5 P5 2 1 N +.4 +.5 -.4 +.7 P6 2 2 Y -.4 -.3 +.8 -.1 This is an example of a small residual matrix.

12. 12 Interpreting Residual Plots The expected value for a residual or sum of residuals is zero. Positive values mean the student(s) are doing better than expected. Negative values mean that the student(s) are not doing as well as expected. The units of the scale are the proportion of correct responses. An average residual +0.05 means that 5 percent more students answered correctly than expected. A -0.10 means that 10 percent fewer students answered correctly than expected.

13. 13 Selection of Aggregates Eight medium to large districts were chosen for analysis. They are used simply to illustrate the methodology. Three common text books used in Grade 6 mathematics instruction were chosen for analysis. Again, they are used simply to illustrate the methodology. Not all districts use the textbooks in the same manner. Therefore, the results may not generalize to a specific district.

14. 14 Item-Level Residual Plots Item by Multiple Districts Plot

15. 15 Practice Test Item Similar to Item 46

16. 16 Item-Level Residual Plots Item by Single District Plot

17. 17 Item-Level Residual Plots Item by Single District Plot

18. 18 Item-Level Residual Plots Item by Single District Plot

19. 19 Item-Level Residual Plots Item with Standards by Single District Plot e = estimation & computation, f = functions & relationships, g = geometry, m = measurement, n = numeration, s = statistics & probability

20. 20 Standard-Level Residual Plots Standard by District Plot – Grade 7

21. 21 Standard-Level Residual Plots Standard by District Plot – Grade 6

22. Standard-Level Residual Plots Comparison of Grade 6 and Grade 7 22

23. 23 Standard-Level Residual Plots Text by Standard Plot

24. 24 What Next? What are some questions districts might ask after looking at residual plots? What are some district actions or next steps? What cautions need to be applied to the interpretation of the results?

25. 25 Where can a district get more?