1. Interpreting Achievement Results Michael C. McKenna University of Virginia

2. Today’s Agenda  Discuss types and uses of data  Propose some formats for summarizing Reading First data  Analyze four student cases  Participate in Basic Organizational and Instructional Strategies study group

3. What kind of reading data do we have in Reading First schools?

4. Criterion-Referenced Tests Compare children against benchmarks Include CRCT and DIBELS Norm-Referenced Tests Compare students against national peers Include PPVT-3 and ITBS

5. Whew! That’s a lot of information! How can we make sense of it all? Start by thinking about four levels.

6. Child Classroom Grade School

7. NAEP Scores % Below Basic 199238 199440* 199840* 200041* 200236 200337 200536

8. Let’s start at the school level. The CRCT, ITBS, and DIBELS are all important indicators of schoolwide achievement.

9. We’ll build a chart that captures these results at a glance.

10. GradeDIBELS Mid-Year Target Skills ITBS Total Reading CRCT High Risk Some Risk Low Risk 1-3 NS 4-6 NS 7-9 NS Does Not Meet <300 Meets 300- 349 Exc. 350+ K (PSF) XXXXXX 1 (NWF) 2 (ORF) 3 (ORF)

11. How can you customize this chart to your own LEA? (10 minutes)

12. What conclusions can we reach from the CRCT and ITBS? Let’s start with some basics.

13. CRCT Basics The USDOE must approve how a state determines AYP. In Georgia, only the CRCT is approved. Changes in national policy may be enacted from year to year, so it’s important to stay current. Only grades 3-5 are considered.

14. CRCT Basics Three subgroups are considered:  Race  Special Education  English language learners

15. CRCT Basics For a subgroup to count, one of these conditions must be true: 1. If the school has 900 students (K-5), then 10% must be in a subgroup. Or2. If the school has fewer than 900 students, then 75 students must be in a subgroup.

16. CRCT Basics A school will be placed on “Needs Improvement” status if it fails to meet AYP two consecutive years in the same academic area. It does not matter which subgroup fails to achieve the benchmark.

17. CRCT Basics Averages for subgroups are not considered by grade level in determining AYP – only averages for grades 3-5 combined. However, it may be useful to track subgroups across grades, even if there are few in a given subgroup.

18. AYP Benchmarks Subject04 05 06 07 08 09 10 11 12 13 14 Rdg/ELA 60 66.7 66.7 66.7 73.3 73.3 73.3 80 86.7 93.3 100 Math 50 58.3 58.3 58.3 66.7 66.7 66.7 75 83.3 91.7 100

19. AYP Benchmarks Subject04 05 06 07 08 09 10 11 12 13 14 Rdg/ELA 60 66.7 66.7 66.7 73.3 73.3 73.3 80 86.7 93.3 100 Math 50 58.3 58.3 58.3 66.7 66.7 66.7 75 83.3 91.7 100 Kashinkerry 78 80.2 82.4 84.6 86.8 89.0 91.2 93.4 95.6 97.8 100

20. AYP Benchmarks Subject04 05 06 07 08 09 10 11 12 13 14 Rdg/ELA 60 66.7 66.7 66.7 73.3 73.3 73.3 80 86.7 93.3 100 Math 50 58.3 58.3 58.3 66.7 66.7 66.7 75 83.3 91.7 100 Kashinkerry 78 73 82.4 84.6 86.8 89.0 91.2 93.4 95.6 97.8 100 100 - 73 = 27 27  9 = 3.0

21. AYP Benchmarks Subject04 05 06 07 08 09 10 11 12 13 14 Rdg/ELA 60 66.7 66.7 66.7 73.3 73.3 73.3 80 86.7 93.3 100 Math 50 58.3 58.3 58.3 66.7 66.7 66.7 75 83.3 91.7 100 Kashinkerry 78 73 76. 79 82 85 88 91 94 97 100

22. Let’s look at the required subgroups.

23. GradeSubgroupN % 300+ 2 White Non Hispanic African American2678 Special Education1734 ELL4045 3 White Non Hispanic African American3079 Special Education1442 ELL3946 4 White Non Hispanic African American3168 Special Education1645 ELL4144 5 White Non Hispanic African American3469 Special Education1540 ELL3242 African American 30 31 +34 95

24. GradeSubgroupN % 300+ 2 White Non Hispanic African American2678 Special Education1734 ELL4045 3 White Non Hispanic African American3079 Special Education1442 ELL3946 4 White Non Hispanic African American3168 Special Education1645 ELL4144 5 White Non Hispanic African American3469 Special Education1540 ELL3242 Special Education 14 16 +15 45

25. GradeSubgroupN % 300+ 1 African American3287 Special Education1343 ELL33 2 African American2678 Special Education1734 ELL4045 3 African American3079 Special Education1442 ELL3946 4 African American3168 Special Education1645 ELL4144 5 African American3469 Special Education1540 ELL3242 Special Education 14 16 +15 45

26. GradeSubgroupN % 300+ 1 African American3287 Special Education1343 ELL33 2 African American2678 Special Education1734 ELL4045 3 African American3079 Special Education1442 ELL3946 4 African American3168 Special Education1645 ELL4144 5 African American3469 Special Education1540 ELL3242 ELL 39 41 +32 112

27. Now you try it!

28. GradeSubgroupN % 300+ 3 African American2679 Special Education1442 ELL3946 4 African American3168 Special Education1645 ELL4144 5 African American2869 Special Education1540 ELL3242 Hamilton Elementary serves 850 students in grades K-5. Would African Americans need to be considered as a subgroup for determining AYP?

29. GradeSubgroupN % 300+ 3 African American2679 Special Education1442 ELL3946 4 African American3168 Special Education1645 ELL3044 5 African American2869 Special Education1540 ELL2642 Franklin Elementary serves 970 students in grades K-5. Must English language learners be considered as a subgroup for determining AYP?

30. GradeSubgroupN % 300+ 3 African American2679 Special Education1442 ELL3946 4 African American3168 Special Education1645 ELL3044 5 African American2869 Special Education1540 ELL2642 Jefferson Elementary serves just 200 students in grades K-5. Must special education students be considered as a subgroup for determining AYP?

31. CRCT Cautions  Same-Grade Comparisons  Vertical Scaling  Year-to-Year Continuity

32. An Action Plan for the CRCT 1. Make a table for the past three years. 2. Contrast same-grade with cohort comparisons. 3. Consider disaggregated scores for key subgroups (special education categories, ethnicity, ELL). 4. Compute equal increases through 2014. 5. Communicate these goals to teachers at each grade level, and discuss steps that might help attain them.

33. How can I keep up to date on AYP? Start by visiting the Georgia DOE web site.

34. Now let’s look at the ITBS.

35. Points to Remember The ITBS is required for all schools at grades 3, 5 and 8 (and RF uses it in 1 and 2). There is no statewide testing window (but RF is at the end). The ITBS is not used to determine AYP. Cohort tracking is not possible unless a school elects to give the ITBS in non-required grades or a third-to-fifth-grade cohort is tracked. The ITBS permits national comparisons but does not gauge mastery of the Georgia Performance Standards.

36. Which norms shall we use? Percentile Ranks  Give a fairly precise estimate of standing compared with national age peers.  Cannot be averaged, subtracted, or easily compared.

37. Percentiles and Football G 50G Avoid differences in percentile ranks. Use NCEs or stanines instead.

38. Which norms shall we use? Stanines  Give a gross estimate of standing compared with national age peers.  Cannot be averaged.  Are not well suited to groups.

39. Interpreting Stanines Above AverageStanines 7-9 BorderlineStanine 6 AverageStanine 5 BorderlineStanine 4 Below AverageStanines 1-3 987654321987654321 { {

40. Which norms shall we use? Normal Curve Equivalents  Were developed to permit comparisons.  Can be subtracted and averaged.  Are statistically equivalent.

41. How NCEs are made NCEs “flatten” the normal curve so that differences are statistically equivalent.

42. Which norms shall we use? Grade Equivalents  Involve shaky estimates.  Compare students of different ages.  Have been condemned by IRA.

43. Three-Year ITBS Cohort Model 2003 Grade 32005 Grade 5 Percentile Rank 3038 Stanine 44 Normal Curve Equivalent 3843 A comparison can be made on the basis of those students present at both testings. But this would take time!

44. Three-Year ITBS Cohort Model 2003 Grade 32005 Grade 5 Percentile Rank 3038 Stanine 44 Normal Curve Equivalent 3843 Any comparison should focus on normal curve equivalents.

45. A chart can capture test data at the grade and classroom level. From the chart, you can identify trends, isolate problems, and recognize successes.

46. TeacherFallWinter PPVTISFLNF % > 80% < 80% High % Some % Low % High % Some % Low Jones Smith Myers All K Year-Long Kindergarten Classroom-Level Chart

47. TeacherFall NWFORF % High % Some % Low % High % Some % Low Jones Smith Myers All 2nd Year-Long Second-Grade Classroom-Level Chart Spring CRCT % Pass

48. A chart of this nature at each grade can help answer important questions... Which classrooms show growth in all areas? Which classrooms show growth In few areas? Which areas represent grade-level strengths? Which areas represent grade-level weaknesses? How well do DIBELS scores predict CRCT results?

49. Now let’s turn to the smallest, but most important, unit of data analysis – the child.