1. Using the PMRN, ECI and EI Indices, and Other Information to Make Effective Decisions Elizabeth Crawford, MS, CCC-SLP Director of Interventions The Florida Center for Reading Research

2. 2 Agenda Risk Levels and Instructional Levels of DIBELS Use Data to: interpret ECI and EI tables calculate ECI and EI indices at the class level manage school level intervention resources monitor progress of specific groups of students Concluding Thoughts and Resources

3. 3 Risk Levels and Instructional Levels of DIBELS

4. 4 Risk Level Key

5. 5

6. 6 SY0607 Florida DIBELS Recommended Level of Instruction Decision Rules http://www.fcrr.org/assessment/pdf/ril_decisionrules_0607.pdf

7. 7 Various Uses of Data

8. 8 Use Data to… Interpret ECI and EI tables Calculate ECI and EI indices at Class Level Manage school level intervention resources Monitor progress of specific groups of students

9. 9 School Level Indices

10. 10 ECI Definition Effectiveness of Core Instruction (ECI) — this index shows the percentage of students who began the year reading “at grade level” and are continuing to meet grade level expectations at the mid-year or end of year assessment. This index tells you whether the core instruction is sufficiently powerful to produce a year’s growth in reading skill for a year’s worth of instruction for all students who begin the year at grade level.

11. 11 ECI – “School” 91% of the 45 kindergarteners who started the year on grade level remained on grade level.91 x 45 = 41 children 41 out of 45 children remained on grade level.97 x 75 = 73 children 73 out of 75 children remained on grade level, 2 children did not remain on grade level In first grade, my school is doing equal to or better than 96% of the RF schools on keeping our 1 st grade students on grade level.

12. 12 EI Definition Effectiveness of Interventions (El)* —this index shows the percentage of your students who began the year at some level of risk for reading difficulties (reading below grade level) but who have grown rapidly enough to advance to a lower level of risk by the mid year or end of year assessment (i.e. they move from intensive to strategic, or from strategic to grade level). This index, and the two explained below, provide information about how effectively intervention programs are working to accelerate the development of students who are lagging behind in reading growth.

13. 13 EI – “School” T 81% of the 42 kindergartners who started the year at some level of risk, have made gains and are now at a lower level of risk..81 x 42 = 34 students In first grade, my school is doing equal to or better than 44% of the RF schools in moving children to strategic or grade level..23 x 30 = 7 students 7 out of 30 3rd graders who started at some level of risk have moved to a lower level of risk. 23 students are still at some risk.

14. 14 EI-S Definition Effectiveness of Interventions for Strategic Students (EI-S) — this index shows the percentage of students that moved from the “strategic” risk level to grade level.

15. 15 EI – S “School”.56 x 27 kindergarten students who started the year at “strategic” moved to “grade level” = 15 27-15 = 12 are still “strategic” In 1 st grade, when compared to the other 583 RF schools in FL, We are doing equal to or better than 48% of them at moving first grade “strategic” students to “grade level” In 3rd grade, 62% of the 16 children (10) who started at “strategic” are now at “grade level”

16. 16 EI-I Definition Effectiveness of Interventions for Intensive Students (EI-I) — this index shows the percentage of students that moved from the “intensive” risk category to the “strategic” risk level or to grade level.

17. 17 EI – I “School” 93% of our 14 kindergarten students who started the year at “intensive” made progress to a lower risk level..93 x 14 = 13 students made growth, 1 is still “intensive” In second grade, we are doing an equal or better job than 47% of the RF schools at moving our children out of the “intensive” category. In all grades, K-3, of the 46 students who started at “intensive” 50% (or 23) of them made growth to “strategic” or “grade level.”

18. 18 State Averages for ECI and EI Indices: Numbers to consider when comparing data across years Due to different level of difficulty with passages from assessment to assessment and year to year, it is sometimes difficult to compare indices across assessments and years.

19. 19 Background For all subtests of DIBELS, the same forms are used at each assessment from year to year, except Oral Reading Fluency (ORF). In order to decrease familiarity with passages, a rotation system was used – passages rotated on a four year cycle.

20. 20 Background This rotation systems began in 2003 and was created based on the assumption that all DIBELS ORF passages were roughly the same level of difficulty. Also, three passages are used with a median score to further to protect against difficulty.

21. 21 ECI State Averages IndexEnd of Year 07-08 Middle of Year 07-08End of Year 06-07 ECI Mean ECI_K 91.4 ECI_K 84.0 ECI_K 90.1 ECI_1 71.3 ECI_1 74.3 ECI_1 73.0 ECI_2 74.5 ECI_2 82.9 ECI_2 74.8 ECI_3 82.9 ECI_3 97.4 ECI_3 87.3 ECI_ALL 78.0 ECI_ALL 82.7 ECI_ALL 79.5 Higher when compared to EOY 07-08 Lower when compared to EOY 07-08

22. 22 EI State Averages Higher when compared to EOY 07-08 Lower when compared to EOY 07-08 IndexEnd of Year 07-08 Middle of Year 07-08End of Year 06-07 EI Mean EI_K 72.5 EI_K 57.7 EI_K 69.8 EI_1 24.0 EI_1 22.1 EI_1 25.5 EI_2 10.6 EI_2 7.8 EI_2 11.6 EI_3 35.1 EI_3 59.9 EI_3 33.2 EI_ALL 42.4 EI_ALL 45.4 EI_ALL 39.3

23. 23 EI_S State Averages Higher when compared to EOY 07-08 Lower when compared to EOY 07-08 IndexEnd of Year 07-08 Middle of Year 07-08End of Year 06-07 EI-S Mean EIS_K 73.2 EIS_K 52.8 EIS_K 70.7 EIS_1 23.4 EIS_1 20.7 EIS_1 25.9 EIS_2 11.0 EIS_2 9.7 EIS_2 13.4 EIS_3 27.8 EIS_3 64.8 EIS_3 31.3 EIS_ALL 42.1 EIS_ALL 44.5 EIS_ALL 40.3

24. 24 EI_I State Averages Higher when compared to EOY 07-08 Lower when compared to EOY 07-08 IndexEnd of Year 07-08 Middle of Year 07-08End of Year 06-07 EI-I Mean EII_K 70.3 EII_K 68.4 EII_K 67.3 EII_1 25.9 EII_1 26.8 EII_1 26.2 EII_2 10.5 EII_2 4.5 EII_2 9.2 EII_3 43.8 EII_3 55.0 EII_3 36.3 EII_ALL 44.5 EII_ALL 47.9 EII_ALL 38.2

25. 25 Points to Remember We recognize that this variability in difficulty makes it difficult to compare classroom and school growth in ORF from one year to next so we are taking steps to correct this in the next set of early progress monitoring measures being developed for use in Florida.

26. 26 Points to Remember If your school wants to know how you are doing in improving your ECI and EI indices, the best way is to examine your percentile rank compared to other Reading First schools in the state. If your percentile rank increased from last year to this year, then you are doing better than average in improving your students’ performance on these indices. Perhaps more important than the year to year change in ECI and EI, though, is relative standing on these measures. If you have a very low percentile rank on any of these indices, that means that your instruction is relatively weak when compared to that provided in other Reading First schools.

27. 27 This school year, 08-09, PMRN will generate this School RLI report with your ECI and EI data. Percentile Ranks will only be generated once all Reading First School Data is entered. The tables on this report are very similar to the tables that were explained above.

28. 28

29. 29 Calculating the Class Level ECI, EI-I, and EI-S How do you do it? AND…what does it all mean?? Refer to “DIBELS Data Analysis Conference” Worksheet Handout

30. 30

31. 31 Key ‘follow-up’ sections on worksheet Items Discussed Roles and Responsibilities Teacher will… Reading Coach will… Principal will…

32. 32 Class Recommended Level of Instruction Report This report provides a summary of the students’ overall progress. It can be used to get an overall sense of instructional levels in the class and to calculate the Effectiveness of Core Instruction (ECI) index and the three Effectiveness of Intervention (EI) indices.

33. 33 Class Recommended Level of Instruction Report

34. 34 Class Recommended Level of Instruction Report This is the same type of report with “all” students included. Students 5, 6, 7, and 19 would not be used to calculate the ECI and EI indices because they were not in the school for all three assessment periods. This is why we select the full year only function for calculation.

35. 35 Class Recommended Level of Instruction Report You can use the pie charts to help you fill out the top part of the DIBELS worksheet Initial Level = Low risk Strategic Level = Moderate Risk Intensive Level = High Risk Assessment #3 % Initial = 44 % Strategic = 50 % Intensive = 6

36. 36 Class Recommended Level of Instruction Report To calculate ECI – Effectiveness of Core Instruction for the year, use only those students that were at Initial Instruction during the 1st assessment and remained in the class throughout the year (that is why we select full year only). Divide the number of the students that finished at Initial Instruction for the 3rd assessment (5) by the number of students that were at Initial Instruction for the 1st assessment (9). The higher the percentage the better.

37. 37 Class Recommended Level of Instruction Report Effectiveness of Core Instruction # stayed initial 5 --------------------- = ------- # started initial 9 5 --- = 56% ECI = 56% 9 Target Students: Students 3, 6, 8, 9 (Initial to Strategic)

38. 38 Class Recommended Level of Instruction Report To calculate EI – Effectiveness of Intervention for the year, use only those students that were in the class for the first assessment and the last assessment (full year only). Take the number of students who finished the year at ‘initial’ or ‘strategic’ (if they started at intensive) level by the students who started the year at some level of risk (intensive or strategic) The higher the percentage the better.

39. 39 Class Recommended Level of Instruction Report Effectiveness of Intervention # moved to initial or strategic from Intensive2 --------------------- = ------- # started strategic & intensive 7 2 --- = 29% EI = 29% 7 Target Students: Students 4, 7, and 10 (strategic to intensive during assessment 2)

40. 40 Class Recommended Level of Instruction Report Effectiveness of Intervention- Strategic # moved to initial 1 --------------------- = ------- # began strategic 5 1 --- = 20% EI-S = 20% 5 Target Students: Students 4, 7, and 10 (strategic to intensive during assessment 2)

41. 41 Class Recommended Level of Instruction Report Effectiveness of Intervention - Intensive # moved to strategic or initial 1 --------------------- = ------- # began intensive 2 1 --- = 50% EI-I = 50% 2 Target Students: Student 5

42. 42 What do all of these indices mean for my school and teachers? This particular teacher in 1 st grade is struggling both with maintaining children at grade level and reaching her struggling readers: ECI = 56% EI = 29% EI-S = 20% EI-I = 50% You may want to first determine if other first grade teachers have similar indices and if so, you want to examine your Core Reading Program and especially your Intervention Programs You can look at the School Recommended Level of Instruction Report to see how different grade levels are doing by comparing the pie charts from one assessment period to the next The goal is to see the green increase and the red decrease – to make the pie chart look more like an olive!!

43. 43 Use Data to… monitor progress of specific groups of students

44. 44

45. 45 School Demographic Reports Demographic data is available, by assessment period, at the school and district levels for individual measures and the Recommended Level of Instruction. The data file allows all of the data to be downloaded so that Users can complete their own analyses.

46. 46

47. 47

48. 48 School Demographic Reports You can also display a bar graph for a specific demographic group from this report. The next slides shows a bar graph for kindergarten students at the third assessment by Ethnicity. The total number of students in each category is in ( ) next to the group name. For example Hispanic (17)

49. 49

50. 50

51. 51 Use Data to… manage school level intervention resources

52. 52 School Level Reports “The Chart” Highlight: School Status (“Home Base”) School Demographics

53. 53 “The Chart” Use the chart as a “quick start” guide to the PMRN for the School Level Reports Refer to separate handout The PMRN User’s Guide is the complete manual and should be read and referenced for specific questions and information

54. 54

55. 55

56. 56 Using the School Status Report – “Home Base” Make a list of school wide resources Who needs extra support? (High Risk, Moderate Risk, Low Risk) Which Grades? Which Teachers? Which skills need to be emphasized in professional development?

57. 57

58. 58 School Status Report – “Home Base”

59. 59 School Status Report – “Home Base”

60. 60 School Status Report – “Home Base”

61. 61 Navigating from the School Status Report – “Home Base” School Grade Summary – Click on the words ‘grade summary’ under the grade want to see Class Status – From the extended view, click on the teacher’s name School Progress – From Grade Summary, click on ‘show progress report’ at bottom Class Progress – From the School Progress, use active header menu to select teacher/class of interest School Comparison – From School Progress, click on ‘show school comparison’ at bottom ** Any time you click on the number below a specific measure you will get a progress report and depending on the level of the report you started from will determine which level of progress report you get

62. Concluding Thoughts and Resources

63. 63 Concluding Thoughts Use PMRN Reports to make school, classroom, and student level decisions: Monitor progress Determine which students need more intensive instruction Offer instructional strategies to the teacher Determine types of professional development from which specific teachers could benefit By asking some of the following questions…

64. 64 Is the core reading program implemented with fidelity? Effectiveness? Are the interventions being implemented with fidelity? Effectiveness? Is there a need for more professional development for a particular area of reading? Is there a need for more intensive coaching? Are materials allocated to maximize learning? Is personnel use maximized to focus on reading instruction? Is the instructional day scheduled such that reading is THE priority? Important Considerations

65. 65 Florida Center for Reading Research 227 North Bronough Street, Suite 7250 Tallahassee, FL 32301 (850) 644-9352 - Phone (850) 644-9085 (Fax) www.fcrr.org PMRN Help Desk 850-644-0931 (Voice) 1-866-471-5019 (Toll Free) 850-645-1700 (Fax) helpdesk@fcrr.org User’s guide http://www.fcrr.org/pmrn/userguides.htm

66. 66 Questions

67. 67 Thank You! Liz Crawford, MS, CCC-SLP Director of Interventions Florida Center for Reading Research ecrawford@fcrr.org After the Leadership Conference, this presentation can be found on www.fcrr.org under my name on the Presentations and Publications section.