Using the PMRN and ECI and EI Indices to Make Effective Decisions Elizabeth Crawford, MS, CCC-SLP Director of Interventions The Florida Center for Reading Research

2 Agenda General Uses of Data PMRN 101 Highlights of DIBELS Use Data to: manage school level intervention resources monitor progress of specific groups of students run an effective class/student level data meeting interpret ECI and EI tables Concluding Thoughts and Resources

3 General Uses of Data

4 Barriers to Using Data to Inform Instruction Grunwald Associates (out of Maryland) surveyed 353 small, medium and large school districts asking them “What are the barriers to you using data-driven decision making more effectively?” The next slides list the results

5

6 Barriers to more effective data-driven decision making Lack of training50% Interoperability – systems that can’t share data42% Lack of understanding about what to do with the data39% Absence of clear priorities36% Failure to collect data uniformly35% Outdated technology31% Incomplete or incorrect data24% Timing of data collection24% User interface is too complicated to understand reports 22%

7 PMRN 101

8 What is the PMRN? (Progress Monitoring and Reporting Network) A Web-based data management system that provides: a convenient place for entering and organizing the results of student assessments a secure, centralized, easily accessible location for the storage of student information a tool for timely and helpful reports so that educators can effectively analyze data, plan instruction, and communicate student progress

9 The PMRN is located at To Sign In to the PMRN, you will need the User Name and Password that were ed to you, as well as your Identifier which was received in a letter from the FCRR. Signing In to the PMRN Please note the “S” in https. The “S” indicates a secure network and is required for your web browser to find the PMRN.

10

11 Active Header Menu Availability of information is limited by the Users access level. The level of access determines if the User can select all or specific grades, teachers, students, and/or tests.

12 Types of Reports and how data is reported Status - Table Progress – Box and Whiskers and Pie chart (except at the student level which only has Box and Whiskers) Class Progress Tracking – Bar graph Historical - Box and Whiskers Risk Level - Table Recommended Level of instruction – Table and/or Pie chart Student Recommended Instructional Level - Table Grade Summary – Box and Whiskers Comparison - Box and Whiskers Demographics – Table and Bar graph Cumulative - Table Ongoing Progress Monitoring – Line graph Year End Outcome – Pie chart and Table

13 Box & Whiskers Graph The Box & Whiskers Graph is a type of distribution report. It is used to report how an individual or group compares to a larger group. The Box & Whiskers used in the PMRN is a modified version of the those that report quartiles.

14 Highlights of DIBELS

15 GRADEDIBELS MEASUREREADING COMPONENT ASSESSED Kindergarten Initial Sounds, Phoneme Segmentation Letter Naming, Nonsense Words Phonemic Awareness Phonics First Phoneme Segmentation Letter Naming, Nonsense Words Oral Reading Phonemic Awareness Phonics Reading Fluency SecondNonsense Words Oral Reading Phonics Reading Fluency ThirdOral ReadingReading Fluency

16 Risk Level Key

17

18 SY0607 Florida DIBELS Recommended Level of Instruction Decision Rules

19 Use Data to… Manage school level intervention resources Monitor progress of specific groups of students Run an effective class/student level data meeting Interpret ECI and EI tables

20 Strong Leadership is critical to success The great leaders are like the best conductors - they reach beyond the notes to reach the magic in the players. - Blaine Lee

21 Use Data to… manage school level intervention resources

22 School Level Reports “The Chart” Highlight: School Status (“Home Base”) School Demographics

23 “The Chart” Use the chart as a “quick start” guide to the PMRN for the School Level Reports Refer to separate handout The PMRN User’s Guide is the complete manual and should be read and referenced for specific questions and information

24

25

26 Using the School Status Report – “Home Base” Make a list of school wide resources Who needs extra support? (High Risk, Moderate Risk, Low Risk) Which Grades? Which Teachers? Which skills need to be emphasized in professional development?

27

28 School Status Report – “Home Base”

29 School Status Report – “Home Base”

30 School Status Report – “Home Base”

31 Navigating from the School Status Report – “Home Base” School Grade Summary – Click on the words ‘grade summary’ under the grade want to see Class Status – From the extended view, click on the teacher’s name School Progress – From Grade Summary, click on ‘show progress report’ at bottom Class Progress – From the School Progress, use active header menu to select teacher/class of interest School Comparison – From School Progress, click on ‘show school comparison’ at bottom ** Any time you click on the number below a specific measure you will get a progress report and depending on the level of the report you started from will determine which level of progress report you get

32 School Status Report – “Home Base”

33 School Grade Summary Report

34 Navigating from the School Status Report – “Home Base” School Grade Summary – Click on the words ‘grade summary’ under the grade want to see Class Status – From the extended view, click on the teacher’s name School Progress – From Grade Summary, click on ‘show progress report’ at bottom Class Progress – From the School Progress, use active header menu to select teacher/class of interest School Comparison – From School Progress, click on ‘show school comparison’ at bottom ** Any time you click on the number below a specific measure you will get a progress report and depending on the level of the report you started from will determine which level of progress report you get

35 School Status Report – “Home Base”

36 Class Status Report

37 Navigating from the School Status Report – “Home Base” School Grade Summary – Click on the words ‘grade summary’ under the grade want to see Class Status – From the extended view, click on the teacher’s name School Progress – From Grade Summary, click on ‘show progress report’ at bottom Class Progress – From the School Progress, use active header menu to select teacher/class of interest School Comparison – From School Progress, click on ‘show school comparison’ at bottom ** Any time you click on the number below a specific measure you will get a progress report and depending on the level of the report you started from will determine which level of progress report you get

38 School Grade Summary Report

39 School Progress Report

40 Navigating from the School Status Report – “Home Base” School Grade Summary – Click on the words ‘grade summary’ under the grade want to see Class Status – From the extended view, click on the teacher’s name School Progress – From Grade Summary, click on ‘show progress report’ at bottom Class Progress – From the School Progress, use active header menu to select teacher/class of interest School Comparison – From School Progress, click on ‘show school comparison’ at bottom ** Any time you click on the number below a specific measure you will get a progress report and depending on the level of the report you started from will determine which level of progress report you get

41 School Progress Report

42 Class Progress Report

43 Navigating from the School Status Report – “Home Base” School Grade Summary – Click on the words ‘grade summary’ under the grade want to see Class Status – From the extended view, click on the teacher’s name School Progress – From Grade Summary, click on ‘show progress report’ at bottom Class Progress – From the School Progress, use active header menu to select teacher/class of interest School Comparison – From School Progress, click on ‘show school comparison’ at bottom ** Any time you click on the number below a specific measure you will get a progress report and depending on the level of the report you started from will determine which level of progress report you get

44 School Progress Report

45 School Comparison Group

46 Navigating from the School Status Report – “Home Base” School Grade Summary – Click on the words ‘grade summary’ under the grade want to see Class Status – From the extended view, click on the teacher’s name School Progress – From Grade Summary, click on ‘show progress report’ at bottom Class Progress – From the School Progress, use active header menu to select teacher/class of interest School Comparison – From School Progress, click on ‘show school comparison’ at bottom ** Any time you click on the number below a specific measure you will get a progress report and depending on the level of the report you started from will determine which level of progress report you get

47 School Status Report – “Home Base”

48 School Progress Report

49 Use Data to… monitor progress of specific groups of students

50

51 School Demographic Reports Demographic data is available, by assessment period, at the school and district levels for individual measures and the Recommended Level of Instruction. The data file allows all of the data to be downloaded so that Users can complete their own analyses.

52

53

54 School Demographic Reports You can also display a bar graph for a specific demographic group from this report. The next slides shows a bar graph for kindergarten students at the third assessment by Ethnicity. The total number of students in each category is in ( ) next to the group name. For example Hispanic (17)

55

56

57 Use Data to… run an effective class/student level data meeting

58 Does the district provide the tools and training to interpret and query data? Have data teams developed a process for identifying, recommending and implementing intervention based on data? Do teachers have access to data in an easy-to-use format soon after assessment? Using Data Checklist

59 Data Utilization and Analysis Key Characteristics of Data Meetings: Held on a regular basis (bi-weekly or monthly) Attendees were able to make school level decisions and personnel changes (usually the principal) Systems/worksheets used for structure Data Compiled from multiple sources Follow-up

60 Class/Student Level Data Meeting Some principals start with discussing: Any students who made great gains (celebrations) Surprises in results (or discrepancies between impressions and actual data – if so, might look at student answer sheet to get more specific info) Any students of great concern (and any circumstances that would be of importance – for example, the student has been tardy for the past 20 days missing all whole group instruction) If any ESE, social work or other services are involved Remind staff to look at scores not just colors and to keep ‘next target’ at bottom of column in mind when looking at scores

61 Class Level Reports Two main class level reports can be used to run the meeting (with worksheets) Class Status Reports To determine groups and areas of focus You might also want to have the status report from the previous assessment period to compare scores Class Recommended Level of Instruction To determine effectiveness of instruction and target students

62 Class Status Report The Class Status report from the PMRN will help you answer 3 main questions: 1. Who needs extra support? 2. How should groups be formed? 3. Which skills need to be emphasized? Refer to “Class Status Report” Worksheet Handout

63

64 Class Status Report 1 st Grade Class Assessment Period 2 Full year students only You can get here from ‘School Status’ by clicking on teacher’s name when the grade is extended

65 Who Needs Extra Support? High Risk:5 Moderate Risk: 7 Low Risk: 4 Look at the “Next Target” to help guide grouping

66 How will small groups be formed? Group 1: Group 2: Group 3: Group 4:

67 How will small groups be formed? Group 1: Students 2,4 Group 2: Students 1,3,5,6,7 Group 3: Students 8,9,11,12 Group 4: Students 13,14,15,16,10 (10 to get peer modeling to improve fluency)

68 Which skills need to be emphasized? Group 1: Group 2: Group 3: Group 4:

69 Which skills need to be emphasized? Group 1: PA and Phonics Group 2: Phonics and Fluency Group 3: Phonics and Fluency Group 4: Fluency and Comprehension

70 Calculate the Class Level ECI, EI-I, and EI-S How do you do it? AND…what does it all mean?? Refer to “DIBELS Data Analysis Conference” Worksheet Handout

71

72 Class Recommended Level of Instruction Report

73 Class Recommended Level of Instruction Report This report provides a summary of the students’ overall progress. It can be used to get an overall sense of instructional/risk levels in the class and to calculate the Effectiveness of Core Instruction (ECI) index and the three Effectiveness of Intervention (EI) indices.

74 Class Recommended Level of Instruction Report This is the same type of report with “all” students included. Students 5, 6, 7, and 19 would not be used to calculate the ECI and EI indices because they were not in the school for all three assessment periods. This is why we select the full year only function for calculation.

75 Class Recommended Level of Instruction Report You can use the pie charts to help you fill out the top part of the DIBELS worksheet Initial Level = Low risk Strategic Level = Moderate Risk Intensive Level = High Risk Assessment #3 % Initial = 44 % Strategic = 50 % Intensive = 6

76 Class Recommended Level of Instruction Report To calculate ECI – Effectiveness of Core Instruction for the year, use only those students that were at Initial Instruction during the 1st assessment and remained in the class throughout the year (that is why we select full year only). Divide the number of the students that finished at Initial Instruction for the 3rd assessment (5) by the number of students that were at Initial Instruction for the 1st assessment (9). The higher the percentage the better.

77 Class Recommended Level of Instruction Report Effectiveness of Core Instruction # stayed initial = # started initial = 56% ECI = 56% 9 Target Students: Students 3, 6, 8, 9 (Initial to Strategic)

78 Class Recommended Level of Instruction Report To calculate EI – Effectiveness of Intervention for the year, use only those students that were in the class for the first assessment and the last assessment (full year only). Take the number of students who finished the year at ‘initial’ or ‘strategic’ (if they started at intensive) level by the students who started the year at some level of risk (intensive or strategic) The higher the percentage the better.

79 Class Recommended Level of Instruction Report Effectiveness of Intervention # moved to initial or strategic from Intensive = # started strategic & intensive = 29% EI = 29% 7 Target Students: Students 4, 7, and 10 (strategic to intensive during assessment 2)

80 Class Recommended Level of Instruction Report Effectiveness of Intervention- Strategic # moved to initial = # began strategic = 20% EI-S = 20% 5 Target Students: Students 4, 7, and 10 (strategic to intensive during assessment 2)

81 Class Recommended Level of Instruction Report Effectiveness of Intervention - Intensive # moved to strategic or initial = # began intensive = 50% EI-I = 50% 2 Target Students: Student 5

82 What do all of these indices mean for my school and teachers? This particular teacher in 1 st grade is struggling both with maintaining children at grade level and reaching her struggling readers: ECI = 56% EI = 29% EI-S = 20% EI-I = 50% You may want to first determine if other first grade teachers have similar indices and if so, you want to examine your Core Reading Program and especially your Intervention Programs You can look at the School Recommended Level of Instruction Report to see how different grade levels are doing by comparing the pie charts from one assessment period to the next The goal is to see the green increase and the red decrease – to make the pie chart look more like an olive!!

83 School Level Indices Breaking down the tables and text you receive in your ECI and EI letters

84 ECI Definition Effectiveness of Core Instruction (ECI) — this index shows the percentage of students who began the year reading “at grade level” and are continuing to meet grade level expectations at the mid-year or end of year assessment. This index tells you whether the core instruction is sufficiently powerful to produce a year’s growth in reading skill for a year’s worth of instruction for all students who begin the year at grade level.

85 ECI – “Technical” Percentage of students who started on grade level and remained on grade level Number of children used to calculate the index score Percentile Rank compared to other RF schools

86 ECI – “School” 91% of the 45 kindergarteners who started the year on grade level remained on grade level.91 x 45 = 41 children 41 out of 45 children remained on grade level.97 x 75 = 73 children 73 out of 75 children remained on grade level, 2 children did not remain on grade level In first grade, my school is doing equal to or better than 96% of the RF schools on keeping our 1 st grade students on grade level.

87 EI Definition Effectiveness of Interventions (El)* —this index shows the percentage of your students who began the year at some level of risk for reading difficulties (reading below grade level) but who have grown rapidly enough to advance to a lower level of risk by the mid year or end of year assessment (i.e. they move from intensive to strategic, or from strategic to grade level). This index, and the two explained below, provide information about how effectively intervention programs are working to accelerate the development of students who are lagging behind in reading growth.

88 EI – “Technical” Percentage of children who progressed to a lower level of risk Percentile rank when compared to the other Reading First schools in Florida The number of children used to calculate the index

89 EI – “School” T 81% of the 42 kindergartners who started the year at some level of risk, have made gains and are now at a lower level of risk..81 x 42 = 34 students In first grade, my school is doing equal to or better than 44% of the RF schools in moving children to strategic or grade level..23 x 30 = 7 students 7 out of 30 3rd graders who started at some level of risk have moved to a lower level of risk. 23 students are still at some risk.

90 EI-I Definition Effectiveness of Interventions for Intensive Students (EI-I) — this index shows the percentage of students that moved from the “intensive” risk category to the “strategic” risk level or to grade level.

91 EI – I “Technical” Percentage of students who started the year at “intensive” and made enough progress to be “strategic” or “grade level” Percentile rank when index score is compared to other Reading First schools in Florida Number of students used to calculate the index.

92 EI – I “School” 93% of our 14 kindergarten students who started the year at “intensive” made progress to a lower risk level..93 x 14 = 13 students made growth, 1 is still “intensive” In second grade, we are doing an equal or better job than 47% of the RF schools at moving our children out of the “intensive” category. In all grades, K-3, of the 46 students who started at “intensive” 50% (or 23) of them made growth to “strategic” or “grade level.”

93 EI-S Definition Effectiveness of Interventions for Strategic Students (EI-S) — this index shows the percentage of students that moved from the “strategic” risk level to grade level.

94 EI – S “Technical” Percentage of students who started the year at “strategic” moved to “grade level” Percentile Rank of index when compared to all other RF schools in Florida Number of students used to calculate the index

95 EI – S “School”.56 x 27 kindergarten students who started the year at “strategic” moved to “grade level” = = 12 are still “strategic” In 1 st grade, when compared to the other 583 RF schools in FL, We are doing equal to or better than 48% of them at moving first grade “strategic” students to “grade level” In 3rd grade, 62% of the 16 children (10) who started at “strategic” are now at “grade level”

Worksheet Slides

97 ECI – “Worksheet” ___% of the ___ kindergarteners who started the year on grade level remained on grade level.___ x ___ = ___ children ___ out of ___ children remained on grade level.___ x ___ = ___ children ___ out of ___ children remained on grade level, ___ children did not remain on grade level In first grade, my school is doing equal to or better than ___% of the RF schools on keeping our 1 st grade students on grade level.

98 EI – “Worksheet” T ___% of the ___ kindergartners who started the year at some level of risk, have made gains and are now at a lower level of risk..___ x ___ = ___students In first grade, my school is doing equal to or better than ___% of the RF schools in moving children to strategic or grade level..___ x ___ = ___ students ___ out of ___ 3rd graders who started at some level of risk have moved to a lower level of risk. ___ students are still at some risk.

99 EI – I “Worksheet” ___% of our ___ kindergarten students who started the year at “intensive” made progress to a lower risk level..___ x ___ = ___ students made growth, ___ is/are still “intensive” In second grade, we are doing an equal or better job than ___% of the RF schools at moving our children out of the “__________” category. In all grades, K-3, of the ___ students who started at “intensive” ___% (or __) of them made growth to “strategic” or “grade level.”

100 EI – S “Worksheet”.___ x ___ students who started the year at “strategic” moved to “grade level” = ___ ___-___ = ___ are still “strategic” In 1 st grade, when compared to the other 583 RF schools in FL, We are doing equal to or ____ than ____% of them at moving first grade “strategic” students to “grade level” In ___ grade, ___% of the ___ children (___) who started at “strategic” are now at “grade level”

101 ECI – “Worksheet” ___% of the ___ kindergarteners who started the year on grade level remained on grade level.___ x ___ = ___ children ___ out of ___ children remained on grade level.___ x ___ = ___ children ___ out of ___ children remained on grade level, ___ children did not remain on grade level In first grade, my school is doing equal to or better than ___% of the RF schools on keeping our 1 st grade students on grade level.

102 EI – “Worksheet” T ___% of the ___ kindergartners who started the year at some level of risk, have made gains and are now at a lower level of risk..___ x ___ = ___students In first grade, my school is doing equal to or better than ___% of the RF schools in moving children to strategic or grade level..___ x ___ = ___ students ___ out of ___ 3rd graders who started at some level of risk have moved to a lower level of risk. ___ students are still at some risk.

103 EI – I “Worksheet” ___% of our ___ kindergarten students who started the year at “intensive” made progress to a lower risk level..___ x ___ = ___ students made growth, ___ is/are still “intensive” In second grade, we are doing an equal or better job than ___% of the RF schools at moving our children out of the “__________” category. In all grades, K-3, of the ___ students who started at “intensive” ___% (or __) of them made growth to “strategic” or “grade level.”

104 EI – S “Worksheet”.___ x ___ kindergarten students who started the year at “strategic” moved to “grade level” = ___ ___-___ = ___ are still “strategic” In 1 st grade, when compared to the other 583 RF schools in FL, We are doing equal to or ____ than ____% of them at moving first grade “strategic” students to “grade level” In ___ grade, ___% of the ___ children (___) who started at “strategic” are now at “grade level”

105 Key ‘follow-up’ sections on worksheet Items Discussed Roles and Responsibilities Teacher will… Reading Coach will… Principal will…

106 Follow Up is Key

107 Concluding Thoughts and Resources

108 Concluding Thoughts Use PMRN Reports to make school, classroom, and student level decisions: Monitor progress Determine which students need more intensive instruction Offer instructional strategies to the teacher Determine types of professional development by grade level or individual teacher By asking some of the following questions…

109 Is the core reading program implemented with fidelity? Effectiveness? Are the interventions being implemented with fidelity? Effectiveness? Is there a need for more professional development for a particular area of reading? Is there a need for more intensive coaching? Are materials allocated to maximize learning? Is personnel use maximized to focus on reading instruction? Is the instructional day scheduled such that reading is THE priority? Important Considerations

110 Florida Center for Reading Research 227 North Bronough Street, Suite 7250 Tallahassee, FL (850) Phone (850) (Fax) PMRN Help Desk (Voice) (Toll Free) (Fax) User’s guide

111 PROCESS TO PLAN FOR FOLLOW-UP FOCUSROLES School-basedNon-school Based PrincipalReading Coach TeachersDistrictRFPD Coordinator Grade-level LeaderClassroom Teacher Instructional Content Explicit Delivery Assessment Professional Development A) Resources 1.Human resources 2.Materials B) Time 1. Master schedule 2. Planning period C) Services 1.Workshops 2.Follow-up

QUESTIONS?? THANK YOU!! Liz Crawford, MS, CCC-SLP Director of Interventions