1. 1 Oregon Reading First Institute on Beginning Reading VII: Evaluating and Planning Institute on Beginning Reading VII: Evaluating and Planning

2. 2 Acknowledgments n Oregon Department of Education n Institute for the Development of Educational Achievement, College of Education, University of Oregon n U.S. Department of Education, Office of Special Education Programs

3. 3 Content developed by: Edward J. Kame’enui Carrie Thomas Beck Scott K. Baker Nicole Sherman Brewer Ben Clarke Hank Fien Barbara Gunn Rachell Katz Trish Travers Jennifer Walt Content Development Additional support: Katie Tate University of Oregon

4. 4 Copyright n All materials are copy written and should not be reproduced or used without expressed permission of Dr. Edward J. Kame’enui or Dr. Deborah C. Simmons. Selected slides were reproduced from other sources and original references cited.

5. 5 Introduction Thomas Beck © 2004

6. 6 Objectives of Reading First (1) To provide assistance to State educational agencies and local educational agencies in establishing reading programs for students in kindergarten through grade 3 that are based on scientifically based reading research, to ensure that every student can read at grade level or above not later than the end of grade 3. (2) To provide assistance to State educational agencies and local educational agencies in preparing teachers, including special education teachers, through professional development and other support, so the teachers can identify specific reading barriers facing their students and so the teachers have the tools to effectively help their students learn to read.

7. 7 Objectives of Reading First (cont.) (3) To provide assistance to State educational agencies and local educational agencies in selecting or administering screening, diagnostic, and classroom-based instructional reading assessments. (4)To provide assistance to State educational agencies and local educational agencies in selecting or developing effective instructional materials (including classroom-based materials to assist teachers in implementing the essential components of reading instruction), programs, learning systems, and strategies to implement methods that have been proven to prevent or remediate reading failure within a State.

8. 8 Objectives of Reading First (cont.) (5) To strengthen coordination among schools, early literacy programs, and family literacy programs to improve reading achievement for all children.

9. 9 What will Reading First look like in the years to come?

10. 10 Oregon Reading First: Year 03 n No IBRs are planned for the third year of implementation. n Cohort A Leadership Strand continues: Leadership Sessions in October and March Coaches’ Training in Sept, Nov, Feb, and April n Schools receive funds for the replacement of consumables and wear of instructional materials. n Funds provided for early literacy narrative and expository text for classroom and school libraries. n Full time Mentor Coach n Professional Development: school-based substitute time for collaboration some statewide PD opportunities

11. 11 Oregon Reading First: Year 03 n ongoing refinement of Summary of CSI Maps n classroom observations (three in Year 03) n lesson progress reports n regular grade level team meetings n ongoing data-based decision making n end-of-year evaluation and planning

12. 12 Oregon Reading First: Year 03 n Cohort A: Levels of Support Model Based on the reports from RF districts and schools, subsequent technical and professional development support will be based on at least two factors: 1. Student reading data 2. Level of implementation of major RF components

13. 13 average reading performance / high implementation strong reading performance / low implementation strong reading performance / high implementation poor reading performance / high implementation average reading performance / low implementation poor reading performance / low implementation Outside In Support Inside Out Support Figure 1: District/School Support Needs Intensive Support Moderate Support Support as Needed

14. 14 Data-Based Leadership Model

15. 15 Q1: What are our goals?

16. 16 Reading Action Plan 04-05 Review progress toward meeting goals on 04-05 Reading Action Plan. Goal 1: Goal 2: Goal 3:

17. 17 Q2: How are we doing? Fien © 2004

18. 18 Two Approaches for Evaluating Effectiveness I. Outcomes Criterion n 95% of all students achieve the early literacy goal. II. Adequate Progress Criteria – are all students making adequate progress? n Core Curriculum and Instruction: Benchmark students make adequate progress and achieve goals n Supplemental Support: Strategic students make adequate progress and achieve goals n Intensive Intervention: Intensive students make adequate progress and achieve goals or at least reduce risk.

19. 19 I. Platinum Standard I. Outcomes Criterion n 95% of all students achieve the early literacy goal. n For example, 95% of students in Kindergarten reach established on PSF Goal of 35 in the Spring. If so, Platinum Standard is MET!!! n If not, continue with Adequate Progress Criterion for Benchmark, Strategic and Intensive Students

20. 20 II. Silver Standard of Adequate Progress n A school will meet silver standard if they are in the top third compared to other schools.

21. 21 CriterionType of Standard Meets Standard? Index of Strength Schoolwide System Outcome Platinum Standard 95% of all students meet goal Benchmark Adequate Progress: Meet Goal Silver Standard Upper 3rd of Benchmark Middle 3rd of Benchmark Lower 3rd of Benchmark Yes Step-by-Step: Evaluating the Effectiveness of the Benchmark Support or Secondary Prevention Schoolwide System Strength No YesNo Yes Needs Support No Yes Needs Substantial Support Relative Strength

22. 22 CriterionType of Standard Meets Standard? Index of Strength Schoolwide System Outcome Platinum Standard 95% of all students meet goal Strategic Adequate Progress: Meet Goal Silver Standard Upper 3rd of Strategic Middle 3rd of Strategic Lower 3rd of Strategic Yes Step-by-Step: Evaluating the Effectiveness of the Strategic Support or Secondary Prevention Schoolwide System Strength No YesNo Yes Needs Support No Yes Needs Substantial Support Relative Strength

23. 23 CriterionType of Standard Meets Standard? Index of Strength Schoolwide System Outcome Platinum Standard 95% of all students meet goal Intensive Adequate Progress: Meet Goal Silver Standard Upper 3rd of Intensive Middle 3rd of Intensive Lower 3rd of Intensive Yes Step-by-Step: Evaluating the Effectiveness of the Intensive Support or Secondary Prevention Schoolwide System Strength No YesNo Yes Needs Support No Yes Needs Substantial Support Relative Strength

24. 24 Activity 1: Year 2 Grade Level Outcomes: By grade and in each classroom, how are students performing at the end of the year on the essential components of RF? Materials Needed n Grade-level histograms n Oregon Reading First histograms n Breakout Forms Directions 1. Discuss activity as a large group 2. Break into grade level teams. 3. For each grade level, document student outcomes on grade- appropriate DIBELS measures. 4. Compare your grade level outcomes with the DIBELS Platinum Standard (95% Established) 5. Compare your grade level outcomes with Oregon Reading First Schools’ overall outcomes.

25. 25 1. Spring Goal: What is the primary instructional goal for the spring of K-3 rd grade? Core Component or Big Idea:_____________ DIBELS Measure: ______________________ Goal Skill Level: _______________________

26. 26 DIBELS Spring Benchmark Goals Grade K Goals: 35 CSPM (correct sounds per minute) on Phoneme Segmentation Fluency 25 CSPM (correct sounds per minute) on Nonsense Word Fluency First Grade Goals: 40-60 CWPM (correct words per minute) on Oral Reading Fluency 50 CSPM (correct sounds per minute) on Nonsense Word Fluency (in Winter of first grade)

27. 27 DIBELS Spring Benchmark Goals Second Grade Goals: 90 CWPM (correct words per minute) on Oral Reading Fluency Third Grade Goals: 110 CWPM (correct words per minute) on Oral Reading Fluency

28. 28 2.Spring Outcomes: At the end of the year, on the essential RF component, what percent of your students are: Established/ Low Risk:_____ Emerging/ Some Risk:_____ Deficit/ At Risk:_____

29. 29 First Grade Example

30. 30 2.Spring Outcomes: At the end first grade, on ORF, what percent of your students are: Low Risk: 62% Some Risk: 20% At Risk: 17% Did this school reach the absolute criteria of 95% established? No

31. 31 3. Compare your grade level outcomes with Oregon Reading First Schools’ overall outcomes. n Insert Project wide histograms

32. 32 Take 10 minutes to Complete Activity #1 in Grade Level Teams

33. 33 Activity #2: Were our grade level instructional plans (Summary of CSI Maps) effective in supporting adequate progress for students with benchmark, strategic and intensive needs? Materials Needed: Summary of Effectiveness Reports by School, Summary of Effectiveness Reports by Class, Oregon Reading First Schools’ Summary of Effectiveness Reports, Adequate Progress School Norms Directions: 1. Discuss activity as a large group 2. Document adequate progress for each grade level. 3. Compare grade-level adequate progress with National Norms 4. Compare grade-level adequate progress with Oregon Reading First schools

34. 34 DIBELS Summary of Effectiveness Reports 4 Ways to Achieve Adequate Progress

35. 35 How are we doing at the systems level (school)? n School: Were our grade level instructional plans (Summary of CSI Maps) effective in supporting adequate progress for students with benchmark, strategic and intensive needs? n For each grade level: n What percent of benchmark students reach established (low risk) on the spring benchmark goal? n What percent of strategic students reach established (low risk) on the spring benchmark goal? n What percent of intensive students reach established (low risk) on the spring benchmark goal? n What percent of intensive students reach emerging (some risk) on the spring benchmark goal? n Data Source: Summary of Effectiveness by School and Summary of Effectiveness by Class

36. 36 1. Initial Skills: In the winter of kindergarten, what percentage of students were: Benchmark: ____ Strategic: ____ Intensive: ____

37. 37

38. 38 2. Adequate Progress Of Benchmark Students: Of the students who were Benchmark in the middle of kindergarten, what percent achieved the PSF goal of 35 for the end of kindergarten? ___________

39. 39

40. 40 2. Adequate Progress Of Benchmark Students: Of the students who were Benchmark in the middle of kindergarten, what percent achieved the PSF goal of 35 for the end of kindergarten? 60%

41. 41 3. How would you rate the effectiveness of the core curriculum and instruction?  Relative Strength- 96% to 100% of benchmark students achieve PSF goal.  Needs Support- 86% to 95% of benchmark students achieve PSF goal.  Needs Substantial Support- 0% to 85% of benchmark students achieve PSF goal.

42. 42 4. Adequate Progress Of Strategic Students: Of the students who were Strategic in the middle of kindergarten, what percent achieved the PSF goal of 35 for the end of kindergarten? ___________

43. 43

44. 44 4. Adequate Progress Of Strategic Students: Of the students who were Strategic in the middle of kindergarten, what percent achieved the PSF goal of 35 for the end of kindergarten? 32%

45. 45 5. How would you rate the effectiveness of the school-wide system of strategic support:  Relative Strength- 75% to 100% of strategic students achieve PSF goal.  Needs Support- 50% to 74% of strategic students achieve PSF goal.  Needs Substantial Support- 0% to 49% of strategic students achieve PSF goal.

46. 46 6. Adequate Progress Of Intensive Students: Of the students who were Intensive in the middle of kindergarten, what percent made adequate progress towards the PSF goal of 35 for the end of kindergarten? ___________

47. 47

48. 48 6. Adequate Progress Of Intensive Students: Of the students who were Intensive in the middle of kindergarten, what percent made adequate progress towards the PSF goal of 35 for the end of kindergarten? 77%

49. 49 7. How would you rate the effectiveness of the school-wide system of intensive support:  Relative Strength- 80% to 100% of intensive students achieve PSF goal or are emerging.  Needs Support- 57% to 79% of intensive students achieve PSF goal or are emerging.  Needs Substantial Support- 0% to 56% of intensive students achieve PSF goal or are emerging.

50. 50 Some other information on the Summary of Effectiveness Report: % of students for each instructional recommendation that do not make adequate progress or

51. 51 Track individual students’ progress D. Mariah an intensive student that made adequate progress to established H. Adrian a benchmark student that did not make adequate progress

52. 52 8. Compare your school’s adequate progress with other Oregon Reading First schools. Based on this comparison: n Discuss areas of strength. (e.g., Compared to the project wide data, our grade has a higher percentage of strategic students meeting spring time benchmarks.) n Discuss areas of needed student support. (e.g., Compared to the project wide data, our grade has a lower percentage of intensive students meeting spring time benchmarks).

53. 53 Oregon Reading First Project Level Summary of Effectiveness 2004-2005: Beginning of First Grade to Middle of First Grade

54. 54 Take 30 minutes to Complete Activity #2 in Grade Level Teams

55. 55 Activity #3: Discussion Points n How did the Fall, Winter and Spring Benchmarking Assessment plan work this year? Any ideas for next year? n How did the Progress Monitoring Plan work this year? Any ideas for next year? If the teachers were not involved this year, how could they get involved next year? n Do we need brush-up DIBELS training next fall? Could we train our EA’s, parent volunteers? How will we train new staff? n What will we do to ensure continued reliable DIBELS data collection? Could we utilize the Assessment Integrity Checklist? n Could we refine how we use data in grade level team meetings?

56. 56 Activity #4: (Optional) Choose 1 Activity from a-c a) Transitioning Activity Materials Needed: Class reports, Summary of Effectiveness Reports n Kindergarten Team join with the First grade Team and discuss Kindergarten students transitioning to first grade. n Second Grade Team join with Third Grade Team and discuss second graders transitioning to third grade. n For each, discuss possible grouping strategies for fall that could maximize instructional intensity. b) Disaggregating Activity Materials Needed: Distribution Reports n Discuss adequate progress for sub-groups of students (ELL, Sped, transfer) relevant to your school. c) Yearly Progress Activity Materials Needed: Cross Year Box Plots n Discuss Year 2 RF outcomes compared to Year 1.

57. 57 Activity #5: How are we doing at the systems level (district)? (optional) n District: Across the district, were our grade level instructional plans (Summary of CSI Maps) effective in supporting adequate progress for students with benchmark, strategic and intensive needs? n For each school and at each grade level: n What percent of benchmark students reach established (low risk) on the spring benchmark goal? n What percent of strategic students reach established (low risk) on the spring benchmark goal? n What percent of intensive students reach established (low risk) on the spring benchmark goal? n What percent of intensive students reach emerging (some risk) on the spring benchmark goal? n Data Source: Summary of Effectiveness by District

58. 58

59. 59 Q3: How do we get there? Thomas Beck, Sherman Brewer, Katz © 2004

60. 60 Linking Q2: How are we doing? with Q3: How do we get there?  How are we doing with our core implementation? What do the data tell us?  How is the combination of current programs/strategies working for our lower performing students?  So what do we do next year?

61. 61 1. How are we doing with our core implementation? What do the data tell us? Activity # 1A: Evaluation of core implementation Materials Needed: Activity #2 from Assessment Section Revisit the Summary of Effectiveness data from Activity #2 of the Assessment section. In particular, examine the number of benchmark students who met benchmark goals over the course of the year. Are most benchmark students remaining on track? What are the trajectories like for benchmark students?

62. 62 Questions to Consider with Core Implementation: INFRASTRUCTURE: n Are all necessary teacher and student materials available in each classroom? n Is a sufficient amount of time allocated? n Is a sufficient number of staff allocated? n Are students grouped based on program recommendations? Are group sizes for large and small group activities appropriate?

63. 63 Questions to Consider with Core Implementation: QUALITY OF IMPLEMENTATION: n Are important activities taught/stressed (e.g., red checks, targets, etc.)? Keep in mind the differing priorities for K vs. 1 vs. 2 vs. 3. Revisit the instructional maps. n Were study groups formed to focus on particular big ideas in beginning reading? n Have the grade level teams worked together to systematically enhance the program as necessary? n How much time are students spending on independent activities? What types of independent activities are provided? n Is the in-program assessment administered regularly and results used to inform instruction? n Are students meeting projections for lesson progress? n Is the program implemented with fidelity?

64. 64 Activity # 1B: Fidelity of Implementation Materials Needed: Fidelity checklists completed by coaches in the Fall, Winter, and Spring (including documentation collected for alternate observation formats). Coach summarizes common issues related to core implementation based on fidelity observations conducted in the Fall, Winter, and Spring. What are common trends in K, 1, 2, 3? What are teachers implementing well? Are there particular activities with which teachers have difficulty (e.g., phonics, PA, etc.)? 1. How are we doing with our core implementation? What do the data tell us? (cont.)

65. 65 To improve quality of core program implementation, consider the following: n Is ongoing, high quality training provided? n Are teachers receiving support from the RF coach? in the classroom? outside the classroom? n Are regular inservice sessions developed around implementation issues identified by coach? n Do teachers have opportunities to observe model lessons - from coach, from peers, from other schools? n Is our school a candidate to participate in program- specific training through WRRFTAC? n Are program-specific consultants brought in to observe in classrooms and provide ongoing support and training? n Are new teachers provided the necessary program training?

66. 66 So what do we do next year? Activity # 1C: Plan for Fall Materials Needed: Winter to Spring CSI Maps blank Fall to Winter Summary of CSI Maps DIBELS data from assessment activities 1 and 2 sample schedules and CSI maps from exemplary schools Complete grade-level Summary of CSI Maps to plan instruction for benchmark students in the Fall. To prepare for this activity, complete the following procedures:

67. 67 To Prepare: Review progress of benchmark students in each grade level. Identify grade level systems that will require support. K: 1: 2: 3:

68. 68 To Prepare: For each grade level system that will require support, list relevant infrastructure and quality of implementation issues: K: 1: 2: 3:

69. 69 To Prepare: n Using the relevant infrastructure and quality of implementation issues as a guide, identify reading action plan goals for each grade level. Note that if benchmark students are making adequate progress within a grade level, it is important to consider how you will sustain the system over time.

70. 70 Grade Level RAPs Institute on Beginning Reading VII Grade Level Reading Action Plan 2005-2006 School: Grade: GOAL 1: Who: When: GOAL 2: Who: When: GOAL 3: Who: When:

71. 71 Professional Development Needs n Determine the professional development that will be necessary for staff to meet the grade level Reading Action Plan goals. List below:

72. 72 To Prepare: Examine sample schedules and Summary of CSI Maps from exemplary RF schools. Are there ideas you can apply to your grade-level instructional plans?

73. 73 So what do we do next year? Now, you are ready to fill out your maps. Break into grade-level teams and use the information from the activities that were just completed to develop grade-level Summary of CSI Maps to plan instruction for benchmark students in the Fall of 2005.

74. 74

75. 75 Activity # 2A: Evaluation of Grade-Level Data Materials Needed: Activity #1 and #2 from Assessment section, Winter to Spring CSI Maps Revisit Winter to Spring DIBELS summary tables from Activity #1 of assessment section and summary of effectiveness reports from Activity #2. Use this data to examine the effectiveness of instruction provided to intensive and strategic students from Winter to Spring. Specifically, be prepared to address the following items: 2. How is the combination of current programs/strategies working for our lower performing students?

76. 76 Evaluation of Grade-Level Data (cont.) n Using spring histograms, discuss how current instructional practices are addressing students needs in the following areas: Phonemic Awareness Alphabetic Principal Fluency with Connected Text  Using the summary of effectiveness reports, examine adequate progress of intensive and strategic students from winter to spring.

77. 77 Evaluation of Grade-Level Data for Strategic and Intensive Systems IF...THEN If grade level system meets the platinum standard... proceed to activity #2C to evaluate performance of individual students. Address the issue of sustainability. If strategic/intensive system is a relative strength (silver)... consider QUALITY OF IMPLEMENTATION questions on following slide. Follow up by examining performance of individual students (activity #2C). If strategic/intensive system needs support or needs substantial support... begin by considering INFRASTRUCTURE questions on following slide. Complete activity #2B.Then address QUALITY OF IMPLEMENTATION questions.

78. 78 Questions to Consider with Supplemental and Intervention Implementation: n INFRASTRUCTURE: Are appropriate reading programs and materials being used to teach the full range of students across K-3? (see next slide) Is adequate instructional time scheduled for students who are struggling? Within the 90-minute block? Outside the 90-minute block? Have staff been assigned in a way such that reading instruction can be delivered to the full range of students each day? Were students grouped appropriately for reading instruction? n QUALITY OF IMPLEMENTATION: Have staff received professional development on programs used in classrooms prior to implementation and at least twice after initial training? Were the reading programs and materials implemented with fidelity? Are efforts to improve fidelity working? Did teachers focus on the general features of instruction that are critical for students who are struggling? (e.g., modeling, opportunities for practice, etc.) Did grade level teams regularly analyze student reading data, plan instruction based on data, and regroup students based on the data?

79. 79 2. How is the combination of current programs/strategies working for our lower performing students? (cont.) Activity #2B: Examine whether appropriate reading programs and materials are being used to teach the full range of students. Materials Needed: Oregon Reading First’s list of approved supplemental and intervention programs and corresponding reviews, sample Summary of CSI Maps from exemplary RF schools  Are we implementing too many supplemental and intervention programs with our lower performing students?  Do we need to focus on providing enhancements/”boosters” to the core program?  Have we selected the strongest intervention programs to implement with our intensive students?  Are there other intervention programs that may better address our students’ needs? List here.  How will the S-I programs fit within your overall grade-level and school-level reading plan?

80. 80 2. How is the combination of current programs/strategies working for our lower performing students? (cont.) Activity # 2C: Evaluation of Progress Monitoring Data (OPTIONAL) Materials Needed: Progress monitoring booklets for strategic and intensive students in each grade level, master list of student groups, Winter to Spring CSI Maps, Alterable Variables Chart Working in grade-level teams, identify one instructional group of intensive students and one instructional group of strategic students. Gather progress monitoring booklets for these students. If you haven’t already done so, draw in the aim line for each student on each measure. Using the 3-point rule, evaluate the effectiveness of the intervention provided from winter to spring.

81. 81 So what do we do next year? Activity # 2D: Plan for Fall Materials Needed: Winter to Spring CSI Maps blank Fall to Winter Summary of CSI Maps DIBELS data from assessment activities 1 and 2 list of approved supplemental and intervention programs program reviews for adopted supplemental and intervention programs sample schedules and CSI maps from exemplary schools Complete grade-level Summary of CSI Maps to plan instruction for strategic and intensive students in the Fall. To prepare for this activity, complete the following procedures:

82. 82 To Prepare: Review progress of strategic and intensive students in each grade level. Identify grade level systems that will require support. K: 1: 2: 3:

83. 83 To Prepare: For each grade level system that will require support, list relevant infrastructure and quality of implementation issues: STRATEGICINTENSIVE K 1 2 3

84. 84 To Prepare: n Using the relevant infrastructure and quality of implementation issues as a guide, identify reading action plan goals to address the needs of strategic/intensive students for each grade level. Note that if strategic / intensive students are making adequate progress within a grade level, it is important to consider how you will sustain the system over time.

85. 85 Grade Level RAPs Institute on Beginning Reading VII Grade Level Reading Action Plan 2005-2006 School: Grade: GOAL 1: Who: When: GOAL 2: Who: When: GOAL 3: Who: When:

86. 86 Professional Development Needs n Determine the professional development that will be necessary for staff to meet the grade level Reading Action Plan goals. List below:

87. 87 To Prepare: Examine sample schedules and Summary of CSI Maps from exemplary RF schools. Are there ideas you can apply to your reading plan?

88. 88 To Prepare: Review important instructional components of Oregon Reading First’s Schoolwide Beginning Model:

89. 89 So what do we do next year? Now, you are ready to fill out your maps. Break into grade-level teams and use the information from the activities that were just completed to develop grade-level Summary of CSI Maps to plan instruction for strategic and intensive students in the Fall of 2005.

90. 90

91. 91 Next Steps: Prioritize PD Needs FALLWINTERSPRING K 1 2 3