1. The Formative Evaluation Component of Your MTIM Data Based Instructional Decision Making from the Building Level to the Individual Student Level Tom Jenkins, Ed.D. Educational Consultation Services, LLC Wilmington, NC

2. Formative Evaluation Collecting data to make determinations about the effectiveness of your MTIM implementation Collecting data to make determinations about the effectiveness of instruction with students – and everything in between

5. Analyzing Your MTIM Implementation Building consensus is necessary in three stages –pre-implementation –implementation –maintenance

6. Analyzing Your MTIM Implementation Pre-implementation tasks –Identify the purpose of implementation and the critical data whose performance will improve as a result of implementation? what aspect of performance will improve? how will you know? take some time to think about this

7. Analyzing Your MTIM Implementation Pre-implementation tasks continued –Identify the mechanisms that will cause student performance to change enhanced delivery of Core screening intervening progress monitoring

8. Analyzing Your MTIM Implementation Pre-implementation tasks continued –Present plan to stakeholders allow for stakeholder input create forms of communication for stakeholder groups don’t forget your school board disagreement is normal and should not be overlooked

9. Analyzing Your MTIM Implementation Implementation tasks –Consensus is most easily gained through success! –implement where leadership is strong –assess treatment integrity –start small –provide support –communicate success to stakeholders

10. Analyzing Your MTIM Implementation Maintenance tasks –Build RTI/TIM into the school culture set up team meetings that are on the calendar present success stories and current status data at staff meetings

11. Analyzing Your MTIM Implementation Maintenance tasks continued –Training is required each and every year assessment tool training intervention training new staff survey your staff activity

12. Analyzing Your MTIM Implementation Maintenance tasks continued –maintain communication with and present results to stakeholders –show connection between implementation and state accountability measures

13. Analyzing Your MTIM Implementation Things that reduce consensus and support for RTI implementation –lack of identifiable team or individual who is in charge –lack of focus on what RTI is about –asking classroom teachers to do everything –not communicating to critical stakeholders –not identifying how or when students move between tiers

14. Data For Each Tier - Where Do They Come From? Tier 1: Universal Screening, accountability assessments, grades, classroom assessments, referral patterns, discipline referrals Tier 2: Universal Screening - Group Level Diagnostics (maybe), systematic progress monitoring, formative assessment large-scale assessment data and classroom assessment Tier 3: Universal Screenings, Individual Diagnostics, intensive and systematic progress monitoring, formative assessment, other informal assessments

15. Building MTIM Infrastructure Formative evaluation process Informed by data Highly involved school-based leadership team (SBLT) School-based MTIM coach –Provide Technical Assistance –Interpretation and Use of Data –Facilitates regular data meetings for building and grade levels

16. Formative Evaluation Component of Infrastructure What we need: –Screening system for identifying students at risk –Diagnostic assessment tools for identifying specific needs of students identified by screening –Systematic, explicit, research based instructional strategies – differentiated instruction –Progress monitoring plan –Evaluation of whether instruction is effective

17. Data For Each Tier – Where Do They Come From? Addl. Diagnostic Assessment Instruction Results Monitoring Individual Diagnostic Individualized Intensive weekly Screen All Students Monthly Screening Bench- Mark Assessment Annual Testing Behavior Academics None Continue With Core Instruction with differentiation Grades Classroom Assessments Yearly Assessments Standard Protocol Small Group Intervention By skill 2 times/month Step 1 Step 2Step 3Step 4 Supplemental 1-5% 5-10% 80-90% Core Intensive

18. Tier I - Core/Benchmark Universal Screening –Academics: Screen all students, begin in kindergarten; 3 times per year with appropriate early literacy and math measures –More intense instruction and monitoring within classroom for students below cut scores –See worksheet

19. Cut Score Worksheet Step One: Put all student scores on the university screening measure on a histogram type chart. Step Two: Calculate typical Growth Rate of specific skills. Three formulas can be used here. EOYBM – BOYBM / 36 weeks = GR Or EOYBM – MOYBM / 18 weeks = GR Or MOYBM – BOYBM / 18 weeks = GR Step Three: Determine the Targeted Growth Rate for students. Two formulas can be used here depending on the desired amount of ambitiousness. GR * 1.5 = TGR Or GR * 2.0 = TGR

20. Cut Score Worksheet Step Four: Calculate the Growth Goal for the instructional period. TGR * NWI (18 or 36) = GG Step Five: Calculate the Cut Score for determination of level of instruction. Two formulas can be used here depending on the length of the instructional period used in step four. MOYBM – GG = CS Or EOYBM – GG = CS Step Six: Using the Cut Score place a line of demarcation on the histogram created in step one. Any students above the Cut Score should obtain the GG via Core instruction. Any students below the Cut Score may need Strategic instruction to obtain the TGR and GG. Students in need of Intensive instruction should be identified using progress monitoring data during Strategic instruction implementation. Progress monitoring within all three tiers allows for students movement between the tiers during the instructional period.

21. Cut Score Worksheet Activity Knowing that your MOYBM is 40 and your BOYBM is 20 what would be the cut score using a accelerator of 1.5? BOYBM = 20 MOYBM = 40 18 weeks of instruction/intervention Accelerator of 1.5

22. Cut Score Worksheet Activity Step Two: MOYBM – BOYBM / 18 = GR 40 – 20 / 18 = 1.11 Step Three: GR * 1.5 = TGR 1.11 * 1.5 = 1.67 Step Four: TGR * NWI = GG 1.67 * 18 = 29.97 Step Five: MOYBM – GG = CS 40 – 29.97 = 10.03 Step Six: All students above the score of 10.03 should be able to meet the expected Growth Goal via Core instruction. All students below the score of 10.03 would probably be initially placed in Strategic instruction to obtain the Targeted Growth Rate and necessary Growth Goal.

23. Universal Screening Results Fidelity Check: Are you doing the right thing? Assess success of instructional program –Percent of students at or above benchmarks –If necessary, examine curriculum, instruction, or both Identify students below benchmarks –Interventions within general education classroom –Assess progress and consider need for more intensive interventions

24. Decision Rules: What is a “Good” Response to Intervention? Positive Response –Gap is closing –Can extrapolate point at which target student(s) will “come in range” of target--even if this is long range –Level of “risk” lowers over time Questionable Response –Rate at which gap is widening slows considerably, but gap is still widening –Gap stops widening but closure does not occur Poor Response –Gap continues to widen with no change in rate.

25. Performance Time Positive Response to Intervention Expected Trajectory Observed Trajectory

26. Decision Rules: Linking RtI to Intervention Decisions Positive Continue intervention with current goal Continue intervention with goal increased Fade intervention to determine if student(s) have acquired functional independence.

27. Decision Rules: What is a “Questionable” Response to Intervention? Positive Response –Gap is closing –Can extrapolate point at which target student(s) will “come in range” of target--even if this is long range Questionable Response –Rate at which gap is widening slows considerably, but gap is still widening –Gap stops widening but closure does not occur –Level of “risk” remains the same over time Poor Response –Gap continues to widen with no change in rate.

28. Performance Time Questionable Response to Intervention Expected Trajectory Observed Trajectory

29. Decision Rules: Linking RtI to Intervention Decisions Questionable –Was intervention implemented as intended? If no - employ strategies to increase implementation integrity If yes - –Increase intensity of current intervention for a short period of time and assess impact. If rate improves, continue. If rate does not improve, return to problem solving.

30. Decision Rules: What is a “Poor” Response to Intervention? Positive Response –Gap is closing –Can extrapolate point at which target student(s) will “come in range” of target--even if this is long range Questionable Response –Rate at which gap is widening slows considerably, but gap is still widening –Gap stops widening but closure does not occur Poor Response –Gap continues to widen with no change in rate. –Level of “risk” worsens over time

31. Performance Time Poor Response to Intervention Expected Trajectory Observed Trajectory

32. Decision Rules: Linking RtI to Intervention Decisions Poor –Was intervention implemented as intended? If no - employ strategies in increase implementation integrity If yes - –Is intervention aligned with the verified hypothesis? (Intervention Design) –Are there other hypotheses to consider? (Problem Analysis) –Was the problem identified correctly? (Problem Identification)

33. Building Your MTIM

34. Thoughts, fleshing out your pyramid, and action plan activity Is your district/school ready for a universal screening process? Consider…… –Do you have a person with the ability to be a data coach? –Who would be your school leadership team/data analysis team? –Current screening assessment tools –Staff ability to analyze data and make instructional decisions –Do you have consensus?

35. Data For Each Tier – Where Do They Come From? Addl. Diagnostic Assessment Instruction Results Monitoring Individual Diagnostic Individualized Intensive weekly Screen All Students ODRs Monthly Bx Screening Bench- Mark Assessment Annual Testing Behavior Academics None Continue With Core Instruction with differentiation Grades Classroom Assessments Yearly Assessments Standard Protocol Small Group Intervention By skill 2 times/month Step 1 Step 2Step 3Step 4 Supplemental 1-5% 5-10% 80-90% Core Intensive

36. Need for CBM Type Assessments All reading probes scored corrects per minute

37. Need for CBM Type Assessments Math computations are scored by correct digits per minute

38. Need for CBM Type Assessments Correct Sequences for written expression Two words form a sequence, word and punctuation form a sequence. Most words and punctuation are used twice Three minutes to brainstorm, write, and edit

40. Tier II Strategic and Tier III Intensive Progress Monitoring Progress monitoring is essential for four reasons –There is no guarantee that interventions will be successful, thus the intervention must be “tested” to evaluate effectiveness –Increased emphasis of specific outcomes for students, data base must be generated to guide intervention decision making –Pre/post testing has be shown to be unreliable (small amount of data) and provides too little data to allow for instructional decision making – progress monitoring allows for evaluation of level of performance and rate of learning –Research has shown that progress monitoring is associated with improved educational outcomes

41. Research has shown that it works! If we use research validated reading practices, monitor students’ progress and make changes to instruction based on what we find, between 95 and 100 percent of children can become proficient readers. Torgesen, 2000, Learning Disabilities Research and Practice. Individual Differences in Response to Early Intervention in Reading: The Lingering Problem of Treatment Resisters

42. Tier II Strategic and Tier III Intensive Progress Monitoring Essential components that must be in place for successful progress monitoring within each tier –A well-defined behavior –Identification of student’s current level of performance (baseline) –Instruction/Intervention –Goal –A measurement strategy –Graph –Decision-making plan

43. Tier II Strategic and Tier III Intensive Progress Monitoring Target behaviors that are observable and measurable Skill specific assessments Focus on enabling skills –Skills that are prerequisite skills for more complex skills –Deficiencies in enabling skills often adversely affects performance on global assessments

44. Tier II Strategic and Tier III Intensive Progress Monitoring Enabling skills for reading –Phonemic awareness –Alphabetic understanding –Fluency –Sight words –Comprehension

45. Tier II Strategic and Tier III Intensive Progress Monitoring Enabling skills for math –Number sense –Facts –Computation –Applications –Problem solving Enabling skills for written expression –Mechanics –Expression

46. Tier II Strategic and Tier III Intensive Progress Monitoring Enabling skills for behavior –Social skills –Work completion –Compliance –Problem solving skills

47. Tier II Strategic and Tier III Intensive Progress Monitoring Goal setting……………. Standard against which progress can be compared Allows for aimline to be established Possible goals –Level of behavior that is expected – several ways to establish this –Most frequently used is accelerated growth rates for academics and percent of time expectations for behavior

48. Tier II Strategic and Tier III Intensive Progress Monitoring To identify an accelerated growth rate –Take the growth rate that is calculated and multiply it by 1.5 to obtain a slightly ambitious growth rate or –Take the growth rate that is calculated and multiply it by 2 to obtain a more ambitious growth rate –Then multiply by the number of weeks of intervention

49. Tier II Strategic and Tier III Intensive Progress Monitoring For behavior research indicates that a 75% level of performance can be used for non threatening behaviors For behaviors that are threatening or dangerous a 100% level of performance should be used

50. Tier II Strategic and Tier III Intensive Progress Monitoring Goal calculation activity –Using the data below what would Nicole’s goal be, in each area, for an intensive intervention plan that was implemented for nine weeks using a 1.5 accelerator? SkillNicole’s Baseline District 3 rd Grade BOYBM District 3 rd Grade MOYBM Sight Word Phrase Fluency 538191 Oral Reading Fluency 83131157

51. Tier II Strategic and Tier III Intensive Progress Monitoring Graph provides a visual representation of a large amount of data A visual representation of student’s acquisition of skills and allows for easier analysis of progress Paper and pencil or electronic based graphing

52. Charting Activity Activity –Plot the baseline data Monday – 14 Tuesday – 10 Wednesday – 6 Thursday – 5 Friday – 10 –Plot the median and signify with a heavy dark line

53. Charting Activity Plot the goal at the end of eight weeks –40 Indicate the aimline

54. Charting Activity Plot the first two weeks progress monitoring data –Week One Tuesday – 16 Thursday – 14 –Week Two Monday – 10 Wednesday – 18 Friday – 16 Make an informed decision regarding the effectiveness of the intervention

55. Charting Activity Plot progress monitoring data for next two weeks –Week Three Tuesday – 14 Thursday - 10 –Week Four Monday – 15 Wednesday – 16 Friday – 16 Make an informed decision regarding the effectiveness of the intervention

56. Charting Activity Phase two of the intervention –Plot the progress monitoring of the next two weeks –Week Five Tuesday – 24 Thursday - 26 –Week Six Monday – 26 Wednesday – 24 Friday – 26 Make an informed decision regarding the effectiveness of the intervention

57. Charting Activity Phase two of the intervention –Plot the progress monitoring of the next two weeks –Week Seven Tuesday – 28 Thursday - 30 –Week Eight Monday – 32 Wednesday – 36 Friday – 38 Make an informed decision regarding the effectiveness of the intervention Final decision?

58. Charting Activity Continued Trend lines Line that you draw through a series of data points that represents the student’s actual rate of progress If trend line slope is flatter than aimline slope, then adjust intervention If trend line slope is steeper than aimline slope, then adjust goal If slopes are the same, make no change

59. Charting Activity Continued Trend line procedures –Draw vertical line in middle of graph, half data points on one side, half on the other – if odd number of data points, put line through middle point –Draw a vertical line in first half of data to separate data half and half –Draw a vertical line in second half of data to separate data half and half

60. Charting Activity Continued Trend line procedures –Draw a horizontal line through median of first half of data – form an intersection with vertical line –Draw a horizontal line through median of second half of data – form an intersection with vertical line –Connect the two intersections –Make your decision, is it the same decision?

61. Formative Evaluation Case Study Skill Prior To Tier 3 Intervention Nicole’s Baselines District 3 rd Grade Fall Mean ProficiencyWBP Phoneme ID32,32,30412822 Blend ID3,4,624115 Sight Word ID50,53,56816858 Words In Passages 80,83,8413110580 Addition0,0,51397 Subtraction2,1,2511 Multiplication7,5,41142 Math Concepts 3,2,1742

62. Formative Evaluation Case Study SkillDistrict 3 rd Grade Winter Mean ProficiencyWBP Phoneme ID544336 Blend ID32169 Sight Word ID917768 Words In Passages 15711494 Addition14108 Subtraction632 Multiplication211511 Math Concepts 1065

63. Formative Evaluation Case Study Nicole –Progress monitoring data for addition Week one 8,3,13 Week two 13,9,8 Week three 11,3,11 Week four 13,12,12

64. Formative Evaluation Case Study Nicole –Progress monitoring data for blends Week one 7,8,6 Week two 6,5,5 Week three 4,6,4 Week four 4,4,2 Week five 6,6,7 Week six 5,8,10

65. Formative Evaluation Case Study Nicole –Progress monitoring data for sight words Week one 61,63 Week two 64,49 Week three 51,53 Week four 50,52 Week five 56,48 Week six 44,50

66. Formative Evaluation Case Study Nicole –Based on your progress monitoring data, complete the Analysis of Intervention Plan Identify median score of last three progress monitoring data points for each skill Compare current level of performance to previously established goals Make a decision about the results of the intervention plan and the next step Is this a student that needs to be considered for specially designed instruction?

67. Weeks Words Correct Per Minute Student in January of First Grade 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 Benchmark=24 Baseline=11

68. Weeks Words Correct Per Minute Determine Goal: Class=1.5 wd growth per week; Student Goal: 2.25 wd growth per week 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 Bchm=24 Baseline=11 Benchmark Line Student goal line

69. Weeks Words Correct Per Minute Monitor Students’s Progress Relative to Goal 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 Bchm=24 Baseline=11 Benchmark Line Student goal line

70. Weeks Words Correct Per Minute Formative Evaluation: Change Intervention 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 Bchm24 Baseline=11 Benchmark Student goal line Change Intervention

71. Weeks Words Correct Per Minute Continue Intervention and Monitor Progress 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 Bnch=24 Baseline=11 Benchmark Student goal line Change Intervention

72. Weeks Words Correct Per Minute Continue Intervention and Monitor Progress 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 Bchmrk=24 Baseline=11 Benchmark Student goal line Change Intervention Fade Tier ?

73. Weeks Words Correct Per Minute Determine Goal: Class=1.5 wd growth per week; Student Goal: 2.25 wd growth per week 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 Bnchmrk= 24 Baseline=11 Benchmark Line Student goal line

74. Weeks Words Correct Per Minute Monitor Student Progress Relative to Goal 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 Bchmrk=24 Baseline=11 Benchmark line Student goal line

75. Weeks Words Correct Per Minute Change Student Intervention 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 Bchmk=24 Baseline=11 Benchmark line Student goal line Change Intervention

76. Weeks Words Correct Per Minute Implement Revised Intervention and Continue to Monitor Progress 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 Benchmark Line Student goal line Change Intervention

77. Weeks Words Correct Per Minute Implement Second Intervention Revision 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 Benchmark student goal line Change Intervention

78. Weeks Words Correct Per Minute Gap Not Closing: Consider Eligibility and More Intensive Interventions 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 Benchmark Change Intervention Class WCM=54 Student WCM=32

79. Tier II Strategic and Tier III Intensive Progress Monitoring If the decision is to adjust an intervention, small changes or refinements are recommended before major changes However, changes should be substantial enough that it has a possibility to result in improved student performance If making an adjustment, do not make two at the same time. It may result in the team being unable to determine what caused increased student performance Consider intensity vs methodology

80. Building Your MTIM

81. Thoughts, fleshing out your pyramid, and action plan activity Is your district/school ready for implementation of a multi-tiered assessment model? Consider…… –Current diagnostic and formative assessment tools –Staff ability to analyze data and make instructional decisions –Do you have consensus? –Where are the gaps in your pyramid?

82. Questions? Dr. Tom Jenkins, Director Educational Consultation Services, LLC Wilmington, NC (910) 367-7209 Fanofstel@aol.com www.educationalconsultationservices.com

83. Thanks Dan Reschly Dave Tilly George Batsche Ed Shapiro Tracy Hall