Nuts and Bolts of Progress Monitoring Laura Boynton Hauerwas Kristen Matthes May 31, 2006

Topics for the morning Progress Monitoring Assessment: Sample PM tools What? Why? Sample PM tools Graphing Data Data-Base Decisions Setting Goals Performance Level Rate of Learning Linking cases to RI RTI processes

Assessment in a RTI model Benchmarking To screen and identify students who are at-risk and in need of interventions All students Three times a year All areas At grade-level Progress Monitoring To monitor progress of individual students and determine rate of improvement and need for adaptation of intervention Students who are not achieving benchmarks (PLP, IEP) Weekly, biweekly, monthly assessments In area of need At instructional level

What assessments do you use? Reading Math Social-Emotional Behavior What measures do you use? Who do you assess? When do you assess? How is the information used? Benchmark/ Progress Monitoring

Progress Monitoring Benefits of Progress Monitoring Parents and students know what is expected Teachers know what is working or not working with their instruction based on data Easy to understand way to show parents progress Teams have comprehensive data on student performance for decision making

Progress Monitoring CBMs Are assessments to monitor progress Are designed to serve as “indicators” of general reading achievement. R-CBM doesn’t measure everything, but measures the important things. Are Standardized tests to be given, scored, and interpreted in a standard way. Are researched with respect to psychometric properties to ensure accurate measures of learning. Are Sensitive to improvement in Short Periods of time. Designed to be as short as possible to ensure its “do ability.” Are linked to decision making for promoting positive achievement and Problem-Solving

Progress Monitoring National Center on Student Progress Monitoring www.studentprogress.org

Samples of CBMs Reading Math Writing Spelling

MAZE - CBM AIMSweb Reading Comprehension Measure www.aimsweb.com

DIBELS Phoneme Segmentation Fluency https://dibels.uoregon.edu

MATH COMPUTATION Taken from Fuchs, L. S., Hamlett, C. A., & Fuchs, D. (1998). Monitoring Basic Skills Progress: Basic Math Computation (2nd ed.). [computer program]. Austin, TX: ProEd. Available: from http://www.proedinc.com

Concepts and Applications Sample page from a three-page test for Grade 2 Math Concepts and Applications From Monitoring Basic Skills Progress

CBM - Writing Total Words Written Correct Word Sequences Words Correctly Spelled www.interventioncentral.org

Spelling AIMSweb Spelling Probes

Graphing Graphing is an essential part of PM Without graphic displays, the decision making process is difficult Teacher graphing vs. Student graphing

How to Develop Graphs Hand Graphing Excel and Chart Dog Graphing Data: Beetle, SUV, Race Car Hand Graphing Excel and Chart Dog Web-based data systems

Hand Graphing Establish Baseline (Median score) Set up graph Set Goal Draw Aimline Measure Student Progress Plot Student Performance Connect Indicators of Student Performance Analyze Student Performance Make Instructional Changes Continue to Measure and Monitor Student Performance

Hand Graphing Goal 44

Hand Graphing 50 45 40 35 30 Testing Sessions Number of Words Read Correctly 35 30 Baseline Session 1 Session 2 Session 3 Session 4 Session 5 Session 6 Session 7 Session 8 Testing Sessions

Hand Graphing Advantages Easy to do No technology required Students can easily maintain their own graphs Can be done immediately Free Disadvantages Added paper Organization required No long-term storage Not automatic

Excel Excel Template Website to access Excel Templates http://www.oswego.edu/~mcdougal/web_site_4_11_2005/index.html To add trend line: Select Data Points Right Click …Add Trend Line Choose Linear, To find slope –Options Add Equation to the chart (y= slopex+ intercept)

Chart Dog Tool at www.interventioncentral.org

Excel and Chart Dog Advantage Automatic Storage capability built-in Easy to do Clear displays of data Free if you have EXCEL, Chart Dog is free Disadvantage Requires technology Time to enter data Students may not be able to do data entry themselves Requires some understanding of EXCEL or Chart Dog

Web Based System

Web-Based System Advantage Web based data entry from anywhere Storage capability built-in Trend line drawn automatically Can annotate graphs interventions/goals Norms –benchmarks and Rate of Improvement Lots of flexibility Email graphs Disadvantage Requires technology Cost Students may not be able to do data entry themselves Requires some training

BREAK To do this will take new learning for everyone

Data-Based Decisions Performance Level Rate of Learning Gaps in Performance PLP Not at Grade Level Special Education Significant Discrepancy Rate of Learning Trend in performance (slope) Response to Instruction General Direction, Rate of Change

Performance Level: Gap/ Discrepancy Be objective. Does it refer to an observable/measurable characteristic of behavior? Use numbers to define the discrepancy. Percentile rank Discrepancy Ratios Cut scores Norms

Norms… What to use? Local, National Local norms can be helpful to determine local performance levels and rates of progress Time consuming and costly to develop National norms and research norms are available. BUT….

1. Performance Level Percentile Ranks Requires a Larger Normative Data Base, Preferably Benchmark Data < 25th At Risk, Consider Problem-Solving at the Group Level <10th Potential Severe Problem, Consider Individual Problem Solving

Performance Level Discrepancy Ratios Sample 5-7 Students or Whole Class, Grade Figure Median and Graph Divide by 2 and Graph Students Who Performance Below the Line May Need Problem Solving

= Discrepancy of 3.6x Can Compute… Peer Median Target Student Median 145 40 = Discrepancy of 3.6x

Performance Level Cut Scores A number which represents the point at which scores can be divided into different groups (for example does not meet, meets, and exceeds expectations) for decision-making purposes. May be based on research (e.g., a correlation between scoring at or above a certain level on a CBM or DIBELS task and future academic success) or expectation (e.g., grades at C or above, no more than 3 office referrals).

Data-base decisions on performance level enables team to make decisions about levels of support and resource from the start. Generally speaking… - A student who is 1.5x discrepant from his/her peers may benefit from intensive group interventions. A student who is 2-2.5x discrepant from his/her peers is appropriate for individualized problem-solving and intensive intervention resources may be appropriate. Example: Jessica is 2.1x discrepant from peers on the Math CBM and may benefit from intensive interventions in math.

Rebecca 2nd grader List all areas of concern: Off-task behavior Reading difficulties Poor handwriting Identify primary area of concern and define it in observable and measurable terms: Reading Definition: number words read correctly when reading a grade level passage orally Collect baseline data on primary area of concern and state discrepancy statement: Baseline data collected in the area of test from CBM reading probes Discrepancy Statement: Rebecca reads 41 WRC per minute in Fall of 2nd grade while her peers read ____ WRC per minute ___________ _______________________________________________________

Rob 7th grader List all areas of concern: Calling out Lack of homework completion Poor handwriting Identify primary area of concern and define it in observable and measurable terms: Work Completion Definition: Turning in teacher assigned work at beginning of class period on the day that it is due. Collect baseline data on primary area of concern and state discrepancy statement: Baseline data collected in the area of review from teacher grade books Discrepancy Statement: Rob currently turns in homework 54% of the time while his peers turn in homework 86% of the time (_____ discrepant)

Data-Based Decisions Performance Level Rate of Learning Gaps in Performance PLP Not at Grade Level Special Education Significant Discrepancy Rate of Learning Trend in performance (slope) Response to Instruction General Direction, Rate of Change

2. Rate of Learning Why? Determine when what we are doing isn’t working and intervene early Better able to predict student success at meeting goals Better able to identify who needs more intensive instruction

Tracking Student Outcomes Using Initial Performance Discrepancies Rate of Learning Tracking Student Outcomes Using Initial Performance Discrepancies Name Grade Area Initial Performance Discrepancy Follow Up Performance Discrepancy Outcome Decision Bill 3 Reading 10th percentile 20th percentile Satisfactory; Maintain Intervention Susie 2 Math 1.2x NA No Severe Problem Rob 4 Homework 65% completion 64% completion No Progress, Problem solve and Modify intervention plan

Rate of Learning Analyzing Rate using PM Data Rules: Setting Goals Data Point Rules Trend Line Rules Slope

Setting Goals End of the Year Benchmarks GLEs for Reading Fluency (2nd grade 80-100 WPM, 5th grade 125-150) AIMSweb Math Computation Norms (1st grade 17 DPM, 5th grade 52 DPM) 2. National Norms for Improvement Math Calculations (>.3 DPM 2nd and 3rd grade, >.5 DPM 4-6th grade) (Fuchs, 2006) Reading Fluency (Deno, 2005) Grade Modest Reasonable Ambitious 1-2 1 Word Per Week 1.5 Word Per Week 2.0 Word Per Week 3-6 .5 Word Per Week 1.0 Word Per Week

Setting Goals 3. Individual ROI Weekly rate of improvement in “baseline slope” calculated from 8 data points (Slope: Difference of highest and lowest/#weeks) Baseline multiplied by 1.5 Product multiplied by number of weeks until end of year Add to student’s final baseline score to produce end of year goal. Baseline Reading scores: 52, 54, 52, 53, 55, 58, 55, 56 Difference: 58-52 =5 Divide by number of weeks: 5/8 =.625 (SLOPE) Baseline multiplied by 1.5: .625 × 1.5 = .9375 Number of weeks left (6 weeks): .9375 ×6 =5.6 Add to final baseline score: 56+5.6 = 61.6 End of the year goal 62

Jim – 5th grader What goal would you set for Jim in math for the end of year? Jim – 5th grader Math Calculation December-January Monitoring 36, 37, 36, 36, 37, 38, 39, 37 January Benchmark 38 Base determination on 18 weeks left in the year

Jim – 5th grader What goal would you set for Jim in math for the end of year? Jim – 5th grader End of Year Benchmarks 52 DPM (.77 ROI) National Norms 47 DPM (.5 ROI) (38+18*.5) Individual ROI 48 DPM (.56 ROI) 3/8*1.5 =.56 ROI 38 + (18*.56)

Decisions based on data-points Decisions are based on at least 4 data points If all 4 scores fall above goal-line, responding to instruction (increase goal if continues for 4 more data points) If scores are hovering about the goal line, continue what you are doing. If all 4 scores are below goal-line, but parallel, decide to “wait” for 4 more points to see if student performance accelerates in level to reach original goal. If all 4 scores fall below goal-line, not responding to instruction, revise plan and implement different teaching strategy. Mark change on graph with vertical line. Derived from: Fuchs and Fuchs (2006) and Shapiro (2006)

Decisions based on trend lines Trend lines based on 6-8 data-points If trend line is steeper than goal line, increase the goal. If trend line is flatter than goal line, revise instruction If trend line equals goal line, make no change at this time.

Data-base decision on slope Rate of Improvement = slope or r(in statistics) Y = slopeX + intercept Consider discrepancy from ROI norms

Is Jim responding to the intervention?

Is Jim Responding to the Math Intervention? Checkpoint: Is Jim Responding to the Math Intervention? Are modifications necessary? End of Year: What would you say about Jim’s Math?

How can you use the literacy, math and social-emotional/behavior assessments that you have? Do they provide information about a student’s performance level rate of learning area of instructional need?

Your chart might include one of these literacy measures? R-CBM Rigby DRA Performance Level YES YES? Rate of Learning NO Instructional needs

What is the difference between data-driven and data informing?

Rebecca- 2nd grader Fall benchmark some concern Instructional Plan 41 WPM, <25th percentile, 1.96X Discrepant Instructional Plan small group guided reading book bag for home with leveled readers

Rebecca- 2nd grader

What additional data is needed to modify interventions and meet Rebecca’s needs? Possible Hypotheses Review Interview Observe Test Rebecca reads 41 WRC per minute in the Fall of 2nd grade while her peers read ____WRC per minute and with small group guided reading limited rate of improvement because …

Case Example: Rebecca Possible Hypotheses Review Interview Observe Rebecca reads 41 WRC per minute in the Fall of 2nd grade while her peers read 80 WRC per minute and with small group guided reading limited rate of improvement because … Possible Hypotheses Review Interview Observe Test She had insufficient opportunity to practice Interview Teacher/Parent Observe during SSR CBM She lacks the phonological skills to read the text fluently. Review her Records R-CBM scores Phonological Assessment Instructional materials are above her reading level. Review her Records/CBM scores CBM survey level assessment

Case Example: Rebecca Predicted Hypotheses Review Interview Observe Test She had insufficient opportunity to practice Instructional Planning Form revealed Rebecca had 15 minutes of independent reading practice per day Observation revealed that during SSR, Rebecca often played with items on desk. CBM probes indicated that she was not performing comparable to peers She lacks the phonological skills to read the text fluently. Review of records indicated that she had done well in remedial phonics program the previous year and was graduated out of the program. IPF suggested that instructional time was given to phonological skills. CBM probes indicated that she was not performing comparable to peers; DIBELS assessments did not point to a phonological awareness or phonics problem. Her reading level is not instructionally appropriate. Teacher reports showed that she was at the 25th percentile in the Spring of 1st grade Teacher felt that curriculum was appropriate Survey level assessment revealed that Rebecca had mastered 1st grade text

Case Example: Rebecca Rebecca reads 41 WRC per minute in the Fall of 2nd grade while her peers read 80 WRC per minute and is improving at a limited rate (.5 ROI) because … She had insufficient opportunity to practice.

What was the impact of progress monitoring assessments for Rebecca and her teachers?

Rebecca’s progress during Reader’s Theatre

3 - 2 - 1 3 things you learned today 3 - 2 - 1 3 things you learned today 2 things you still have questions about 1 statement about how you feel about PM now

We will have tough choices to make – we’ll decide based on what’s best for our kids

If you’re not hopelessly confused, you’re out of touch If you’re not hopelessly confused, you’re out of touch! If you are hopelessly confused, then you only have one choice — try stuff. Tom Peters, Embracing Chaos, 1993

We will make mistakes – and fix them along the way

Thank You! Questions: Laura Hauerwas lhauerwa@providence.edu Kristen Matthes matthesk@bpsmail.org

References Deno, S., Lembke, E. and Anderson, A. (2005) Progress Monitoring Study Group Content Module available at www.studentprogress.org. Fuchs and Fuchs (April, 2006) Progress Monitoring: Identifying LD and Improving Student Outcomes. Presentation at National SEA Conference on SLD Determination. Kansas City, MO. Hawkins, and Tilly, D. (Sept. 2005). Response to Intervention On the Ground: Diagnosing the Learning Enabled. Pacific Northwest Institute on Special Education and the Law. Shapiro, E. (2006). Advanced Issues in Monitoring Student Progress in Reading. Presentation for Pennsylvania Training and Technical Assistance Network on on RTI. Tilly (Sept, 2005) Problem Identification and Analysis. Presentation at the Principal Lead Problem Solving Initiative, Highland Park, IL.