Progress Monitoring Goal Setting/Graphing Administration of ORF/Maze Keith Drieberg, Director of Psychological Services John Oliveri, School Psychologist Cathleen Geraghty, School Psychologist

Overview (Continued) Comparison of Terms Between Systems TermAIMSWEBDistrictGrade Level Frequency I/II/IIILevelsTiers III (Red Zone)Progress Monitor IntensiveInstructionalWeekly II (Yellow Zone)StrategicTargetedAssigned1 x Monthly I (Green Zone)BenchmarkSchool-wide Screenings Assigned3 Times a Year

Screening  Screening allows for quick identification of students that are at-risk for academic difficulties  Different Screening Measurements  When to do Screenings  Who is Screened?

Screening When and Why?  1st Screening - identification  2nd Screening - prediction and identification  3rd Screening - goal attainment

Which Measures  Kindergarten Fall - Letter Naming Fluency (LNF) Winter - LNF & Phoneme Segmentation Fluency (PSF) Spring - LNF & PSF  1st Grade Fall - Nonsense Word Fluency (NWF) & Oral Reading Fluency (ORF) Winter - NWF & ORF Spring - NWF & ORF  2nd Grade Fall - ORF Winter - ORF Spring - ORF

Administration for Early Literacy

Letter Naming Fluency

Example: Letter Naming Fluency

Phoneme Segmentation Fluency

Example: Phoneme Segmentation Fluency

Nonsense Word Fluency

Example: Nonsense Word Fluency

Who is Screened?  Ideally, all students are screened in the Fall, Winter, and Spring  Students scoring below the 25th percentile are considered at-risk and should be progress monitored

What Is & Why Progress Monitor  What is Progress Monitoring Formative assessment tool  Why Progress Monitor Tell us whether students are profiting from the curriculum, and whether or not an intervention is effective for that particular student  Some Everyday Examples of How We Use Informal Progress Monitoring

What is Progress Monitoring  Technically adequate reliability and validity  Capacity to model growth able to represent student achievement growth within and across academic years  Treatment sensitivity scores should change when students are learning  Independence from specific instructional techniques instructionally eclectic so the system can be used with any type of instruction or curriculum  Capacity to inform teaching should provide information to help teachers improve instruction  Feasibility must be doable

Why Progress Monitor  Screening is not enough for some students because they may be in ineffective programs for too long.  Progress monitoring allows for individualized goals to be written and determination of a feasible amount of time for the goal to be reached.  Allows for an analysis of student need and resources for determining progress monitoring frequency. Programs that are more intensive (e.g., special education), should monitor student outcomes more frequently that 3x per year.

More Frequent Evaluation

Progress Monitoring Plan  Progress Monitoring Plan No Learning Center  Monitor at-risk students, once a month at grade level Learning Center  Monitor at-risk students once a month at grade level every week at instructional level  Collect data on Wednesday or Thursday

Goal Setting - General  Each student should have a year-long goal (you want the student to be performing at the 50th percentile on grade level material)  Every goal should have: Time frame (when the goal should be reached) Behavior (what the desired level of performance is) Condition (which measure and where you obtained it from) Criterion (which grade level passage you are using - if applicable)

Sample ORF Goal  Oral Reading Fluency (Fluency) In (#) of weeks (student name) will read (#) word correct in 1 minute as measured by a (grade ___ ) (DIBELS or AIMSWeb Oral Reading Fluency Measure).

Norms and Growth Rates

Year-Long Goal  Expected weekly growth If the student is not at-risk use the average rate of growth If the student is at-risk use the ambitious rates of growth (e.g., below the 25th percentile)  Multiply the growth rate by 36 (weeks of school) and add it to the baseline level of performance

Year-Long Goal Example  In the fall, a 4th grade student’s instructional level is 2nd grade and the student reads 22 words correct per minute. Ambitious growth for 2nd grade is 2 words per week 22 + (2 x 36) = 94 The year-long goal would be 94 words correct per minute

Short-Term Goal  All students being progress monitored need shorter-term goals  These goals should use ambitious rates of growth to help the students ‘catch-up’

Short-Term Goals  Determine what level of performance is needed to move up a percentile rank (e.g., 10th to 25th; 25th to 50th)  Take the difference in performance (e.g., how many words correct the student will need to move from the 10th to 25th percentile), and divide by the ambitious rate of growth This product will be the approximate number of weeks needed for the student to reach the next percentile rank

Short-Term Goals  If using early literacy measures (LNF, PSF, NWF) move up instructional level until the student has mastered that skill, then move up to the next skill

Short-Term Goals  In the Winter, a 3rd grade student’s instructional level is 2nd grade and the student reads 29 words correct per minute. What short-term (progress monitoring) goal would you write for this student?

Charting Progress  Once you set the goal, you need to chart progress to see if the student is responding to the change in instruction

Evaluating Goal Attainment  Process of assessing student achievement during instruction to determine whether an instructional program is effective for individual students. When students are progressing, keep using your instructional programs. When tests show that students are not progressing, you can change your instructional programs in meaningful ways. Has been linked to important gains in student achievement (Fuchs, 1986) with effect sizes of.7 and greater.

Trendline Goal Line

Trendline

Goal line Trend Line

Special Education How can Progress Monitoring be used in Special Education?

Educational Benefit  Progress monitor at both instructional and grade level Grade Level Goals  How is the student performing relative to his/her peers? Instructional Level Goals  How is the student’s performance changing as a function of the current instruction / intervention? Instructional level progress monitoring is sensitive to growth.

General Goal Setting  Long-Term Goal The long-term goal should be either to move the student:  Up a full grade level (Ex. 50 th percentile on a 1 st grade probe to 50 th percentile on a 2 nd grade probe)  Up to a higher skill (Ex. Letter Naming Fluency to Letter Sound Fluency)

General Goal Setting (continued)  Short-Term Goals (Benchmark 1 & 2) The benchmark goals should be to move the student up a percentile level within their instructional level (could be the same as grade level). (Ex. Move from the 25 th to the 50 th percentile).

First Example (Topic Two ) The Movie “City Slickers” The Cattle Drive----

That Great Line:  “Don’t Know Where We Are At;  Don’t Know Where We Are Going;  But We Are Sure Are Making A Lot Of Progress”

That Great Line: A Case For Level I Bench Mark Screenings At Assigned Grade Levels For All Students Don’t Know Where We Are At No Base Line Point Don’t Know Where We Are Going No Goal Point Not Sure Of Progress No Goal Line or Aimline

What We Already Know : A Case For Level II Strategic/ Targeted Assigned Grade Level Progress Monitoring On a Regular Basis The Earlier We Start Interventions; The Less Behind the Student is in Comparison to Their Grade Level Peers; and The Sooner A Student Starts to Makes Progress Their Peer Growth Rate Is High And Student “Catches Up Quickly” (within about a year) Because They are Not Too Far Behind (Assigned Grade Level Progress Monitoring / Strategic/Targeted – ideal 1 x Monthly)

What We Need to Think More About: A Case For Level III Progress Monitor/Intensive Instructional Level Progress Monitoring  Higher Grade Levels (Especially 4 th Grade +)  Student Significantly Farther Behind Assigned Grade Level Peers  Need Long Term Goals (Takes Longer Than One Year to “Catch Up”)  Kids Give Up On Themselves (Highest Drop Out Rate 9 th Grade)  We Expect Less of Students at Higher Grade Levels and Blame Others  (Instructional Grade Level Progress Monitoring To Know If Intervention is Workng; Appreciate Progress Sooner; and Receive Frequent Feedback )

What We Need to Think More About: Instructional Grade Level Progress Monitoring  Higher Grade Levels (Especially 4 th Grade +)  Student Significantly Farther Behind Assigned Grade Level Peers  Need Long Term Goals (Takes Longer Than One Year to “Catch Up”) In Addition,  Peer Growth Rate Is Slower at Higher Grade Levels  If We and the Student Continue Ambitious Goals Each Year  The Student Will Make Significant Gains and Eventually “Catch Up”-  (Instructional Grade Level Progress Monitoring)

Why We Need Both Assigned Grade Level and Instructional Grade Level Goals and Progress Monitoring 6 th Grade90 WC 7 th Grade120 WC 8 th Grade150 WC

Administration of ORF and MAZE

ORF (R-CBM) Administration  Student’s read the passage aloud for 1 minutes  Number of words read correct and number of errors are counted WRC/errors

Administration and Scoring of R-CBM  What Examiners Need to Do Before testing students While testing students After testing students

Things You Need Before Testing Standard Reading Assessment Passage Student Copy: No numbers No numbers Between words (exception: 1st grade) Between words (exception: 1st grade) An informative first sentence An informative first sentence Same font style and size Same font style and size Text without pictures Text without pictures

Things You Need Before Testing (Continued) Standard Reading Assessment Passage Examiner Copy: Pre-numbered so they can be scored quickly and immediately.

R-CBM Standard Directions for 1 Minute Administration unnumbered 1)Place the unnumbered copy in front of the student. numbered 2)Place the numbered copy in front of you, but shielded so the student cannot see what you record. 3)Say: When I say ‘Begin,’ start reading aloud at the top of this page. Read across the page (DEMONSTRATE BY POINTING). Try to read each word. If you come to a word you don’t know, I will tell it to you. Be sure to do your best reading. Are there any questions? (PAUSE) “” 4)Say “Begin” and start your stopwatch when the student says the first word. If the student fails to say the first word of the passage after 3 seconds, tell them the word, mark it as incorrect, then start your stopwatch. 5)Follow along on your copy. Put a slash ( / ) through words read incorrectly. “.” 6)At the end of 1 minute, place a bracket ( ] ) after the last word and say, “Stop.” 7) Score and summarize by writing WRC/Errors

“Familiar” Shortened Directions When students are assessed frequently and know the directions. Say: When I say ‘Begin,’ start reading aloud at the top of this page.

Items to Remember Emphasize Words Read Correctly (WRC). Get an accurate count. 3-Second Rule No other corrections Discontinue rule Be polite Best, not fastest Interruptions

What is a Word Read Correctly? Correctly pronounced words within context Correctly pronounced words within context Self-Corrected Incorrect Self-Corrected Incorrect Words within 3 seconds

What is an Error? Mispronunciation of the Word or Substitutions Mispronunciation of the Word or Substitutions Omissions Omissions Stops or struggles with a letter for more than 3 seconds (examiner provides correct word) Stops or struggles with a letter for more than 3 seconds (examiner provides correct word)

What is not Incorrect? (Neither a WRC or an Error) Repetitions Repetitions Dialect Dialect Differences Insertions Insertions (Consider them Qualitative Errors)

Calculating and Reporting Scores Count the total number of words the student read Count the total number of words the student read Count the number of errors and subtract Count the number of errors and subtract Report in standard format of WRC/Errors (72/3) Report in standard format of WRC/Errors (72/3)

Example of Calculating Scores Juan finished reading after 1 minute at the 145th word, so he read 145 words total Juan finished reading after 1 minute at the 145th word, so he read 145 words total Juan also made 3 errors Juan also made 3 errors Therefore, his WRC was 142 with 3 errors Therefore, his WRC was 142 with 3 errors Reported as 142/3

Practice Exercise 1: Let’s Score

This student read 72 WRC/8 Errors Practice Exercise 1: Answer Key

Practice Exercise 2: Let’s Score

This student read 96 WRC/4 Errors Practice Exercise 2: Answer Key

Practice Exercise 3: Let’s Score

This student read 141 WRC/2 Errors Practice Exercise 3: Answer Key

Curriculum Based Measurement Reading Maze AreaTimingTest ArrangementsWhat is Scored? CBM Maze Reading 3 minutes Individual, Small Group, or Classroom Group # of Correct Answers CBM Maze is designed to provide educators a more complete picture of students’ reading skills, especially when comprehension problems are suspected.

Curriculum Based Measurement Reading Maze (Continued) Maze is a multiple-choice cloze task that students complete while reading silently. Maze is a multiple-choice cloze task that students complete while reading silently. The students are presented with word passages. The students are presented with word passages. The first sentence is left intact. The first sentence is left intact. After the first sentence, every 7 th word is replaced with three word choices inside a parenthesis. After the first sentence, every 7 th word is replaced with three word choices inside a parenthesis. The three choices consist of, The three choices consist of, 1)Near Distracter 2)Exact Match 3)Far Distracter

An example of CBM Maze

Administration and Scoring of CBM Maze What Examiners Need to Do... Before testing students While testing students After testing students

Items Students Need Before Testing What the Students Need for Testing: CBM Maze practice test (optional) CBM Maze practice test (optional) Appropriate CBM Maze passages Appropriate CBM Maze passages Pencils Pencils

Items Administrators Need Before Testing What the Tester Uses for Testing: Stopwatch Stopwatch Appropriate CBM Maze Answer Key Appropriate CBM Maze Answer Key Appropriate Standardized Directions Appropriate Standardized Directions

Setting up Assessment Environment The classroom if assessing the entire class. A cluster of desks or small tables in the classroom for small group assessment. Individual desks or “stations” for individual assessment. Assessment environments are flexible and could include… Assessment environments are flexible and could include…

Things You Need to do While Testing Attach a cover sheet that includes the practice test so that students do not begin the test right away. Attach a cover sheet that includes the practice test so that students do not begin the test right away. Do a simple practice test with younger students. Do a simple practice test with younger students. Monitor to ensure students are circling answers instead of writing them. Monitor to ensure students are circling answers instead of writing them. Be prepared to “Prorate” for students who may finish early. Be prepared to “Prorate” for students who may finish early. Try to avoid answering student questions. Try to avoid answering student questions. Adhere to the End of Timing. Adhere to the End of Timing. Follow the Standardized Directions Follow the Standardized Directions

CBM Maze Standard Directions 1)Pass Maze tasks out to students. Have students write their names on the cover sheet, so they do not start early. Make sure they do not turn the page until you tell them to. 2)Say this to the student (s): When I say ‘Begin’ I want you to silently read a story. You will have 3 minutes to read the story and complete the task. Listen carefully to the directions. Some of the words in the story are replaced with a group of 3 words. Your job is to circle the 1 word that makes the most sense in the story. Only 1 word is correct. 3)Decide if a practice test is needed. Say... Let’s practice one together. Look at your first page. Read the first sentence silently while I read it out loud: ‘The dog, apple, broke, ran after the cat.’ The three choices are apple, broke, ran. ‘The dog apple after the cat.’ That sentence does not make sense. ‘The dog broke after the cat.’ That sentence does not make sense. ‘The dog ran after the cat.’ That sentence does make sense, so circle the word ran. (Make sure the students circle the word ran.)

CBM Maze Standard Directions (Continued) Let’s go to the next sentence. Read it silently while I read it out loud. ‘The cat ran fast, green, for up the hill. The three choices are fast, green, for up the hill. Which word is the correct word for the sentence? (The students answer fast) Yes, ‘The cat ran fast up the hill’ is correct, so circle the correct word fast. (Make sure students circle fast) Silently read the next sentence and raise your hand when you think you know the answer. (Make sure students know the correct word. Read the sentence with the correct answer) That’s right. ‘The dog barked at the cat’ is correct. Now what do you do when you choose the correct word? (Students answer ‘Circle it’. Make sure the students understand the task) That’s correct, you circle it. I think you’re ready to work on a story on your own.

CBM Maze Standard Directions (Continued) 4)Start the testing by saying... When I say ‘Begin’ turn to the first story and start reading silently. When you come to a group of three words, circle the 1 word that makes the most sense. Work as quickly as you can without making mistakes. If you finish a/ the page/first side, turn the page and keep working until I say ‘Stop’ or you are all done. Do you have any questions? 5)Then say, ‘Begin.’ Start your stopwatch. 6)Monitor students to make sure they understand that they are to circle only 1 word. 7)If a student finished before the time limit, collect the student’s Maze task and record the time on the student’s test booklet. 8)At the end of 3 minutes say: Stop. Put your pencils down. Please close your booklet. 9)Collect the Maze tasks.

CBM Maze Familiar Directions 1)After the students have put their names on the cover sheer, start the testing by saying... When I say ‘Begin’ turn to the first story and start reading silently. When you come to a group of three words, circle the 1 word that makes the most sense. Work as quickly as you can without making mistakes. If you finish a/ the page/first side, turn the page and keep working until I say ‘Stop’ or you are all done. Do you have any questions? 2)Then say, ‘Begin.’ Start your stopwatch. 3)Monitor students to make sure they understand that they are to circle only 1 word. 4)If a student finished before the time limit, collect the student’s Maze task and record the time on the student’s test booklet. 5)At the end of 3 minutes say: Stop. Put your pencils down. Please close your booklet. 6)Collect the Maze tasks.

CBM Maze Scoring What is correct? The students circles the word that matches the correct word on the scoring template. What is incorrect? An answer is considered an error if the student: 1)Circles an incorrect word 2)Omits word selections other than those the student was unable to complete before the 3 minutes expired

Making Scoring Efficient 1)Count the total number of items up to the last circled word. 2)Compare the student answers to the correct answers on the scoring template. Mark a slash [/] through incorrect responses. 3)Subtract the number of incorrect answers from the total number of items attempted. 4)Record the total number of correct answers on the cover sheet followed by the total number of errors (e.g., 35/2).

CBM Maze Prorating 1)When the student finished must be recorded and the number of correct answers counted. For example, the student may have finished in 2 minutes and correctly answered 40 items. 2)Convert the time taken in seconds. (2 minutes = 120 seconds) 3)Divide the number of seconds by the number correct. (120/40 = 3) 4)Calculate the number of seconds in the full 3 minutes. (3 minutes = 180 seconds) 5)Divide the number of full seconds by the calculated value from step 3. (180/3 = 60) If a student finishes all the items before 3 minutes, the score can be prorated.

Practice Exercise 1: Let’s Score

Practice Exercise 2: Let’s Score

Interrater Reliability  Why do we need to calculate this? Reliability of reporting for student data Protect against observer drift  Agreements / (Agreements + Disagreements) x 100 = Interrater reliability

Practice Administration and Reliability Checks  Page 21 in the Administration and Scoring of Reading Curriculum-Based Measurement (R-CBM) Training Workbook Break into groups of three  1 person read  2 people score (primary scorer/observer) - calculate Interrator reliability among the two scorers  Rotate until every person has been a reader, a primary scorer, and an observer