Administration and Scoring of AIMSweb Probes Presented by: Allison Whitney, School Psychologist & Rachel Minelli, Educational Diagnostician

Objectives Discuss characteristics and uses of Curriculum-Based Measurement as compared to other types of assessment. Learn the key guidelines related to the administration of AIMSweb CBMs. Identify the methods used for scoring AIMSweb CBMs.

Summative vs. Formative Assessment Summative Assessment: Culmination measure. Mastery assessment. Assessment after instruction. Pass/fail type assessments which summarize the knowledge students learn. Typical summative assessments include: End of chapter tests High-stakes tests (e.g., State assessments) GRE, ACT, SAT, GMAT, etc. tests Driver’s license test Final Exams. Formative Assessment: Process of assessing student achievement frequently during instruction to determine whether an instructional program is effective for individual students. Informs: When students are progressing, continue using your instructional programs. When tests show that students are not progressing, you can change your instructional programs in meaningful ways. (Shinn, Shinn, & Langell, 2008) Summative = assessment OF learning while Formative = assessment FOR learning. Summative is what HAPPENED , Formative is what is happening. This would include data collected as part of an intervention package.

Curriculum-Based Measurement An example of FORMATIVE assessment used to assess skills such reading, spelling, mathematics, and written language. CBMs are designed to serve as “indicators” of general reading achievement: CBM probes don’t measure everything, but they measure the important things. CBM involves monitoring student progress through direct, continuous assessment of basic skills. CBM probes are typically quick to administer and simple to score which allows them to be given frequently to provide continuous progress data. Student performance is scored for speed and accuracy to determine proficiency. The results are charted to provide for timely evaluation based on hard data. (Shinn, Shinn, & Langell, 2008) Correlates strongly with research-supported methods for assessment and intervention Focus is on repeated measures of performance (This cannot be done with most norm-referenced and standardized tests due to practice effect or limited forms.) Also referred to as General Outcome Measure : Examples in other fields: Medical (weight. Blood pressure) Business (items sold).

Curriculum-Based Measurement Standardized tests to be given, scored, and interpreted in a standard way. Researched with respect to psychometric properties to ensure accurate measures of learning CBM is endorsed by the United States Department of Education as a method for assessing student progress. CBM was initially developed more than 20 years ago by Stanley Deno and others at the University of Minnesota Institute for Research on Learning Disabilities to develop a reliable and valid measurement system for evaluating basic skills growth and is supported by 30 years of school- based research. (Shinn, Shinn, & Langell, 2008) Are sensitive to improvement in brief intervals of time Also can tell us how students earned their scores (offers opportunity to gather qualitative information) Designed to be as short as possible (2-4min) to ensure its “do-ability” Are linked to decision making for promoting positive achievement and Problem-Solving

Curriculum Based Measurement Across Tiers Tier One: All Students Benchmark: Through Universal Screeners Three times per year To identify students at-risk Tier Two: Some Students (Mild to Moderate Risk) Strategic Monitoring Curriculum Based Measurement probes At least once per month To monitor students with some risk Tier Three: Some Students (At-Risk) Intensive monitoring towards a goal Once per week To closely monitor at-risk students (Shinn, Shinn, & Langell, 2008) Make Yellow and Red Associations

AIMSweb Curriculum-Based Measures Available: Tests of Early Literacy Letter Naming Fluency Letter Sound Fluency Phoneme Segmentation Nonsense Word Fluency Tests of Early Numeracy Oral Counting Number Identification Quantity Discrimination Missing Number R-CBM (Oral Reading Fluency MAZE (Reading Comprehension) M-COMP (Math Computation) M-CAP (Math Concepts and Applications) S-CBM (Spelling) WE-CBM (Written Expression) Also single skill probes for math

Benchmarking Fall: September 1st –October 15th Winter: January 1st –January 31st Spring: May 1st –May 31st Benchmarking should be completed within two weeks of starting. The shorter the time the better as this reduces the effect of additional instruction for some students. Benchmark probes should always be given at grade level.

Assessment Recommendations: Kindergarten Fall Winter Spring Letter Naming Fluency Letter Sound Fluency Phoneme Segmentation Nonsense Word Oral Counting Number Identification Quantity Discrimination Missing Number

Assessment Recommendations: First Grade Fall Winter Spring Letter Naming Fluency Letter Sound Fluency Phoneme Segmentation Nonsense Word Oral Counting Number Identification Quantity Discrimination Missing Number R-CBM M-COMP S-CBM WE-CBM

Assessment Recommendations: Second Grade Fall Winter Spring R-CBM M-COMP M-CAP S-CBM WE-CBM

Assessment Recommendations: Third – Eighth Grades Fall Winter Spring R-CBM MAZE M-COMP M-CAP S-CBM WE-CBM

R-CBM Before Testing: Materials: Environment: Student Probe Administrator Probe Stopwatch Pencil Environment: Quiet (students should NOT hear each other reading). Can also administer on the computer!

During Testing (Shinn & Shinn) Follow the standardized directions: R-CBM is a standardized test Administer the assessment with consistency Remember it’s about testing, not teaching Don’t teach or correct Don’t practice reading the passages Remember best, not fastest reading Sit across from, not beside student (Shinn & Shinn) These rules apply to all probes Okay modifications

During Testing Student receives unnumbered copy while the teacher follows along on the numbered copy. Say: When I say ‘Begin,’ start reading aloud at the top of this page. Read across the page (DEMONSTRATE BY POINTING). Try to read each word. If you come to a word you don’t know, I will tell it to you. Be sure to do your best reading. Are there any questions? (PAUSE) Say “Begin” and start your stopwatch when the student says the first word. If the student fails to say the first word of the passage after 3 seconds, tell them the word, mark it as incorrect, then start your stopwatch. Follow along on your copy. Put a slash ( / ) through words read incorrectly. At the end of 1 minute, place a bracket ( ] ) after the last word and say, “Stop.” Score and summarize by writing WRC/Errors (Shinn &Shinn)

What Happens If… The student skips a row? Mark all skipped words as errors and continue. The student pauses on a word? After waiting for 3 seconds, provide the student with the word and mark it as an error. The student inserts or repeats a word? No error is marked. The student omits a word? Score the omission as an error. The fire alarm goes off? Discontinue testing and administer an alternate probe. The student self-corrects? Count the word as correctly read The student mispronounces a word? Count the word as an error

After Testing Score immediately Indicate score by words read correct/number of errors. Total words =180 Errors= 7 Score is recorded as 173/7

Let’s Practice!

Practice! Talk about what scores mean fluency vs. accuracy, what types of errors,

MAZE R-MAZE is a multiple-choice cloze task that students complete while reading silently. The students are presented with 150-400 word passages. The first sentence is left intact. After the first sentence, every 7th word is replaced with three word choices inside a parenthesis. (Kennedy)

Sample Grade 4 R-MAZE Passage "Where are you going, Dad?" I ask excitedly. I wonder if something interesting is (followed, happening, shuffling). "I'm going to search for some (deer, stop, pink). Would you like to come along? (Who, Want, We'll) take a trek in the woods," (replies, eating, ground) Dad. "I love going for walks. (Her, Live, Wait) for me!" I reply. "I want (for, to, and) go too!" yells Mike, my younger (brother, clicks, headed). "Please help me tie my shoes!“ "(We’ll, Deer, Don't) worry, Mike. I will help you. (His, Dad, If) always waits for both of us," (Me, I, We) explain calmly.

Before Testing Administrators need : Stopwatch Students need : MAZE Cover Sheet MAZE Worksheet Something to write with Administrators need : Stopwatch Standardized Instructions

During Testing Can administer in the classroom setting. Provide cover sheet or present tests upside down to prevent some students from starting right away. Do a simple practice test with younger students. Monitor to ensure students are circling answers instead of writing them. Be prepared to prorate for students who may finish early. Try to avoid answering student questions. Adhere to the end of timing.

MAZE Instructions Decide if a practice test is needed. Say . . . “Let’s practice one together. Look at your first page. Read the first sentence silently while I read it out loud: ‘The dog, apple, broke, ran after the cat.’ The three choices are apple, broke, ran. ‘The dog apple after the cat.’ That sentence does not make sense. ‘The dog broke after the cat.’ That sentence does not make sense. ‘The dog ran after the cat.’ That sentence does make sense, so circle the word ran. (Make sure the students circle the word ran.)

…continued Let’s go to the next sentence. Read it silently while I read it out loud. ‘The cat ran fast, green, for up the hill. The three choices are fast, green, for up the hill. Which word is the correct word for the sentence? (The students answer fast) Yes, ‘The cat ran fast up the hill’ is correct, so circle the correct word fast. (Make sure students circle fast). Silently read the next sentence and raise your hand when you think you know the answer. (Make sure students know the correct word. Read the sentence with the correct answer). That’s right. ‘The dog barked at the cat’ is correct. Now what do you do when you choose the correct word? (Students answer ‘Circle it’. Make sure the students understand the task). That’s correct, you circle it. I think you’re ready to work on a story on your own.

Administration Instructions Say this to the student(s): When I say ‘Begin’ I want you to silently read a story. You will have 3 minutes to read the story and complete the task. Listen carefully to the directions. Some of the words in the story are replaced with a group of 3 words. Your job is to circle the 1 word that makes the most sense in the story. Only 1 word is correct.

During Testing After reading standardized instructions: Say, ‘Begin.’ Start your stopwatch. Monitor students to make sure they understand that they are to circle only 1 word. If a student finished before the time limit, collect the student’s R-MAZE task and record the time on the student’s test booklet. At the end of 3 minutes say: “Stop. Put your pencils down. Please close your booklet.” Collect the R-MAZE tasks.

After Testing Score immediately What counts as an error? Circles an incorrect word Omits word selections (other than those the student was unable to complete before the 3 minutes expired). Record the number of correct answers over the number of errors (CR/E).

What if they finish early?? The student’s score will need to be prorated: When the student finished, the time must be recorded and the number of correct answers counted. For example, the student may have finished in 2 minutes and correctly answered 40 items. Convert the time taken in seconds. (2 minutes = 120 seconds) Divide the number of seconds by the number correct. (120/40 = 3) Calculate the number of seconds in the full 3 minutes. (3 minutes = 180 seconds) Divide the number of full seconds by the calculated value from step 3. (180/3 = 60)

M-COMP: Before Testing Administrator will need: Standardized Instructions Student copy of probe (make sure to remove the answer key) Stopwatch Students will need: Pencil Students may use scratch paper but not calculators.

M-COMP: Directions “We are going to take an 8-minute math test. Read the problems carefully and work each problem in the order presented, starting at the first problem on the page from left to right. Do not skip around” “If you do not understand how to do a problem, mark it with an X and move on. Once you have tried all of the problems in order, you may go back to the beginning of the worksheet and try to complete the problems, you marked.” “Although you may show your work and use scratch paper if that is helpful for you in working the problems, you may not use calculators or any other aids.” “Keep working until you have completed all of the problems or I tell you to stop.” “Do you have any questions?” “Here are your tests. Do not turn the test back over or start working until I tell you to begin.” “Begin.” After eight minutes say, “Stop and put down your pencil”. Tells them to put name date and teacher’s name on each page and then turn the test over.

Scoring Scoring sheets allow for simple and efficient scoring. Responses are scored on a “all or none” principle. The answer key provides some examples instances where multiple answers are correct. But, if you come across an answer that is not indicated on the key but demonstrates understanding of the concept being measured and is correct, you may assign credit. Problems worth different points.

M-CAP: Before Testing Administrator will need: Students will need: Standardized Instructions Student probe (remember to remove answer key) Stopwatch Students will need: Pencil

M-CAP Instructions “We’re going to take an 8 (or 10) minute math test. Read the problems carefully and work each problem in the order presented. Do not skip around.” “If you do not know how to do a problem, mark it with an X and move on. Once you have tried all of the problems in order, you may go back to the beginning of the worksheet and try to complete the problems you marked.” “Write the answers to the problems in the blanks. For multiple choice questions, place the letter(A, B, or C) of the correct answer in the blank.” “You do not have to show your work, but you may if that is helpful for you in working the problems.” “Keep working until you have completed all of the problems or I tell you to stop. Do you have any questions.”

M-CAP: During Testing If a student asks a question say, “Read the directions, again and work the problem as best you can. If you still do not understand the problem or are unable to work it, you may move on to the next question.” If a student is skipping around or crossing out the difficult problems say, “Try to work each problem. Do not skip around.” Allow 8 minutes for grades 2-6 and 10 minutes for grades 7-12.

M-CAP: After Testing Score using the answer key provided on the student probe. Can conduct error analysis using the M-CAP Curriculum Reference.

WE-CBM: Before Testing Administrators: Story starter(s) Standardized Instructions Stopwatch Students: Lined paper Pencil (Powell-Smith & Shinn, 2004)

WE-CBM: During Testing Instructions: “You are going to write a story. First, I will read a sentence, and then you will write a story about what happens next. You will have 1 minute to think about what you will write and 3 minutes to write your story. Remember to do your best work. If you don’t know how to spell a word, you should guess. Are there any questions? Put your pencils down and listen.” “For the next minute, think about…(Insert story starter). After thirty seconds say, “You should be thinking about…(insert story starter). At the end of one minute say, “Now begin writing.” After ninty seconds say, “You should be thinking about… (insert story starter). At the end of three minutes say, “Stop. Put your pencils down.” (Powell-Smith & Shinn, 2004)

WE-CBM: During Test If a student stops writing for about ten seconds, say, “Keep writing the best story you can.” Do not answer questions about spelling, punctuation, etc. (Powell-Smith & Shinn, 2004)

WE-CBM: Scoring Important to print manual and refer to when scoring. Total Words Written: Word = any letter or group of letters separated by a space, even if the word is misspelled or is a nonsense word. Hyphenated words: Each morpheme is counted as one word if it can stand alone. Ex. Sister-in-law = 3 while re-evaluate = 1 Abbreviations: Commonly used abbreviations are counted as words. Numbers: Dates, currency, and written out numbers count as words. (Powell-Smith & Shinn, 2004)

WE-CBM: Scoring Correct Writing Sequence: Two adjacent writing units (words and punctuation) that are correct within the context of what is written. Rules: Pairs of words must be spelled correctly. Words must be capitalized and punctuated correctly with the exception of commas. Words must be syntactically correct. Words must be semantically correct. Contractions must contain apostrophes if the word can’t stand alone without it. Words containing reversed letters are included in the total CWS count unless the reversed letter causes a word to be spelled incorrectly. Story titles or endings must be capitalized and spelled correctly. Commonly used abbreviations are included in CWS count. Hyphenated words are counted correctly as long as each morpheme is spelled correctly. With the exception of dates, numbers are not included. Symbols are not counted. (Powell-Smith & Shinn, 2004)

WE-CBM: Scoring Words Spelled Correctly: Must be spelled correctly within the context of the story. Reed vs. read or red vs. read Each morpheme in a hyphenated word counts if it can stand alone. Proper nouns must be capitalized to count as correct. Words with reversed letters are counted as correct as long as the reversal does not result in an incorrect spelling. Apostrophes must be present in contractions to be counted as correct. Qualitative Features and Error Analysis worksheets available in the manual. (Powell-Smith & Shinn, 2004)

Administration Summary Area Timing Administration Scoring R-CBM 1 minute Individual WRC/Errors MAZE 3 minutes Individual, Small Group, Group CR/Errors M-COMP 8 minutes Digits Correct M-CAP 10 minutes WE-CBM 4 minutes 1-Think 3-Write TWW WSC CWS

References Kenndy, Jillyan. Administration and scoring of Reading MAZE for use in general outcome measurement. Mathematics Computation: Administration and Technical Manual (2010) AIMSweb, Pearson Mathematics Concepts and Applications: Administration and Technical Manual (2009). AIMSweb, Pearson Powell-Smith, K.A. & Shinn, M. (2004). Administration and scoring of Written Expression Curriculum-Based Measurement (WE-CBM) for use in general outcome measurement. Pearson. Shinn, M.M. & Shinn, M.R. (2002) AIMSweb training workbook: Administration and scoring of Reading Curriculum-Based Measurement (R-CBM) for use in general outcome measurement. Pearson. Shinn, M.R. & Shinn, M.M. (2002). AIMSweb training workbook: Administration and scoring of Reading MAZE for use in general outcome measurement. Pearson Shinn, M.R. & Shinn, M.M. Administration and scoring or Reading Curriculum-Based Measurement for use in general outcome measurement. Pearson Shinn, M.R., Shinn, M.M. & Langell, L.A. (2008). Overview of curriculum-based measurement and AIMSweb. Pearson.