Dr. Barbara Foorman Florida Center for Reading Research (FCRR)

Florida Assessments for Instruction in Reading (FAIR) – Using Scores for Growth and Instruction
Dr. Barbara Foorman Florida Center for Reading Research (FCRR) Stuart Greenberg Florida Department of Education

© 2011 Florida Department of Education
What is FAIR? A K-2 assessment system administered to individual students 3 times a year, with electronic scoring, Adobe AIR version, and PMRN reports linked to instructional resources. A 3-12 computer-based system where students take the assessments 3 times a year. Several tasks are adaptive. PMRN reports are available, linked to instructional resources. Printed toolkit available. © 2011 Florida Department of Education

The K-2 “Big Picture” Map
Broad Screen/Progress Monitoring Tool (BS/PMT) “All” students Letter Naming & Sounds Phonemic Awareness Word Reading Broad Diagnostic Inventory (BDI) “Some” students for vocabulary Listening Comprehension Reading Comprehension Vocabulary Spelling (2nd grade only) Targeted Diagnostic Inventory (TDI) “Some” students; some tasks K = 9 tasks 1st = 8 tasks 2nd = 6 tasks Ongoing Progress Monitoring (OPM) “Some” students K – 2 = TDI tasks 1 – 2 = ORF Let’s review the various components of the K-2 assessment. Review Slide. Notes to clarify what “all”, and “some” mean with regard to who must take the Broad Screen and Broad Diagnostic: Progress monitoring must be reported three times per year for students who have been identified with a reading deficiency, based upon locally determined assessments, statewide assessments, or through teacher observations. Students identified with a reading deficiency must be given intensive reading instruction immediately following the identification of the reading deficiency. For elementary students not participating in the statewide reading assessment (FCAT), substantial deficiency in reading must be defined by the district school board. For students required to participate in the statewide assessment, a substantial deficiency in reading is defined by scoring Level 1 or Level 2 on the Florida Comprehensive Assessment Test (FCAT) in Reading. The Broad Screen must be administered three times per year to students who have been identified with a reading deficiency. We HIGHLY recommend that districts assess all K-2 students, but districts may opt to assess just those students who have been identified with a reading deficiency. The assessments may be delivered to other students that the district identifies. Thus ‘All’ means the students that are rostered on the PMRN. © 2009 Florida Department of Education 3 3

K-2 Targeted Diagnostic Inventory (TDI) Map
Kindergarten Print Awareness Letter name and sound knowledge Phoneme Blending Phoneme Deletion Word Parts/Initial Letter Sound Connection Initial Letter Sound Connection Final Word Building –Initial Consonants Word Building –Final Consonants Word Building –Medial Vowels First Grade Letter Sound Knowledge Phoneme Deletion Initial Phoneme Deletion Final Word Building –Consonants Word Building –Vowels Word Building –CVC /CVCe Word Building –Blends Second Grade Word Building –Blends & Vowels Multisyllabic Word Reading The TDI is a set of tasks designed to more precisely indicate the areas of instructional need based upon performance on the Broad Screen. Here is a listing of the tasks by grade level.

The K – 2 “Score” Map BS/PMT PRS = Probability of Reading Success BDI
LC = Listening Comprehension Total questions correct (implicit/explicit) RC = Reading Comprehension Total questions correct (implicit/explicit), Fluency, Percent Accuracy Target Passage VOC = Vocabulary Percentile Rank SPL = Spelling TDI ME = Meets Expectations BE = Below Expectations OPM ORF = Adjusted Fluency OPM TDI Tasks = ME or BE and Raw Score When looking at the data provided by this assessment you will see several different types of scores. This table is a snapshot of the types of scores reported for each section of the assessment. You will see that the BDI contains the most score types. 5 5

Target RC Passages for Grades 1 and 2 (BDI)
Provide participants with the handout. This document provides some basic or broad guidelines for teachers to use with the Reading Comprehension task of the BDI. This chart provides teachers with target stories that can be used as a guideline to determine if a student is meeting developmental reading expectations for each assessment period. They received this during their training. During field studies, reading accuracy and comprehension data from a large, representative sample of Florida students were collected to establish these benchmarks or target stories. Data from the school year were also examined to make refinements. The goal is for students to read and understand the target story. How were Target Passages determined? Because of the huge variability in readability formulae on first and second grade text (due to the fact that “difficulty” is more a matter of within-word linguistic issues than word frequency of sentence length) we take an empirical route. In other words, we used our implementation study data to see what passage the majority of first and second graders read at each AP and designate that as the Target Passage. The goal is for students to read and understand the target story. This includes reading accurately (95 % of words read correctly), fluently (progressing toward meeting the end of the year wcpm target goals) and with understanding (answering at least 4 out of 5 comprehension questions). These guidelines were designated to support teachers in setting instructional reading goals for their students. This information, combined with teacher observation and classroom performance will help with instructional decision making for all students. It is important to note that students who are not successfully reading the target story for a designated assessment period may benefit from additional instructional support. Additional instruction can provide students with the knowledge and skills required to successfully read the target passage by the end of the year. Based on fluency norms from several research studies (Foorman, York, Santi, & Francis, 2008; Fuchs, Fuchs, Hosp, & Jenkins, 2001; Hasbrouck & Tindal, 2006), it is recommended that teachers use 60 wcpm as a target goal for the end of first grade and 90 wcpm as the target goal for the end of second grade. The passages in the BDI are primarily to measure comprehension and the OPM ORF passages, which provide the teacher with an adjusted fluency score (adjusted for passage difficulty), are the passages and scores that should be used to measure and monitor growth in fluency (wcpm). We will need to use these target passages when looking at the school status report. 6 Florida Center for Reading Research 6 6

Grade 1 PRS Chart ( )

Probability of Reading Success
Look for trends within grade levels, within teachers, and across grades. Let’s look across grades in our sample data. What do you notice when you compare across grades? 85% & above (Green) 16%-84% (Yellow) 15% & below (Red) K 28% 61% 11% 1st 36% 56% 8% 2nd 25% 70% 5%

Instructional Information from FAIR
In grades K-2: Are my students in the green zone (>.85) on the Broad Screen? At or above 50%ile in vocabulary? On the target LC or RC passage? If not, look at performance on the targeted diagnostic inventory tasks to see which skills students have mastered (>80%). Teach to those they haven’t mastered. © 2011 Florida Department of Education

Student Score Detail Box (K-2)
Excellent report to include in a student’s cumulative folder This is the full view of the Student Score Detail Box (SSDB) It lists the Grade and Year on the first line, the Student’s name and assessment period on the next line and then the student’s scores. In this box, is all of the same information presented on the main status report for the individual student and those scores not displayed on the main page. It includes all scores for the Broad Screen and BDI, but on this report comprehension questions are broken down by implicit/explicit. This is also where a teacher will find the raw scores for a student’s responses on the TDI. It can be printed from the pop-up box link (point to the print button at the bottom) or the teacher can print the detail box for each student in the whole class with one link from the class status report. This is also an excellent report to include in a student’s cumulative folder. ©2011 Florida Center for Reading Research 10

Questions to answer using the School Grade Summary Report
What is the distribution of scores in a particular grade level? The School Status told us the Median Score for Vocabulary was the 24th percentile, but now I see the full distribution and can see that the majority of my students fell within the 11th – 30th percentile range. This tells me that overall, the First Grade group is below average in their general vocabulary knowledge I would want to see if this is true in Kindergarten and Second Grade as well (on the School Status Report) Kindergarten Median Vocabulary = 29th percentile Second Grade Median Vocabulary = 41st I might look to see how the Core Reading Program addresses vocabulary and if we have any additional programs addressing vocabulary (maybe we are using something different or additional at second grade?)

Grades 3-12 Assessments Model
Broad Screen/Progress Monitoring Tool Reading Comprehension Task (3 Times a Year) Targeted Diagnostic Inventory Maze & Word Analysis Tasks Diagnostic Toolkit (As Needed) Ongoing Progress Monitoring If necessary This slide gives you a visual of the flow of the 3-12 WAM. You can see the four components of the assessment module that we just mentioned. Today we’ll take a much closer look at each of these four components. RC Screen Helps us identify students who may not be able to meet the grade level literacy standards at the end of the year as assessed by the FCAT without additional targeted literacy instruction. Mazes Helps us determine whether a student has more fundamental problems in the area of text reading efficiency and low level reading comprehension. Word Analysis Helps us learn more about a student's fundamental literacy skills--particularly those required to decode unfamiliar words and read accurately. © 2011 Florida Department of Education

Purpose of Each 3-12 Assessment
RC Screen Helps us identify students who may not be able to meet the grade level literacy standards at the end of the year as assessed by the FCAT without additional targeted literacy instruction. Mazes Helps us determine whether a student has more fundamental problems in the area of text reading efficiency and low level reading comprehension. Relevant for students below a 6th grade reading level. Word Analysis Helps us learn more about a student's fundamental literacy skills--particularly those required to decode unfamiliar words and read and write accurately. Let’s review the purpose of each type of assessment which we briefly discussed on the big picture map. 13

How is the student placed into the first passage/item?
Task Placement Rules Reading Comprehension - Adaptive For AP 1, the first passage the student receives is determined by: Grade level and prior year FCAT (if available) If no FCAT, students placed into a specific grade-level passage All 3rd grade students are placed into the same initial passage For AP 2 and 3, the first passage is based on students’ final ability score from the prior Assessment Period (AP). Maze – Not adaptive Two predetermined passages based on grade level and assessment period (AP). WA - Adaptive AP 1-3 starts with predetermined set of 5 words based on grade level. Student performance on this first set of 5 words determines the next words the student receives. 5-30 words given at each assessment period based on ability. Currently this estimate for Reading Comprehension is comprised of the student’s grade level and their prior year FCAT. If the student does not have a prior year’s FCAT, then the mean FCAT score for that school and that grade level is used instead. For AP 2 and 3, the first passage the student receives is based on the student’s prior FSP. If the child has not taken the RC Screen before then the logic for AP 1 is used. All students taking the Maze task receive two passages that are predetermined based on grade level and assessment period (AP). For example, all students in grade 7 at AP 2 would receive the same two passages. On the Word Analysis task, a predetermined set of 5 words is given based upon grade level at each assessment period. Based on how the student performs on these first five words will determine an estimate of ability. The student will then be given harder or easier words based on the estimate of ability until the data provides a reliable estimate of ability. The minimum number of words a student will receive is five and the maximum is thirty.

How is the student placed into subsequent passages?
Based on the difficulty of the questions the student answers correctly on the first passage, the student will then be given a harder or easier passage for their next passage. Difficulty of an item is determined using Item Response Theory (IRT). Because of this, the raw score of 7/9 for Student A and 7/9 for Student B, when reading the same passage, does not mean they will have the same converted scores.

The 3-12 “Big Picture” Map Type of Assessment Name of Assessment
Broad Screen/Progress Monitoring Tool (BS/PMT) – Appropriate for ‘All’ students Reading Comprehension (RC) Targeted Diagnostic Inventory (TDI) – “Some” students Maze Word Analysis (WA) Ongoing Progress Monitoring (OPM) – “Some” students ORF RC Informal Diagnostic Toolkit (Toolkit) – “Some” students Phonics Inventory Academic Word Inventory Scaffolded Discussion Templates Review the information to identify the components of the 3-12 assessment for administrators. Point out that the 3-12 TDI does not target specific skills like the TDI in K-2, but gives them a general direction and then they need to do a bit more digging with OPM and the informal toolkit to determine specific areas of need. Remind participants that the Broad Screen and TDI are progress monitoring tools administered three times a year and the OPM is progress monitoring for those students who require progress monitoring more often than the three times a year. The FLDOE has put the plan to have RC OPM available on hold due to limited funds for programming and demands on computational resources. The RC OPM mimics the Broad Screen – each month, the student would take a set of passages, one informational/expository and one narrative. ***Below is the explanation for why the slide says Appropriate for all students – but it is not required except for level 1 and 2 students. “Progress monitoring is required for all students scoring at Level 1 and 2 on the prior year’s FCAT Reading. The assessments may be delivered to other students that the district identifies. Thus ‘All’ means the students that are rostered on the PMRN” 16 16

The 3-12 “Score” Map Reading Comprehension - BS/PMT
FCAT Success Probability (FSP) Color- coded Percentile Standard Score Ability Score and Ability Range FCAT Reporting Categories Maze - TDI Adjusted Maze Score Word Analysis - TDI Ability Score (WAAS) OPM RC – Ability Score, Ability Range, Reporting Categories Maze – Adjusted Maze Score ORF (3rd – 5th) Adjusted Fluency Score Here is a listing of the scores you will receive from the PMRN on each of the tasks. We will briefly discuss each one and what they mean. 17 17

Questions to answer using the School Grade Summary Report
What is the distribution of the ‘yellow’ zone (16-84%)? Say that 38% of Seventh Graders fall in the yellow zone. Looking at a particular distribution, are more of them at the high end or the low end of the zone? Say that it is a relatively even distribution across the zone with between students falling in each section of the zone until the last two sections, 65-74% chance has 26 students and 75%-84% chance has 45 students. We want to see the skew fall to the right with more students closer to the high success zone of 85% chance or higher.

Assessment/Curriculum Decision Tree for Reading Improvement Using FAIR’s RC’s FSP and Maze and Word Analysis Percentile Ranks. This is a handout used for the 3-12 assessment. When we think about profiles or patterns - we can use a general decision tree to help us narrow the profiles down into 5 basic boxes. Other than box 1, student’s scores will fall into two boxes so at the bottom of the decision tree, we have provided instructional ideas for the different combinations. We will briefly review the handout as a whole and then we will break it down into the individual profiles looking at data from the class status report. The first question you ask yourself is what was the student’s FSP on the Broad Screen because remember the Broad Screen determines if we need to find out more information about this student or if the current levels of instruction will support the student enough to be successful on the end of the year FCAT. If the student has an 85% chance or better of scoring at Level 3 or higher on the FCAT, then that student falls in Box 1 because they will not be required to take the TDI – we don’t have to gather further information. If the student has an 84% chance or lower of scoring at or above level 3 then we will need to look at their percentile scores on the Maze and WA to help us determine what type of intervention/instruction will help improve that student’s likelihood of being successful. There are four boxes remaining and the student will fall into two of those boxes. It is this combined performance on the two TDI tasks that will provide us with direction as we plan instruction. Before we discuss the cut points for decision making we want to stress that this is ONE WAY to think about it and plan instruction. Please read the note at the bottom of the handout for the participants. Note: The 30th percentile cut point used in this document is given as a guide to estimate the level of instructional support necessary for student success. This cut point may need to be refined within each school and district depending on local circumstances such as available resources and student performance. The 30th percentile was used based on data gathered during the field trials of the new assessments. You need to look at the student’s percentile score on Maze and if it is above 30th then the student is in Box 2 and if below the 30th in Box 3. If the student is exactly at the 30th then the teacher will need to use their judgment and other data they have available to them. If the student’s performance on WA is above 30th percentile then the student is in Box 4 and below the 30th in Box 5. The boxes in the middle section provide you with a more detailed look at the individual tasks, such as in Box 3, if the student scores below the 30th on the Maze the teacher might want to probe a bit further to determine if it appears to be more of a fluency concern or a comprehension concern. (Read the suggestions from Box 3 to the participants) The same can be done with the WA, Box 5. (Read the suggestions from Box 5 to the participants). At the bottom of the page, the boxes provide instruction implications for the combined profiles. 19

Common Questions What score types should be used to measure growth? For RTI decision-making? Are scores dropping from AP1 to AP2 this year? Is FAIR norm-referenced? Criterion-referenced? Why does FAIR have adaptive tests (RC and WA) when we want grade-level information? Why doesn’t FAIR report at the benchmark level? How do we get instructional information out of FAIR? © 2011 Florida Department of Education

Measuring Growth on FAIR
In K-2, use PRS descriptively : E.g., establish % of students to be in green zone at each AP. In 3-12, use the reading comprehension ability score (RCAS) because it has the same metric across time (like using FCAT’s DSS rather than SS for growth). FAIR’s RCAS has a mean of 500; SD=100; range= In 3-12, do not use FSP to measure growth in reading comprehension ability because FSP includes prior FCAT but RCAS may change over time. For students without a matched FCAT score, FSP is based solely on RCAS. When a student’s FCAT score becomes available in the PMRN, the FSP may look unexpectedly high or low compared to the previous AP’s FSP. 21

Why we use Ability Scores

PM = progress monitoring; SS = standard score
Reading Comprehension Mazes Word Analysis AP Score PM score AP Score PM score AP Score PM score student score student score Percentile rank WAAS Percentile rank Adj. Maze SS FSP SS RCAS %ile & SS PM = progress monitoring; SS = standard score

3-12 Instructional Toolkit
Phonics screener and academic word list can be used to monitor progress. © 2009 Florida Department of Education

Progress Monitoring in K-2
G2 Broad Screen in G2 is same across each AP--# words read correctly in 45 sec. Thus, monitor progress in timed decoding. Equated OPM passages in G1-G5 allow for ORF progress monitoring (with 40th %ile at each grade provided at as criterion. Percent of G1 & G2 students who read target passage with comprehension. Percent of students achieving 80% mastery on TDI tasks. © 2011 Florida Department of Education

AP AP2 Scores Mean FSP and RCAS increased slightly in grades 3-8 and stayed the same in grades 9-10 at the state level. Standard deviations are fairly large, explaining why some are reporting drops. In grades 3-8 the majority of students tested were at FCAT level 3 or above. This reflects FAIR norms for FCAT 2.0. Slightly lower growth from 09-10SY to 10-11SY likely due to demands of FCAT 2.0. © 2011 Florida Department of Education

Norm- vs. Criterion-Referenced
The K-2 Broad Screen predicts to a nationally normed test; percentiles for Vocabulary and G2 Spelling are based on FL grade-level norms. The K-2 diagnostic inventory is criterion-referenced (80% of skills mastered). The 3-12 Broad Screen predicts to a criterion—passing FCAT. (i.e., the FSP) Percentiles for the 3-12 Broad Screen, Maze, and WA are based on FL grade-level norms. © 2011 Florida Department of Education

Value of Computer-Adaptive Tests
Provides more reliable & quicker assessment of student ability than a traditional test, because it creates a unique test tailored to the individual student’s ability. Provide more reliable assessments particularly for students at the extremes of ability (extremely low ability or extremely high ability). Grade-level percentiles are currently provided; Grade Equivalent scores will be provided next year. © 2011 Florida Department of Education

Benchmark Conundrum Benchmark tests rarely have enough items to be reliable at the benchmark level. Besides, teaching to benchmarks (e.g., “the student will use context clues to determine meanings of unfamiliar words”) results in fragmented skills. Teach to the standard(s) (e.g., “The student uses multiple strategies to develop grade appropriate vocabulary). Assess at aggregate levels (e.g., Reporting Categories), if CFA show categories are valid. © 2011 Florida Department of Education

FCAT 2.0 Reporting Categories
Reporting Category 1: Vocabulary Reporting Category 2: Reading Application Reporting Category 3: Literary Analysis Fiction/Nonfiction Reporting Category 4: Informational Text/ Research Process

FCAT 2.0: Benchmarks x Grade
Category 1 Category 2 Category 3 Category 4 Total Grade 3 4 6 1 14 5 2 15 7 8 9/10 To achieve reliable estimates of the benchmarks in each grade, classical test theory would suggest 15 items, on average, per benchmark. That’s items per grade. © 2011 Florida Department of Education

Possible Benchmark Solutions
Stop-gap: start each students with grade-level passage. Provide % correct on Reporting Categories. Then continue to current adaptive system to obtain reliable, valid FSP and RCAS. For the future: Align FAIR to the Common Core. Develop grade-level CAT that is item adaptive. Challenges: Dimensionality; multi-dimensional IRT; testlet effects. © 2011 Florida Department of Education

Scarborough (2002)

Dimensions of Word Knowledge
Knowledge of word’s spoken form (pronunciation) Written form (spelling) Grammatical behavior (syntactic/morphological features) Co-locational behavior (occurs with other words) Frequency (orally and in print) Stylistic register (e.g., academic language; informal) Conceptual meaning (antonyms, synonyms) Association with other words (inter-relatedness) Nation (1990) in Nagy & Scott (2000)

Vocabulary Knowledge Task
Word Meanings FCRR has developed a 3- min. adaptive vocabulary sentence task that strongly predicts RC. Students pick 1 of 3 morphologically related words to best complete a sentence. Teachers can use percentiles, standard scores, & ability scores to drive vocabulary instruction Text

New Vocabulary Knowledge Task

In Support of FAIR Other Resources: PMRN Help Desk and Just Read, Florida! staff available 8:00-5:00 Monday through Friday FAIR technical tips, Users’ Guides, and frequently asked questions are available at the FCRR website: and the Just Read, Florida! website: LEaRN (Literacy Essentials and Reading Network) FAIR resources for teachers, coaches, and principals which include training videos and clips of master trainers administering FAIR to students. © 2011 Florida Department of Education

For more detailed information regarding the PMRN:
The PMRN User’s Guides located at: There is a section that provides an Overview of Reports and a section that provides an Explanation of Score Types Complete Power Points can be found on the PMRN website: Topics include: K-2 Data Entry K-2 Electronic Scoring Tool 3-12 Web-based Assessment Module (WAM) School Level Users 1, 2, 3 Phone: or Toll free: (866)

In summary, remember… FAIR was designed to inform instruction.
FAIR data are just one part of the puzzle. Teachers bring information from daily instruction that provides additional, important information. As FCRR gains more experience with the data and collects data statewide, we will be able to provide guidelines on what are ‘typical’ scores at each grade level or what is a ‘good’ percentage to have in the high success zone (green zone). We will make comparisons to the State Mean scores and percentages at the end of each AP New 3-12 Tech Manual available at end of summer, 2011.

Thank you for your time and attention!
Any Questions? 41 41

Dr. Barbara Foorman Florida Center for Reading Research (FCRR)

Similar presentations

Presentation on theme: "Dr. Barbara Foorman Florida Center for Reading Research (FCRR)"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Dr. Barbara Foorman Florida Center for Reading Research (FCRR)

Similar presentations

Presentation on theme: "Dr. Barbara Foorman Florida Center for Reading Research (FCRR)"— Presentation transcript:

Similar presentations

About project

Feedback