1. Florida State University
The relevant effectiveness of two approaches to early literacy intervention in grades K-2
Barbara Foorman Sarah Herrera Jennifer Dombek Chris Schatschneider Yaacov Petscher
Florida State University

2. Two Practically Important Approaches
Early Literacy Interventions Study
The selection of instructional materials is an important consideration when implementing early literacy interventions (Dombek, Foorman, Garcia, & Smith, 2016).
Two Practically Important Approaches
Stand-alone
Purchased in addition to core reading program
Not specifically aligned with classroom instruction & materials
Can be selected from the What Works Clearinghouse with strong levels of evidence in the domains of reading
Embedded
Purchased with core reading program
Aligned with classroom instruction & materials
Uses “research based” approaches, but rarely evaluated empirically
Lack of no intervention or business-as-usual control
No intervention is not an option in FL schools
Business-as-usual means different things in different sites and within sites is constantly changing
The embedded approach could be described as the prototypical control condition for this study because it consisted of the tier 2 intervention materials contained within the adopted core reading program. However, lack of resources to create manuals detailing the scope & sequence and implementation procedures or preference for other tier 2 materials made it unlikely that participating schools had planned to implement the embedded intervention as it was used in the current study.

3. Early Literacy Interventions Study: Research Questions
The study addressed the following research questions separately for kindergarten, grade 1, and grade 2:
What is the impact of a stand-alone intervention relative to an embedded early literacy intervention on reading and language outcomes?
Is the impact different between the two cohorts of schools or dependent on baseline test scores?
Are there differences in outcomes between interventions for English learner and non-English learner students? Are there differences within intervention between English learner and non-English learner students?

4. Early Literacy Interventions Study
Study Design
Daily (October-May) for 45 minutes per session Grade K & 1: 4 students per group Grade 2: 5 students per group
Stand-alone Intervention
Embedded Intervention
Sound Partners
Bridge of Vocabulary
Language in Motion
Everyday (25-30 min.)
3x per Week (15 min.)
2x per Week (15 min.)
Journeys Tier 2 Strategic Intervention
Journeys Tier 2 Curious About Words
Everyday (25-30 min.)
Everyday (15 mins.)
Both intervention approaches were taught daily from mid-October to the end of May for 45 minutes and consisted of a 25–30 minute reading component and a 15 minute oral language component.

5. Stand-alone Intervention
Sound Partners
An empirically validated and designed for students struggling to master foundational reading skills.
Explicit and systematic instruction in:
phonemic awareness
phonics
Spelling
Fluent blending
Bridge of Vocabulary
An evidence-based intervention that systematically teaches curriculum standards.
A systematic and intensive approach to build:
listening vocabulary
speaking vocabulary
reading vocabulary
writing vocabulary* (not included in this study)
Language in Motion Focuses on improving syntax and language skills, in turn increasing listening comprehension skills. Developed by Dr. Beth Phillips as part of Dr. Christopher Lonigan’s IES-funded Reading for Understanding grant at FSU/FCRR. Demonstrated effectiveness in pilot and efficacy studies in FL.
REL Southeast staff reviewed the levels of evidence at the WWC for reading interventions that had been studied with at-risk students in grades K–2 and implemented in small groups.
The program that met these criteria and had the strongest levels of evidence in the relevant domains was Sound Partners (Vadasy & Sanders, 2012; Vadasy, Sanders, & Abbott, 2008; Vadasy, Sanders, & Peyton, 2006).
No academic language intervention programs for at-risk students in grades K–2 are listed on the WWC.
Therefore, a vocabulary program with good clinical evidence called Bridge of Vocabulary (Montgomery, 2007) and an inferential language program with evidence of efficacy called Language in Motion (Phillips, 2014) were added to Sound Partners to create the stand-alone approach used here.

6. Stand-alone InterventionM T W R F
25-30 minutes
15 minutes

7. Embedded Intervention
Journeys Tier II Strategic Intervention
Aligned with Tier I classroom instruction & provides additional support to build skills in:
phonological awareness
phonics
fluency
comprehension
vocabulary
Research-based strategic intervention strategies; provides extra support for students; and assists students in building independence in reading through accessible text.
Curious About Words
Intensive oral vocabulary instruction for children with limited vocabularies in Grades K-3. Designed to supplement the core vocabulary instruction in HMH Journeys.
Daily lessons include the following to help develop listening and speaking vocabularies:
read alouds
graphic organizers
teacher-led discussion
partner activities
Houghton Mifflin Harcourt (HMH) Journeys was the most widely adopted core reading program in Florida at the start of this study
Therefore, the embedded intervention in this study consisted of the tier 2 Strategic Intervention and supplementary vocabulary piece called Curious about Words that are part of HMH Journeys.
Strategic Intervention is similar to Sound Partners in the alphabetic skills taught but it also includes vocabulary and comprehension skills.
Another difference is that Sound Partners requires that interventionists repeat lessons when skills have not been mastered, whereas HMH Journeys has no specific provision for remediation.
Curious About Words is based on Beck, McKeown, and Kucan’s (2013) strategies for teaching vocabulary words embedded in challenging text read aloud by a teacher.

8. Embedded InterventionW R F
25-30 minutes
15 minutes

9. Columbia County Year 1 (2 Schools) Taylor County Year 1 (1 School)
Putnam County
Year 2 (3 Schools)
Pinellas County
Year 1 (8 Schools)
Pinellas County
Year 2 (9 Schools)
Broward County
Year 1 (16 Schools)
This slide highlights that our study sample reflects the demographics of the state of Florida.
Across both years, 55 schools were randomly assigned within geographic region to either the embedded or stand-alone intervention approach. Hereafter we will refer to years 1 and 2 as cohorts 1 and 2.
Broward County
Year 2 (16 Schools)

10. Florida Center for Reading Research Reading Assessment (FRA) Tasks
Student Eligibility
Florida Center for Reading Research Reading Assessment (FRA) Tasks
Grade K:
Below 30th percentile on FRA Phonological Awareness and/or Vocabulary Pairs
Grade 1 & 2:
Below 30th percentile on FRA Word Reading and/or Vocabulary Pairs
Intervention groups were formed based on ability scores from these FRA screening tasks
Grade K
Grade
1 & 2
Tasks
Pre
Post
Phonological Awareness X
Letter
Sounds
Vocabulary Pairs
Following Directions
Sentence Comprehension
Word
Reading
Spelling (G2 only)
Red & blue boxes highlight subtests used for eligibility by grade.
FLEXIBLE GROUPING: During the first 10 weeks between 6 and 15 percent of students across grades K–2 moved to another group because their performance on skill mastery tests in the intervention indicated a better fit with another group.

11. Study Sample Cohort 1 Cohort 2 Total Schools 27 28 55 Students 1,598
1,870
3,468
Groups
370
424
794 N
ELL
FRL
Kindergarten
1061 38 86
Grade 1
1098 34 82
Grade 2
1309 33 83
This table shows the total number of schools, eligible students, and small groups that were served in each cohort of the study across grades and interventions.
This table shows the total number of eligible students served in each grade. You’ll notice that more than 30% of the sample in each grade are English learner students and more than 80% are eligible for free or reduced-price lunch.

12. Interventionists Training:
Total N
143
% District Employees 26
% Female 93
% Bachelors Degree 63
% Masters Degree 17
% Certified Teacher 35
Training:
All interventions received a two-day training in late September
All interventionists received ongoing support as needed
This table reports information on the interventionists. Across both cohorts, 143 interventionists served small groups of students. We had roughly 3-4 interventionists per school & 4-6 small groups per school. Approximately, 26% of the interventionists were district employees and slightly more than 1/3 were certified teachers.
All interventionists received a two-day training in late September and
All received ongoing support as needed throughout the school year if the need was indicated by project leadership observations and/or requests for support from interventionists.

13. Lesson Coverage Stand-alone Embedded Component N Mean (SD)
Stand-alone
Embedded
Component N
Mean (SD)
Kindergarten
Reading
131
80 (16)
126
88 (8)
Oral Language
79 (10)
87 (8)
Grade 1
129
55 (10)
137
78 (9)
Grade 2
133
62 (11)
87 (12)
77 (11)
86 (12)
N reflects the number of groups per grade and condition
Stand-alone:
Due to remediation in sound partners, it is not surprising that, on average, coverage of the reading component in the stand-alone intervention was 15–23 percentage points lower than the oral language coverage in grades 1 and 2.
It is important to note that when an intervention group was required to remediate instruction in the reading component, it was because an average of 45–59 percent of the intervention group had not demonstrated mastery on the skill assessment, confirming that the groups were relatively well matched on ability.
Remediation targeted lessons that included concepts students had not mastered on the skill assessments.
On average, intervention groups across grades 1 and 2 required remediation on 8–11 out of a possible 30 skill assessments.
Coverage:
Lesson progression for stand-alone reading component was dependent on students’ mastery of content.
Lesson progression for embedded reading component & all oral language components moved sequentially through the curriculum.

14. Fidelity of Implementation
Fidelity is the percent of the lesson in which instruction followed the lesson sequence and script within each of the skills taught.
All groups were observed in both fall and spring
Overall fidelity of implementation was high (average of percent across groups for all components)

15. Assessments Pre & Post:
Grade K
Grade 1 & 2
Tasks
Pre
Post
FRA Phonological Awareness X
FRA Letter Sounds
FRA Vocabulary Pairs
FRA Following Directions
FRA Sentence Comprehension
FRA Word Reading
FRA Spelling (G2 only)
SESAT Word Reading
SESAT Sentence Reading
SAT-10 Reading Comprehension
Pre & Post:
Florida Center for Reading Research Reading Assessment (FRA)
Post Only:
Kindergarten: SESAT Word reading & Sentence Reading
Grade 1 & 2: SAT10 Reading Comprehension

16. Percentile Ranks: Kindergarten Outcomes
Outcome measure
Stand-alone
Embedded
Pre
Post
Diff.
Reading
FRA phonological awareness 1 21 20 26 25
SESAT word reading na
Language
FRA following directions 7 27 5
FRA vocabulary pairs 34 9 24 33
FRA sentence comprehension 10 35 32 23
SESAT sentence reading 22
The following 3 slides report average pre- and post-test percentile ranks for each outcome and their difference by grade and condition.
For students in kindergarten, there was a 20+ percentile point difference between average pre- and post-test for FRA Phonological Awareness, following directions & sentence comprehension across intervention conditions.
Note. Percentile ranks are based on winter norms.

17. Percentile Ranks: Grade 1 Outcomes
Stand-alone
Embedded
Outcome measure
Pre
Post
Diff.
Reading
FRA word reading 1 23 22 26 25
Language
FRA following directions 10 19 9 11 21
FRA vocabulary pairs 12 18 6
FRA sentence comprehension 29 64 35 27 66 39
SAT 10 reading comprehension na 13
For students in Grade 1, there was a 20+ percentile point difference between average pre- and post-test for FRA word reading.
The largest average percentile point difference between pre- and post-test for any FRA task was in grade 1 for Sentence Comprehension.
This reflects an average difference from 35 to 39 percentile points across interventions.
It is important to note that Sentence Comprehension is a kindergarten normed task therefore the Grade 1 and 2 percentile ranks are on the kindergarten scale.
Note. Percentile ranks are based on winter norms. Grade 1 and 2 Sentence Comprehension percentile ranks are on the kindergarten scale.

18. Percentile Ranks: Grade 2 Outcomes
Stand-alone
Embedded
Outcome measure
Pre
Post
Diff.
Reading
FRA word reading 5 24 19 9 26 17
FRA spelling 3 22 4 13
Language
FRA following directions 15 30
FRA vocabulary pairs 12 10 18 8
FRA Sentence Comprehension 58 87 29 57 82 25
SAT 10 reading comprehension na 14
The average difference between pre- and post-test for FRA word reading, spelling, and following directions tasks ranged from 13 to 19 percentile points.
Again, Sentence Comprehension is a kindergarten normed task so Grade 1 and 2 percentile ranks are on the kindergarten scale.
Sentence comprehension pretest was only available for cohort 2 students.
Note. Percentile ranks are based on winter norms. Grade 1 and 2 Sentence Comprehension percentile ranks are on the kindergarten scale.

19. Analyses 3-level Multilevel model by grade Student level:
FRA pretests, ELL (only ELL models)
Small group level:
FRA pretests
School level:
FRA pretests, region, cohort, treatment
Three sets of treatment effects were estimated using multilevel models by grade with student – small grp – schools.
The first set of analyses focused on the primary impact of the stand-alone intervention on each outcome. These models included all pretests and region as covariates

20. Analyses 3-level Multilevel model by grade Student level:
FRA pretests, ELL (only ELL models)
Small group level:
FRA pretests
School level:
FRA pretests, region, cohort, treatment
Cross-level Interactions
Removal of non-significant predictors occurred systematically.
The second set of analyses investigated cohort and baseline differences. Here, a top-down approach was used in which the full model included all pretest covariates, region, cohort, treatment, and cross-level interactions including pre*cohort, pre*treatment, and pre*cohort*treatment.
The removal of non-significant predictors from the full model followed a systematic process, such that 3-way interactions were removed first, then 2-way interactions, and finally cohort.
Pretest covariates at all levels and region were retained regardless of significance to increase the precision of the treatment effect.
If the final model included a significant treatment interaction, the highest level interaction (that is, the 3-way or 2-way interaction) involving the treatment variable was explored further.

21. Analyses 3-level Multilevel model by grade Student level:
FRA pretests, ELL (only ELL models)
Small group level:
FRA pretests
School level:
FRA pretests, region, cohort, treatment
Cross-level Interaction
Final model was used as base for ELL models.
The final model for each outcome by grade was then used as the base model for the third set of analyses investigating treatment differences for English and non-English learner students.
At a minimum, two variables were added to the base model: Student level English learner status and the cross level English learner status by Treatment interaction.

22. Analyses 3-level Multilevel model by grade Student level:
FRA pretests, ELL (only ELL models)
Small group level:
FRA pretests
School level:
FRA pretests, region, cohort, treatment
Two domains of outcomes:
Reading: Phonological Awareness, Word Reading (FRA and SESAT), and Spelling.
Language: Following Directions, Vocabulary Pairs, Sentence Comprehension, Sentence Reading, and Reading Comprehension
Outcomes were divided into two broad domains: Reading and Language.
Reading outcomes include PA, WR (FRA and SESAT), and SP.
Language outcomes include FD, VP, SC, SR, and RC
The following results slides will highlight outcomes demonstrating statistically significant effects after applying a Benjamini-Hochberg correction for multiple comparisons as well as outcomes demonstrating Hedges’ g effect size differences of .25 or greater, which will be described as substantively important differences per What Works Clearinghouse guidance.

23. Primary Impact Results Kindergarten – Grade 2
Kindergarten and grade 1:
No effects on reading or language
Grade 2 FRA Spelling
Stand-alone significantly higher than embedded
Grade 2:
No effects on other reading or language outcomes.
g = 0.18*
The first set of results is focused on the average difference between the two interventions across all students in each grade.
No statistically significant or substantively important differences between the interventions on reading or language outcomes in kindergarten or grade 1 was observed.
However, a statistically significant difference in favor of the stand-alone intervention on FRA Spelling in grade 2 was found.
Orient audience
The figures report adjusted means on the y-axis by outcome for each condition (red is stand-alone and grey is embedded). Hedge’s g effect sizes are reported for each outcome. The x-axis denotes the outcome and specifies information about the sample. In some cases you will notice that the sample deviates from the full sample.
No other significant or substantively important differences between interventions was found in grade 2 on any other reading or language outcomes.

24. Impacts by cohort and pretest Kindergarten Reading Outcomes
This slide and the 3 subsequent slides report findings from the set of models that investigated impacts by cohort and pretest.
Again, the adjusted means for the outcome is on the y-axis and the x-axis denotes the outcome and specifies information about the sample. The red bar denotes the stand-alone intervention and the grey bar denotes the embedded intervention.
In Kindergarten, a substantively important difference favoring the stand-alone intervention was found for those in cohort 1 with a FRA SC pretest score one SD above the mean.
A substantively important difference favoring the stand-alone intervention was also found for those in cohort 2 with a FRA SC pretest score one SD below the mean.
No evidence of treatment effects were observed on FRA PA or WR.

25. Impacts by cohort and pretest Grade 1 Reading Outcomes
In grade 1, a substantively important effect for cohort 1, in favor of the embedded intervention was found on FRA word reading. No effects were found for cohort 2.

26. Impacts by cohort and pretest Grade 2 Reading Outcomes
In grade 2, a statistically significant difference in favor of the stand-alone intervention was found for those with SP pretest scores 1 SD below the mean.
No effects were founds on FRA WR.

27. Impacts by cohort and pretest Grade 2 Language Outcomes
In grade 2, the stand-alone intervention resulted in significantly improved FRA Sentence Comprehension scores for students in cohort 1 with FRA Vocabulary Pairs pretest scores 1 SD below the mean.
There was no evidence of effects on any other language measures in grade 2.
The next 2 slides report findings from the ELL models.
No effects were observed on any outcomes in grades 1 and 2 or language outcomes in kindergarten, so I’ll be presenting ELL effects on reading outcomes in kindergarten only.

28. Impacts for Kindergarten English Learner and Non-English Learner Students
For English learner students:
Substantively higher FRA Phonological Awareness in Embedded
For non–English learner students:
Substantively higher SESAT Word Reading in Stand-alone
g = -0.32
g = 0.31
English learner students in the embedded intervention had substantively higher FRA Phonological Awareness scores than English learner students in the stand-alone intervention.
However, non-English learner students in the stand-alone intervention had substantively higher SESAT Word Reading scores than non-English learner students in the embedded intervention.

29. Impacts for Kindergarten English Learner and Non-English Learner Students
In Embedded:
English learner students had substantively higher SESAT Word Reading than non–English learner students
g = 0.23
g = 0.27
g = 0.18
This graph is slightly different than the others. This graph is comparing English learner students to non-English learner students within the embedded interventions. Here, the red bar denotes English learner students and the grey bar denotes non-English learner students.
English learner students in the embedded intervention had substantively higher SESAT Word Reading scores than non–English learner students in the same intervention.

30. Summary of Results Relative impacts of the two interventions
Stand-alone > Embedded on Grade 2 Spelling outcome*
Differences in outcomes between interventions by pretest and cohort
Stand-alone > Embedded:
Kindergarten Cohort 1 high Sentence Comp on SESAT Word Reading outcome
Poor Grade 2 Spellers on Spelling outcome*
Grade 2 Cohort 1 poor Vocab on Sentence Comprehension outcome*
Embedded > Stand-alone:
Grade 1 Cohort 1 on FRA Word Reading
Differences in outcomes based on English learner status in kindergarten
Embedded > Stand-alone for English learners on FRA Phonological Awareness
Stand-alone > Embedded for non-English learners on SESAT Word Reading
English learners > non-English learners in Embedded on SESAT Word Reading
The two interventions had similar impacts on reading and language outcomes except for spelling in grade 2 where students in the stand-alone intervention had significantly better spelling outcomes than those in the embedded intervention.
There were several differences in outcomes between interventions by pretest and cohort.
The Standalone intervention demonstrated substantively better SESAT WR scores for cohort 1 kindergarten students with high SC pretest scores, significantly better Spelling scores for G2 students with low spelling pretest scores, and significantly better SC scores for cohort 2 G2 students with low Vocab pretest scores compared to students in the embedded intervention.
Conversely, the embedded intervention demonstrated substantively better FRA WR scores for cohort 1 G1 students compared to students in the standalone intervention.
There were several differences in outcomes based on English learner status in kindergarten.
Specifically, English learner students in the embedded intervention scored substantively higher on FRA PA than English learner students in the standalone intervention.
Non-English learner students in the standalone intervention scored substantively higher on SESAT WR than Non-English learner students in the embedded intervention.
Finally, English learner students had substantively better SESAT WR scores than Non-English learner students in the embedded intervention

31. Conclusions
Well-implemented, daily intensive intervention did not ensure that students who started below the 10th percentile performed above the 25th percentile on reading outcomes.
Solution:
Intervene earlier
Reduce group size
Scaffold EL students with comprehension activities
Streamline intervention components to facilitate implementation
When implementing a tier 2 intervention, consider:
Alignment to tier 1 instruction
Whether an adequate scope and sequence and implementation manual exists
How PD will be provided
How will fidelity of implementation be monitored
Although there were significant effects in grades K-2, well-implemented intervention daily for 45 minutes for 27 weeks did not ensure that students who started below the 10th percentile, on average, performed above the 25th percentile on reading outcomes.
Solution:
Intervene with students in K or earlier; provide greater intensity by reducing group size (which would also help with the remediation component in Sound Partners).
In kindergarten, provide low-performing EL students with comprehension activities so as to build off their PA sensitivity in order to connect to the sound-spelling patterns necessary for learning to read.
Streamline the reading and language components of intervention to be the same on a daily basis, rather than alternating days for the language pieces (as was done in stand-alone).
Embedded intervention has benefits of cost and alignment, but cannot be implemented straight out of the shrink-wrap package. Practitioners must (a) develop a manual of scope & sequence and implementation procedures, (b) provide training; and (c) monitor for fidelity.

32. Questions?