Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012

Overview Being a good consumer of Head Start research Early cohort research The National Head Start Impact Study Early Head Start Research What’s new in Head Start Research Remaining research questions

“...finally there is indisputable evidence about the program's effectiveness, provided by the Department of Health and Human Services: Head Start simply does not work.” -Joe Klein, Time Magazine, July 2011

“Head Start improves the odds and the options for at-risk kids for a lifetime. Kids that have been through Head Start and Early Head Start are healthier, more academically accomplished, more likely to be employed, commit fewer crimes, and contribute more to society. Simply put: Head Start works. It’s been proven.” -Letter to Congress, signed by over 270 top early childhood researchers March, 2011

Being a good consumer of Head Start research Lots of Head Start research Lots of rhetoric –Which research is good research? –What do the results tell us about Head Start’s effectiveness? –What are the caveats? Bottom line… –There is a lot to be excited about!

Challenges in Head Start Research Lots of things affect educational, health, family outcomes –Hard to isolate the “true” impact of Head Start programming Finding the appropriate control group –Head Start targeted at particularly disadvantaged children and families

Questions to ask 1)Is the evaluation design strong enough to produce trustworthy evidence? 2)What program services were actually received by participating children and families in the comparison group? 3)How big an impact did the program have? 4)Do the program’s benefits exceed its costs? 5)How similar are the programs, children, and families in the study to those in my community?

Research 101: Research Design Gold Standard=Experiment –Randomized controlled trial Quasi-experimental approaches –Does the control group really look like the Head Start population? –What services do children and families in the control group get? External validity –If I reproduced the study in my community, would I get the same results?

Research 101: Effect Size What is an effect size? –Measure of “magnitude” of impact The bigger, the better! More meaningful than “statistically significant” –Fraction of the difference between the treatment and control group over variation Allows comparison across different measures Need this to do a cost/benefit analysis

Early cohort research: Some history The Westinghouse study (1968) –Quasi-experimental approach –Focus on cognitive outcomes only –Almost defunds Head Start –Consortium of researchers rallies against the study We need better research on more outcomes! We waited a long time...

Janet Currie and colleagues: Sibling fixed effect studies (1990s) Use nationally representative data sets Include families in which one sibling went to Head Start and one didn’t Teases out all the “fixed” family- related factors that influence outcomes Still questions about why one sibling attended and not the other

Currie et al. findings Large gains in vocabulary test scores –Equal to about 1/3 of achievement gap –Effects persist into adulthood for whites, but fade quickly for African-Americans Reduced probability of retention (47%) –Again, only for whites More likely to be immunized (8-11%) –Both whites and African-Americans Cost-benefit implications –Increased test scores would increase expected future wage by 4% –Reduction in retention would lead to 5% decline in probability of high school drop-out –Preventative health savings –Definitely passes test for whites, question for African-Americans

More Currie et al. findings Educational attainment (whites only) –22 percentage points more likely to complete high school –19 percentage points more likely to attend some college Less likely to be arrested or charged for crime –12 percentage points, African-Americans only Positive effects for Hispanic children –Closes 1/4 gap in test scores –Closes 2/3 gap in probability of grade repetition Maybe effects less, fade for African-American children because they attend lower quality schools later

Ludwig & Miller (2007) 50 to 100% increase in Head Start funding in 1960s and 70s associated with –Increase of schooling attainment of about 1/2 year –15% increase in likelihood of attending some college –Found for both African-Americans and whites –Decrease in infant mortality Benefit/cost ratio as large as 7 to 1 –Decline in mortality $120 per 4-year-old –Extra half year of schooling increases earnings 2% or $15,000 –Even higher if consider increased schooling reduces crime

Deming, 2009 Uses same method as Currie et al., but more recent cohort Effects on math and reading that fade –0.15 at age 5-6 –0.13 at age 7-10 –0.06 at age –Larger for African-Americans at first, but fade more quickly Larger positive effects on long-term outcomes –18.6 percentage points more likely to complete high school (African-Americans) –“Index” of long-term outcomes, 0.23 Mechanism = less retention, special education

Hard to generalize these findings to today More working parents More early childhood programming options Decrease in achievement gaps Increase in Head Start quality

The National Head Start Impact Study (NHSIS) Randomized controlled trial required by Congress in 1990s Nationally representative sample –84 Head Start programs –Approximately 4,600 children –Begin attending in 2002 Follow kids through end of 1 st grade

First year findings: 4-year-olds 7 positive cognitive effect sizes –Effect sizes = 0.16 to 0.31 More likely to visit the dentist –(73.2% vs. 56.9%) Parents more likely to read to children –(3x/wk vs. 2.8x/wk) No significant findings for more advanced reading skills, math, or behavior problems

First year findings: 3-year-olds Same positive impacts as 4-year-olds, but bigger, plus... –Math –Fewer problem behaviors –Parents spanked children less –Overall health Still no effects on more advanced reading skills

Lots of Caveats “Intent to treat” analysis –17% of 3-year-olds and 14% of 4-year-olds never got Head Start services Crossovers –18% of 3-year-olds and 14% of 4-year-olds in control group attended Head Start 60% of control kids in other programs Evidence of circumventing randomization –Center directors enroll those with highest needs

Ludwig & Phillips re-analysis “Treatment on the treated” (TOT) –Adjust for crossovers and no-shows –Larger effect sizes (0.14 to 0.50) Pool together 3- and 4-year-olds –Larger sample, more “power” –Statistically significant impacts for nearly all main cognitive outcomes

First grade follow-up Significant effects from first year mostly disappear –For 4-year-olds, only impact on receptive vocabulary remains (0.09) –For 3-year-olds, only impact on oral comprehension remains (0.08)

Why we shouldn’t compare these results to pre-k study results Pre-k studies use different research design that is not as strong as NHSIS Fewer kids in pre-k control groups attend other programs Different populations Pre-k only looks at academic outcomes No long-term follow-up of pre-k (therefore, no info about “fade out”)

So, what does this study really tell us? Impacts represent short-term effects of offer of one year of Head Start Measures impact of Head Start versus everything else available Initial impacts found are about the same size as those found for children attending Head Start in the 1960s-1980s For those cohorts, Head Start seems to have produced long-term benefits large enough to outweigh program costs, despite fade-out of initial test score impacts (nothing like that measured here) But test score impacts do seem to fade more quickly in new cohort—not sure why or whether we should worry about this

Does Head Start pass the benefit/cost test today? Ludwig & Phillips (2008) –Cost = $9,000 per child –Head Start might pass a benefit-cost test if short-term effects on achievement test scores were as small as 0.1 to 0.2

Early Head Start research Randomized controlled trial Follow through 5 th grade Positive impacts at age 3 –Cognitive and language development –Socio-emotional development –Home environment and parenting behavior –Parents’ progress toward self-sufficiency –Positive impacts specifically for fathers “Sleeper” effects Differences across different service models

Early Head Start Home Visiting Evidence

What’s new in Head Start research? Focus on “quality” Curriculum enhancements Professional development Early Head Start and home visiting Head Start Family and Child Experiences Survey (FACES) –New! “Baby FACES”

Meta-analysis (Shager et al., 2012) Study of 33 Head Start evaluations Overall, positive effect of Head Start on cognitive and achievement outcomes (effect size = 0.27) What happens in the control group matters a lot! –Effect sizes for studies that have an “active” control group are 0.33 smaller –Type of skill, measures used, and where published also matter

Conclusions Some high quality Head Start research exists, but you need to be an alert consumer Head Start and Early Head Start have had positive impacts on children and families across a variety of domains Head Start appears to pass the benefit/cost test We still have a lot to learn about how and why early childhood programs improve long-term outcomes

Remaining questions Instead of asking whether Head Start is effective at all, we must ask how effective Head Start is compared to the range of other options available. What happens to kids after they attend Head Start? –Do effects really “fade out,” or do other kids “catch up”? Which behavioral and socio-emotional skills are important, and how can Head Start foster those? Other benefits? –Health, parenting, family self-sufficiency

What are your questions?