Training is Not Enough: Intervention and Evaluation Supplements Christina J. Groark and Robert B. McCall University of Pittsburgh Office of Child Development

Overview—Intervention Training alone is a minimally effective intervention Need: −Hands-on technical assistance in situ −Supportive work environment −Continuous monitoring, positive supervision

Assessing trainee’s satisfaction and learning is a minimum evaluation Need: −Changes in trainee’s on-the-job behavior −Changes in the behavior/outcomes of clients of the trainees −Analyses to show trainees behavior change mediated the intervention's effect on clients’ behavior/outcomes Overview—Evaluation

Intervention Training alone is minimally effective Why? Trainees often do not change their behavior Need: On-the-ground technical assistance Supportive work environment Monitoring and positive supervision

Intervention On-the-ground technical assistance Observe (videotape) client on job Constructive, positive feedback Modeling with client on job Daily review

Intervention

Work environment Supports behaviors taught in training Hand washing—need a sink, waterless soap handy Early care and education—need small group size, low children:caregiver ratio, a few consistent caregivers

Chaotic Child Environment

Orderly Child Environment 9

Monitoring and positive supervision Sets expectations, standards, reminds, rewards Continuing system, supervisor, staff meetings Hand washing Early care and education—teachable moments, responsiveness to child- directed initiatives

Illustrative Example I Latin American Orphanage Intervention (birth-6 years) Training—sensitive, responsive interactions ◦6 one-day training sessions On-the-ground technical assistance ◦12 days more than planned because caregivers were not implementing

BUT – Limited environmental change ◦4 primary caregivers per ward ◦8-10 children per caregiver No continuing supervision

Mean Total ITERS/ECERS Intervention Pre- Post- Results—Caregiving behavior/environment (ITERS/ECERS) Younger Older 6 Typical USA Early Care

Battelle Intervention Pre- Post- Results—Children’s development (Battelle) Younger Older Typical Parent-Reared

Conclusion Needed hands-on technical assistance to promote implementation Limited supportive work environment No supervisory system Some improvement in caregivers, children, but NOT MUCH

Illustrative Example II St. Petersburg (Russia) Orphanage Intervention More intensive, comprehensive All major components

Minimum warm, sensitive, contingently- responsive caregiver-child interactions Perfunctory, business-like; little talking Caregiver directed Conformity 10/10/ St. Petersburg-USA Orphanage Research Team, 2005, 2008

12-14 children/ward 9-12 caregivers/week Homogeneous age, disability groups Periodic graduations caregivers before age 19 months 10/10/ St. Petersburg-USA Orphanage Research Team, 2005, 2008

Intervention Training ◦ Train Trainers ◦ 12 sessions, 2-3 hrs. ◦ Warm, sensitive, responsive ◦ “Love these kids…” Hands-on technical assistance

Intervention—Supportive Work Environment Reduced group size to 6-7 Assigned primary, secondary caregivers—6 vs 9 cgrs. Changed work schedules Integrated groups by age, disabilities Eliminated periodic graduations to new groups Family hour

Intervention—Monitoring and Positive Supervision Training for supervisors Trainer observed, coached both caregivers and supervisors Supervisors met periodically to problem solve, case reviews, mutual support

Three orphanages (Baby Homes), birth – 4 years T+SC/M&S—Training plus Structural Changes, Monitoring, Supervision TO—Training Only NoI—No Intervention Evaluation

Evaluation—Trainee Learning

Longitudinal Sample Evaluation—Improved Caregiving T+SC

Evaluation – Children’s Development First 4-9 mos 9+ mos Time in Intervention Developmental Quotient T+SC TO NoI 100 Typical Parent Reared

Evaluation--Mediation Analysis Intervention improved caregiving Intervention improved children’s development But was children’s developmental improvement associated with improved caregiving?

Evaluation—Mediation Analysis

Evaluation--Mediation Analysis Independent Variable T+SC vs NoI Intervention Outcome Battelle Total Effect Ĉ = 14.14(2.07)*** Mediator HOME Independent Variable T+SC vs NoI Intervention Outcome Battelle Effect of Mediator on Outcome Controlling for Indep. Variable Direct effect Controlling for Mediator Ĉ = 8.00(2.87)*** Mediator accounts for 43% of Total Effect Effect of Indep. Var. on Mediator  = 3.41(.22)***  = 1.80 (.59)**

Intervention ◦ Training alone is often minimally effective ◦Has greater effect if accompanied by  On-the-ground technical assistance  Supportive work environment  Monitoring and supervision Conclusion

Evaluation is improved if ◦Some training vs. No Training comparison ◦Measures of –Learning –Work-place behavior change –Outcome behavior of clients ◦Mediation analysis Conclusion

Evaluation Mediation Analysis: MacKinnon, D. P., & Dwyer, J. H. (1993). Estimating mediated effects in prevention studies. Evaluation Review, 17(2),