Maryland Early Childhood Mental Health Consultation (ECMHC) Evaluation Webinar Presentation by Deborah F. Perry Ph.D. Georgetown University October 27, 2010

Overview of Presentation Overview of statewide ECMHC evaluation in Maryland Maryland’s approach Logic model and components Levels of data collection and other features Future directions

Maryland ECMHC Evaluation Funding: Maryland State Department of Education (MSDE), Division of Early Childhood Involves 12 sites across Maryland Three year evaluation (2008-2011), with two option years Principal Investigators: Sharon Stephan: University of Maryland Bruno Anthony, PhD & Deborah Perry, PhD: Georgetown University Minority Business partner: CKD Communications, LLC

Early Childhood Mental Health Consultation (ECMHC) Collaborative, capacity building services provided to early care and education (ECE) providers and family members Can be initiated in response to concerns about an individual child’s social-emotional or behavioral development The impact can be measured in the domains of child, parent, staff, and family 4

Approach to Evaluation Collaborative Participatory Comprehensive Sensitive to respondent burden No site comparisons Technical assistance related to evaluation and data collection provided

Funded Projects and Evaluation Liaisons

Evaluation Logic Model Skills and Qualifications Satisfaction Relationship Early Childhood Mental Health Consultation Early Care and Education - Attitudes, Beliefs, and Stress - Classroom Environment Parenting - Behavior and Attitudes - Stress Young Children ↓Challenging Behavior ↑Social Skills Expulsions

Evaluation Components ECE Providers SDQ TOS RQS DECA Parents PSI PBI Consultants Climate ECE Directors GAS Implementation Factors Core Outcomes Program Directors Model Description, Quarterly Interview Knowledge & Skills, Service Log, Exiting Child Form, Roster, Preschool Climate Scale Exiting Child Study Consultants, ECE Directors, ECE Providers, and Parents Qualitative Interviews Note: instruments in pink will only be collected for child-specific referrals

Levels of Data Collection Program-level outcomes Measures assess changes in: ECE provider behaviors and beliefs; job stress; structural and interpersonal characteristics of the ECE setting; and the quality of the relationship between the MHC and ECE providers. Child-/Family level outcomes For child-specific referrals: Measures assess changes in the child’s behavior (rated by teacher and parent), parenting stress and behaviors; and the quality of the relationship between the MHC and ECE providers.

ECMHC Program Directors Tools Construct Frequency Model Description Program Structure Baseline, Annual Quarterly Site Interviews Program Changes Quarterly

Mental Health Consultants Tools Construct Frequency Knowledge and Skills Inventory Consultant Characteristics Baseline, Annual Service Log ECMHC Services Ongoing Classroom Roster Administrative Tool Preschool/Infant MH Climate Scale Classroom Environment Baseline, every 4 months (ongoing) Consultant Final Perceptions Form Quality of Relationship 4 months, or at Discharge Exit Study (Qualitative Interviews) Perceptions and predictors of expulsion Following an exit due to behavior issues

Preschool/Infant Climate Scale These tools are completed by the mental health consultants for every child-specific and programmatic consultation they initiate. Preschool Mental Health Climate Scale (Gilliam, undated) Infant Climate Scale (FSU, 2009) The instruments assess multiple dimensions of the classroom environment and ECE providers’ behaviors. Consultants complete these scales at baseline, at 4 months or at discharge; and if ongoing, every 4 months thereafter.

Child Care Directors Tools Construct Frequency Goal Achievement Scale Opinions and Attitudes Baseline, 4 months or at discharge Exit Study Interviews Perceptions of Expulsion When child exits due to behavior

Early Care and Education Providers Tools Construct Frequency Strengths and Difficulties Questionnaire Child Functioning Baseline, 4 months or at discharge Teacher Opinion Survey Teacher Attitudes Provider Job Stress Index Job Stress Teacher Final Perceptions Form Quality of Relationship Exit Study Interviews Perceptions of Expulsions When child exits due to behavior

Parents * For those who consent to long-term follow along Tools Construct Frequency Parenting Stress Index-Short Form Parent stress Baseline, 4 months or at discharge Parent Behavior Inventory Parenting practices Parent Final Perceptions Form Quality of relationship Devereux Early Childhood Assessment Child functioning Baseline, 4 months, 12* & 24* months Exit Study Interview Perceptions of expulsions If their child is expelled * For those who consent to long-term follow along

Exit Study Interviews Motivated by MSDE’s interest in this policy-relevant (but infrequent) outcome Always interview the consultant; seek the additional perspectives of ECE Director, provider and parent Includes open-ended questions such as: What behaviors led to the child’s exit? What was done to try to maintain the child in the placement? Did you expect this child to be expelled? Why or why not?

Professional Activity Units Working with MSDE, established process for ECE Directors and Providers to get professional activity units when complete baseline and follow-up measures for evaluation Increase participation and help with follow-up data

Current Status and Future Directions Data collection in the field: good participation from all sites Exit interviews completed with consultants, and a few ECE providers Comparison group Addition of CLASS as objective measure of change Paper on Preschool Mental Health Climate Scale with Walter Gilliam