MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public Affairs University of Illinois

Overview What are the benchmarks and why are they necessary? Federal requirements Evaluation and data collection Performance measures Continuous quality improvement (CQI)

Purpose of Presentation Increase understanding for role of Benchmarks, Benchmark Reporting Requirements Understand the reporting and data collection requirements for the MIECHV programs Identify the roles that each organization involved in Benchmark reporting will play, and what additional roles that CPRD can play for MIECHV and home visiting services Identify key performance measures for MIECHV programs, both for Benchmarks and continuous quality improvement efforts Understand the role of the field data collectors

And what is not the purpose of this presentation?

Maternal, Infant, Early Childhood Home Visiting What is MIECHV? Maternal, Infant, Early Childhood Home Visiting Over time, families and home visitors build partnerships and work together to: Improve health and development. Prevent child injuries, child abuse, neglect, or maltreatment, and reduce emergency department visits. Improve school readiness and achievement. Reduce crime, including domestic violence. Improve family economic self-sufficiency. Improve the coordination and referrals for other community resources and supports.

How do we define success? Providing high-quality home visiting services that positively impact the lives of children and families enrolled in the MIECHV programs Meeting and exceeding Benchmarks Findings ways to pool our knowledge collectively to improve statewide delivery of home visiting services.

How does MIECHV fit in? MIECHV is designed to expand HV services into priority populations: Reside in communities in need of services & have low income Include pregnant women who have not attained the age of 21 Have a history of child abuse Have a history of substance abuse Have users of tobacco products Have a history of, or have children with, low student achievement Have children with developmental delays or disabilities Include members of the military

MIECHV Priority Communities Rockford- Winnebago County 65.1% White, 20.5% African American, 15.8% Hispanic Vermilion County 83.6% White, 13.3% African American, 4.4% Hispanic Chicago- Cook County 45% White, 32.9% African American, 28.9% Hispanic Elgin City ( border of Cook and Kane Counties) 69.5% White, 7.4% African American, 43.6% Hispanic Cicero Town- Chicago 51.9% White, 3.8% African American, 86.6% Hispanic Macon County 79.8% White, 16.6% African American, 2.0% Hispanic Doula Sites: N. Lawndale, E. Garfield Park, Vermilion County, Rock Island County and Waukegan

The program goals, per the HRSA website: Strengthen and improve the programs and activities Improve coordination of services for at-risk communities Identify and provide comprehensive services to improve outcomes for families who reside in at-risk communities

MIECHV Benchmarks There are a total of 6 Benchmark areas These reflect overall domains in which we expect home visiting services to have an impact. Each benchmark has a set of constructs, and these total 37 constructs across the 6 benchmarks Benchmarks were developed in the Legislation, and the constructs were developed by HRSA to address the benchmarks.

Why do we have Benchmarks? The legislation funding MIECHV required quantifiable, measurable improvements for the populations receiving services. Programs must demonstrate improvement in the following Benchmark areas: Improved maternal and newborn health Prevention of child injuries, child abuse, neglect, or maltreatment, and reduction of emergency department visits Improvement in school readiness and achievement Reduction in crime or domestic violence Improvements in family economic self-sufficiency Improvements in the coordination and referrals for other community resources and supports

Where do the Benchmarks come from?

Now, let’s look at the Benchmarks individually The Benchmarks contain constructs, specific measures to address each concept Reflect outcomes of home visiting services

Improved Maternal and Newborn Health Constructs: Prenatal Care Prenatal use of alcohol, tobacco, or illicit substances Postpartum use of contraception and Interpartum Interval Screen for Maternal Depressive Symptoms Duration of breastfeeding, well-child visits, and maternal health insurance coverage

2. Prevention of Child Injuries, Child Abuse or Maltreatment and Reduction of Emergency Department Visits Constructs: Visits for child and mother to the emergency department from all causes Information provided or training of participants on prevention of child injuries Incidence of child injuries requiring medical treatment Reports of suspected maltreatment for children in the program, reported substantiated maltreatment, and first time victims of maltreatment

3. Improvements in School Readiness and Achievement Constructs: Parent support and knowledge for children’s learning and development Parenting behaviors and parent-child relationship Parent emotional well-being or parenting stress Child’s communication, language and emergent literacy, cognitive skills Child’s positive approaches to learning and attention Child’s social behavior, emotional regulation and emotional well-being Child’s physical health and development

4. Domestic Violence Constructs: Screening for domestic violence Of families identified for the presence of domestic violence, number of referrals made to relevant domestic violence services

5. Family Economic Self-Sufficiency Constructs: Household income and benefits Employment or education of adult (mother and father) members of the household Health Insurance Status

6. Coordination and Referrals for Other Community Resources and Supports Constructs: Number of families identified and referred to available community resources The number of families who complete referrals to available community resources Number of agencies with which the home visiting provider has a clear point of contact in the collaborating community agency Number of agencies with which the home visiting provider has established a formal memorandum of understanding

What the Evaluation is and is not A way to measure and address the challenges in scale-up implementation of HV services A way to show Congress that tax dollars spent on additional HV services is important and useful A way to introduce statewide continuous quality improvement efforts A goal setting mechanism The Evaluation is not: A performance assessment to be used for funding decisions A method to weed out the poor performing HV sites A way to collapse the variety of model HV programs into a single government-managed HV organization

Illinois MIECHV: A Partnership Approach CPRD is involved in the following areas: In partnership with the state, developing measurement procedures and systems which adequately address the Benchmark reports Coordinating Benchmark data collection with state and sites Work together with home visitors to gather benchmark data Query data from MIECHV data system and prepare reporting to the feds annually Utilize data to track individuals through time in services Collect home visiting services satisfaction information Help clarify possible barriers to engagement Lead Home Visiting Continuous Quality Improvement efforts in MIECHV and throughout the Home Visiting State System

What if we don’t make improvements on the benchmarks after 3 years? Would need to develop a plan of action CPRD, the Governor’s Office, and individual community programs would work together to develop a plan and next steps The CQI process is crucial to next steps

What is CQI and how will CPRD aid in the CQI efforts? I can tell you with 95% confidence that there is less than a 65.6% possibility that the CQI will simply generate 34.8% more meaningless statistics.

What are the cohorts mentioned in the Benchmarks? The term “cohort” refers to the idea of a group of families who enter services together. But families are entering and exiting HV services all the time Cohorts in MIECHV follow one group of people and compare them to another group enrolled a year later Cohorts help us measure program change and effectiveness over time How will we know if individuals are changing? Individual change is reviewed in CQI process

Who are Field Data Collectors? What does it mean to be a field data collector? How will we all stay in contact? How will we ensure regular communication and updates? Working with the home visiting sites and home visitor Will use a shared calendar through Google Calendar (E.g. 'macondc@gmail.com‘), e-mail, call and text.

The Role of Field Data Collectors in the Benchmarks Some Measures require rigorous approach for reliability The Benchmarks created a large and taxing load on home visitors Some benchmarks require that the parent not be hindered by watching their child, such as needing to fill out a longer assessment

On-site Data Collection Field data collectors will work with home visitors to collect measures from families Assess the families at three time periods 2-4 weeks after enrollment 1 year and 2 year follow-ups Informed Consent – depending on age or emancipation status Parental consent – if mother is less than 18 years old, written parental/guardian consent is required and self-assent Self-consent – emancipated or 18 years or older, written assent is required Gift cards are provided to participants O

What is the PICCOLO? Parenting Interactions with Children: Checklist of Observations Linked to Outcomes PICCOLO items describe positive parenting PICCOLO helps practitioners identify parenting strengths to encourage parents to do more

What does PICCOLO measure? From the research literature, there are identified four important kinds of “developmental parenting” that support child development: • Affection • Responsiveness • Encouragement • Teaching

What does PICCOLO look like?

29 THINGS PARENTS DO THAT PREDICT SCHOOL READINESS Speak warmly Encourage child to do things with toys Smile at child Support child’s choices Praise child Help child do things on his or her own Stay physically close to child Verbally encourage child’s efforts Say positive things to child Offer suggestions to help child Interact in positive ways with child Show enthusiasm about what child does Show emotional warmth Explain reasons for something to child Pay attention to what child is doing Suggest activities to build on what child is doing Change activities to meet child’s interests or needs Repeat or expand child’s words or sounds Be flexible when child changes interests Label objects or actions for child Follow what child is trying to do Engage in pretend play with child Respond to child’s emotions Do activities in a sequence of steps Look at child when child talks or makes sounds Talk about characteristics of objects Ask child for information Reply to child’s words or sounds   Wait for child’s response after making a suggestion

Knowledge of Infant Development Inventory What is the KIDI? Knowledge of Infant Development Inventory The theoretical base for the KIDI was the theory that parental knowledge of infant development can influence parental behavior and child rearing practices. The KIDI is a 58-item instrument that was designed to obtain comprehensive information on parents’ factual knowledge of parental practices, child developmental processes, and infant norms of behavior. The KIDI is designed to be easily accessible to persons with limited education and to be culturally neutral.

What is the PSI? The Parenting Stress Index - Short Form The PSI-SF consists of 36 items derived from the PSI which comprise three scales: Parental Distress, Difficult Child Characteristics, and Dysfunctional Parent-Child Interaction. Designed to evaluate the magnitude of stress in the parent-child system The PSI-SF is commonly used as a screening and triage measure for evaluating the parenting system and identifying issues that may lead to problems in the child's or parent's behavior. Asking parents to complete the PSI—SF can be uncomfortable. Field Data Collectors explain that it will provide information that will help identify the sources and different types of stress that every parent can experience.

What is the HOME? Home Observation for Measurement of the Environment (HOME) is a descriptive profile which yields a systematic assessment of the caring environment in which the child is reared. The primary goal of the instrument is to measure, within a naturalistic context, the quality and quantity of stimulation and support available to a child in the home environment. Its focus is on the experience of the child in the home environment.

Other Resources Visit Tracker has various recorded webinars available. New staff should watch Illinois MIECHV Training for New Staff. https://visittrackerweb.com/ The Governor’s Office webpage has numerous up-to-date documents posted including a MIECHV Data Collection Manual. http://www2.illinois.gov/gov/OECD/Pages/MIECHVP.aspx Field Data Collection brochures for each MIECHV community (contact Mary Anne Wilson for copies) Contacts Teresa Kelly, Project Director, Strong Foundations Partnership- Governor's Office of Early Childhood Development 312-814-0905 teresa.m.kelly@illinois.gov Lesley Schwartz, Manager of Program Evaluation- Governor's Office of Early Childhood Development 312-814-4841 lesley.schwartz@illinois.gov Joanna Su, Community Systems & Capacity Building Manager-Governor's Office of Early Childhood Development 312-814-6741 Joanna.su@illinois.gov Audrey Moy, Manager of Compliance- Governor's Office of Early Childhood Development 312-814-3141 audreymoy@illinois.gov Peter Mulhall, Director , CPRD-University of Illinois 217-244-3231 mulhall@illinois.edu Mary Anne Wilson, MIECHV Research Project Specialist at CPRD, University of Illinois 217-333-3231 mawilso@uillinois.edu Deborah Kemmerer, MIECHV CQI Specialist at CPRD, University of Illinois 217-300-4158 dkem@uillinois.edu