1. Center on Early Learning Spring Symposium Bobbie Weber Wednesday, April 15, 2015

2. Goals for Today  Validation Study team able to thank Quality Improvement Specialists for support  Participants will be able to: Describe a validation study and differentiate it from other studies Understand benefits of the QRIS Validation Study for parents, providers, and the system Describe both Validation Studies 1 and 2  Study team will increase understanding of parent and provider perspectives on the Validation Study

3. Comparison of Types of Studies: Questions Asked  Validity study: Do QRIS ratings accurately differentiate levels of quality?  Implementation study: To what extent was QRIS implemented as designed? What works and does not work?  Evaluation study: Does QRIS meet the objectives set for it? To what extent does QRIS achieve desired goals?

4. Validity Study: Different Quality Measures on the Same Facility  A validity study requires two measures: The measure whose validity is being tested (QRIS ratings) The measure used as the gold standard (or at a minimum a measure which has been shown to be reliable and valid) against which the QRIS rating is validated  Both QRIS rating and gold standard are measures of quality

5. Validation Study Includes Two Studies  Study 1 QRIS Rating of quality compared to measure of: ○ Classroom quality—CLASS observation score (standard)  Study 2 QRIS Rating of quality compared to measure of: ○ Child engagement—inClass (standard) ○ Family engagement—the Family Provider/Teacher Relationship Questionnaire (standard)

6. Reasons for Doing a QRIS Validity Study  Determine if programs with higher QRIS ratings have higher scores on standard measure: Across types of care Across geographic areas of the state  Provide data to inform QRIS standard revision  Increase sustainability of QRIS by showing it is accurate and valid Increase trust of providers, parents, policy makers and the public Increases efficiency--Identify which standards/measure most effectively differentiate quality

7. Benefits of a QRIS Validity Study  Public, parents, and providers Assures that ratings actually reflect quality of care Assures that ratings work for all types of care  Providers Continuously improve the accuracy and efficiency of QRIS ratings Increase confidence of parents and community in the provider’s rating  Parents Increase confidence in using ratings to select care

8. Study 1: Purpose  Research shows an association between behavior measured by QRIS standards and higher quality educator- child interactions  CLASS observations used as measure of quality of teaching and standard to which QRIS ratings are measured: Expect to see higher CLASS scores associated with higher QRIS ratings

9. Study 1: Data Collection  Goal to observe 300 programs 20 programs of each type (certified family, registered family, center) at each of 5 “star” levels  Have observed 279 programs to date Limited number of programs have earned level 4 Limited number of registered family earned higher than a 3  Also have collected demographic surveys of classroom/group staff 420 center staff 120 family child care providers/staff

10. Study 1: Incentives  In centers observe up to 4 classrooms Center receives $20 gift card for each observed classroom  In family child care observe 1 group FCC home receives $20 gift card

11. Study 1: Request of Provider  Centers Allow PSU staff to observe up to 4 classrooms (about 2 hours per classroom) Have observed teachers complete a brief background survey, and provide consent  Certified and registered family child care Allow PSU staff to observe a group of preschooler or mixed toddlers and preschoolers Observed staff complete brief background survey, and provide consent

12. Study 2: Purpose  Stakeholders want evidence that children do better in programs with higher QRIS ratings  Research shows higher quality teaching and learning environments are associated with higher levels of: Child engagement Family engagement  Child (inClass) and family (Family and Provider/Teacher Relationship Quality-- FPTRQ) engagement scores will be used as standard for QRIS rating Expect to see higher inClass and FPTRQ scores associated with higher QRIS ratings

13. Study 2: Data Collection  Goal to observe 196 programs 116 centers with ratings of 2 to 5 stars ○ 4 children per center 80 CF with ratings of 2 to 5 stars ○ 2 children per certified family child care  Data collection to begin May 2015

14. Study 2: Incentives  $100 per observed center $75 gift card for the program $25 gift card for the classroom  $75 gift card per observed certified family  $20 gift card per participating family

15. Study 2: Request of Provider  Help recruit families and have them complete FPTRQ and a consent form  Allow PSU staff to observe a classroom to complete both CLASS and inClass (about 2 hours per classroom)  Have observed teachers complete FPTRQ (provider version)  Director completes FPTRQ (Director version)

16. Continuation of CLASS Observations  CLASS observations will continue through Study 2  Additional data will strengthen ability to find associations between CLASS scores and QRIS ratings  CLASS scores will strengthen ability to understand what we learn from inClass and FPTRQ measures.

17. Study Timeline  Data collection Study 1-initial – July 2013-May 2015 Study 1-continuation – July 2013-June 2016 Study 2 – May 2015-June 2016  Analyze data Study 1-initial — June-September 2015 Study 1-continuation — July-December 2016 Study 2 — July-December 2016  Report findings Study 1-initial — Fall 2015 Study 1-continuation — Spring 2017 Study 2 — Spring 2017

18. Question for Quality Improvement Specialists  In discussing the Validation study with providers, what language should we use or avoid?  Do you think getting providers to approach parents and invite them to participate will be challenging? How can we give them language that will make them feel comfortable doing this? How can we make sure providers invite all parents and not just a few 'select' parents?  How do we not overwhelm providers with what we are asking them to do? What else can we offer besides coming out there and inviting families ourselves? What could make providers feel a part of the study?  Do you have ideas on what would make parents more likely to participate?