1. A joint venture between the Forum and High/Scope. Austin Youth Program Quality: A Snapshot Dr. Raphael Travis – Texas State University School of Social Work

2. Raphael Travis, Jr., DrPH, LCSW 2 Overview  Five waves of data collection and analysis  Sixth wave coming soon  219 assessments entered and analyzed  Only possible from a dedicated few volunteering time, energy and resources … you!

3. Raphael Travis, Jr., DrPH, LCSW 3

4. 4 A Snapshot 1.Average levels of program quality areas across programs  Successes and Opportunities 2.Analyses across each wave of data 3.Distinctions between self-assessments and external assessments 4.A spotlight on what predicts high youth engagement (our most desirable indicator of quality)

5. Raphael Travis, Jr., DrPH, LCSW 5 The Pyramid of Program Quality 5 Plan Make choices Reflect Partner with adults Lead and mentor Be in small groups Experience belonging Engagement Reframing conflictEncouragement Skill building Active engagement Session flow Welcoming atmosphere Supportive Environment Interaction Safe Environment Healthy food and drinks Program space and furnitureEmergency procedures Psychological and emotional safety Physically safe environment Professional Learning Community Youth Voice and Governance

6. Raphael Travis, Jr., DrPH, LCSW 6 Score Reports: Keep in Mind  Observation scores represent a snapshot –this has limitations and value.  These are aggregates where multiple observations occurred.  The overall story is more important than the actual numbers… But, today we will talk about numbers  What you do with it matters (how you step up to action). Contemplation Preparation Action

7. Raphael Travis, Jr., DrPH, LCSW 7 7 Wave 5: National Sample vs. External Assessments

8. Raphael Travis, Jr., DrPH, LCSW 8 8 Wave 5: Self Assessment vs. External Assessment

9. Raphael Travis, Jr., DrPH, LCSW 9 Wave 4 vs. Wave 5 (All Assessments)

10. Raphael Travis, Jr., DrPH, LCSW 10 Wave 5 Safe Environment: Very High Scores

11. Raphael Travis, Jr., DrPH, LCSW 11 Wave 5 Supportive Environment: Very High Scores – Even Higher (~5.0)

12. Raphael Travis, Jr., DrPH, LCSW 12 Wave 5 Interaction: Big Spread in Programs Occur; Scores Dip Below 2 and 3

13. Raphael Travis, Jr., DrPH, LCSW 13 Wave 5 Engagement: Substantial Variability; But Majority are 3 and Below

14. Raphael Travis, Jr., DrPH, LCSW 14 Predicting Engagement  Certain processes are consistently more important than others in predicting high levels of engagement

15. Raphael Travis, Jr., DrPH, LCSW 15 Safety  SUCCESS:  SAFEB3/4 – Ventilation, lighting, temperature okay  OPPORTUNITY:  SAFEC4/C6 – Appropriate safety and emergency equipment; access to outdoor space is supervised

16. Raphael Travis, Jr., DrPH, LCSW 16 Supportive  SUCCESS:  SUPPORTF2/F3/J1/J2 – Warm tone, friendly, smile; staff actively involved, acknowledge accomplishments specifically  OPPORTUNITY:  SUPPORTK2/K3 – (Conflict) Staff seek input from both youth about cause and solution of negative behavior; encourage youth to reflect on actions and consequences when conflict

17. Raphael Travis, Jr., DrPH, LCSW 17 Interaction  SUCCESS:  INTERACTL3/O2 – Youth identify with program (shared language, gestures, and traditions); staff provides explanations, guidelines and directions  OPPORTUNITY:  INTERACTM2/N2/N3 – Staff use two or more ways to form groups; youth have chances to mentor others and lead groups

18. Raphael Travis, Jr., DrPH, LCSW 18 Engagement  SUCCESS:  ENGR1/R4 – All youth engage in intentional reflecting on activities; staff provide structure for youth feedback on activities  OPPORTUNITY:  ENGP2/Q2 – Two or more strategies used to plan projects; all youth can make at least open-ended choice during program processes

19. Raphael Travis, Jr., DrPH, LCSW 19 Explaining High Youth Engagement  Safety, supportive, and interaction explained half of all the variability in engagement scores,  in addition, even more (58%) when looking only at self- assessments.

20. Raphael Travis, Jr., DrPH, LCSW 20 Significant Predictors of Engagement (All)  Wave 1: SUPPORTH  Wave 2: SUPPORTI, SUPPORTJ*  Wave 3: SUPPORTG*  Wave 4: SUPPORTH, SUPPORTJ, INTERACTN  Wave 5: SUPPORTK  ALL: SAFED, SUPPORTH, SUPPORTI, SUPPORTK, INTERACTN

21. Raphael Travis, Jr., DrPH, LCSW 21 Significant Predictors of Engagement (Self and External Assessments)  Predictors of Engagement (Self)  ALL: SUPPORTG, SUPPORTH, SUPPORTI, SUPPORTK, INTERACTL, INTERACTN  Predictors of Engagement (External)  ALL: SUPPORTH, INTERACTN

22. Raphael Travis, Jr., DrPH, LCSW 22 What Explains Engagement THE MOST?  SUPPORTH: Active Engagement  Youth can create, combine, reform materials/ideas toward tangible products; reflect on what they do  SUPPORTI: Skill-Building  Support youth in building new skills; persistence  SUPPORTJ: Encouragement  Acknowledge what youth do; actively involved with youth; open-ended questions  INTERACTN: Leading and Mentoring  Youth have opportunities to act as group facilitators and mentors

23. Raphael Travis, Jr., DrPH, LCSW 23 So What? What Does This Mean for You?  For assessors  it means continuing to do the great work you are doing.  You have collected high quality, meaningful information – both self- assessors and external assessors.  For methods trainers  it means there may be indicator training areas to highlight even if that is not the focus of your particular workshop  (e.g., active engagement, skill-building, encouragement, leading and mentoring).  These processes seem to help distinguish programs that are highest in engagement from those that are not.  For program leadership and staff  it means continuing to pay attention to how your program operates as much as you pay attention to the results.  Highlight and pay attention to colleagues that promote these influential processes and model effective program facilitation. For others, continue to support the work!