Operational Excellence: Outcomes Evaluation System Impact and Outcomes Breakout Session June 14 th, 2011 Sandra LaFleur, Senior Director of Youth Outcomes 2011 National Conference Dallas, Texas June

Purpose & Desired Outcomes Review critical components of BBBS’ Outcome Evaluation System (OES) Highlight latest data analyses / results Reinforce focus on and clarify understanding of the OES Others?

2011 National Conference Dallas, Texas Today’s framing… Understand it! – The rationale for it – The connection to both the past and the present Do it! – Are you using RR, SoR, YOS? – How are you tracking it? Use it! – Case for support – Indications for improvement

Big Brothers Big Sister’s Outcomes Evaluation System (OES) Click to add text 2011 National Conference Dallas, Texas BBBS’ collective, evidenced-based approach to mentoring–a professional system that changes the lives of our children for the better, forever. Builds upon our two P/PV study findings that indicate specific areas where our work yields impact Builds upon BBBS’ ability to institute common service delivery methodologies and measures through AIM Gives us real-time, agency-specific information on our progress at achieving critical youth outcomes in addition to our nationwide reach

2011 National Conference Dallas, Texas Established positive impact of mentoring on key youth development areas – risk reduction, educational success and attitudes 1995 BBBS P/PV Randomized-Controlled Study (CBM) Continued to support positive impact of mentoring; but highlighted the importance of match quality (retention) 2007 BBBS P/PV Randomized-Controlled Study (SBM) The YOS survey is designed to measure those areas where impact was found in BBBS Randomized- Controlled Studies. YOS Survey Data Connecting Research to YOS Survey Leveraging long-standing research to account for real-time progress in our youth across the Big Brothers Big Sisters network

2011 National Conference Dallas, Texas We didn’t get here overnight… BBBS has developed Unmatched Capabilities to Link Research & Process to Drive Outcomes on a National Scale

2011 National Conference Dallas, Texas Components of BBBS’ Outcomes Evaluation System (OES) Eight (8) Measurable Outcome Areas Focused on: Community Impact Future: Additional 3 rd party research Future: Targeted hard data collection Youth Outcomes: YOS survey Match Quality Strength (SOR survey) Outcomes/Accountabilities Educational success Avoidance of risky behavior Aspirations, confidence, relationships What We Achieve & Why It Matters Match Quality Length (RR=retention rate) Community Impact

2011 National Conference Dallas, Texas Research…to Outcomes…to Accountabilities

2011 National Conference Dallas, Texas Reflect… Does your board know this information? Does your agency leadership understand this information? Does your program staff recognize their connection to this?

2011 National Conference Dallas, Texas

MATCH SUPPORT Match Support Complete % 96 %95.5%89.7%6.3% Match Support On Time % 84.7 %84.9%66.5%18.2% % Completion SOR/SITE 3 MO % 75%100% 52.0% % Completion SOR/COMM 3 MO %64% 59.0% 46.0% % Completion of YOS/Baseline %97.9% 91.7% 92.2 %6.2% % Completion of YOS/Annual201170% 52.9%65.0% 0 % Average Match Length SITE mo mo 11.5 mo-19.1 mo Average Match Length COMM mo 16.7 mo18.7 mo 17.3 mo -3.2 mo 12 Month Retention COMM % 56.6 %49.2% 15.1%41.5% 12 Month Retention SITE % 37.3 %36.1% 11.8%25.5% TEAM Total Youth Served /680CB 248SB 1028/727CB 301SB 1,225/704 CB 310SB -10.5% Closures201140%12.3%8.6%12.4% -1.1%

2011 National Conference Dallas, Texas SoR 3 month vol compliance YOS follow-up compliance REGIONS 2010Q Q NE 45%64% 45%50% SE 61%75% 59%66% MW 55%61% 63%52% SW 63%78% 58%60% W 57%70% 57%53%

2011 National Conference Dallas, Texas Reflect… Are you currently tracking your agency’s performance on 12 month RR, SoR compliance, and YOS compliance? Do you share these numbers with the whole staff? Are these goals broken down into individual goals for appropriate staff? Are they held accountable for performance on those individual goals critical to OES?

2011 National Conference Dallas, Texas Developing Your Case for Support Define Your Strategic Outcome Areas Select Appropriate YOS Measures Based on an Understanding of Your Trends Collect & Analyze Data (compliance is key) Tell Your Story

2011 National Conference Dallas, Texas Outcomes Data – More Terms Correlates to the three accountability areas identified in the BBBSA Strategic Direction. Your local needs may differ or may require a focus in one particular area. Strategic Outcomes Areas Also called Outcome categories (8) and sub-categories on the YOS Score and Outcome reports. Outcomes Measures Refers to the number of youth whose scores indicate improvement (NOT the percentage change from baseline score to follow-up score) Percentage of Youth Improving Percentage of Youth Staying the Same Statistically Significant Refers to the number of youth whose scores indicate no change. For example, staying the same in a measure that tends to decline as a child enters teen years could be meaningful. A statistical significance test determines whether a change in scores is due to chance. For example, if the p value =.05, there is only a 5% likelihood that the change is due to chance

2011 National Conference Dallas, Texas Preliminary YOS Data: Early evidence of positive youth outcomes YOS ScalesCBSB Social acceptance** Scholastic competency** Grades* Educational expectations** Attitudes towards risk*(*) Parental trust** Truancy Special adult**

2011 National Conference Dallas, Texas Preliminary YOS Data: Early evidence of positive youth outcomes SUB SCALES YOS ScalesCBSB Finish High School (Educ Exp)** Go to college* Finish college* Drugs* Hitting* Breaking rules in school* Being late to school*

2011 National Conference Dallas, Texas Preliminary YOS Data: SPECIAL POPULATIONS YOS ScalesCBMSBMCBM Children w/ incarcerated parent SBM Hispanic Girls SBM Hispanic Boys Social acceptance***** Scholastic competency ***** Grades** Educational expectations **** Attitudes towards risk *(*)* Parental trust*** Truancy Special adult*****

2011 National Conference Dallas, Texas Preliminary YOS Data: (Future Analysis) SPECIAL POPULATIONS YOS Scales Foster Care Youth Juvenile Justice referrals CBM African American SBM African American CBM Native American SBM Native American Social acceptance Scholastic competency Grades Educational expectations Attitudes towards risk Parental trust Truancy Special adult

2011 National Conference Dallas, Texas Predicting match closures with SoR… when controlling for background variables, both 3 month volunteer and child SoR scores are predictive of match closures Vol SoR scale ranking: Confidence Lack of Frustration Closeness Connectedness Child SoR scale ranking: Closeness Lack of Disappointment Coping

2011 National Conference Dallas, Texas Revisiting SoR Cut Points Volunteer 3 month SoR Score > % more likely to “survive” than matches below this level Child 3 month SoR Score > % more likely to “survive” than matches below this level

2011 National Conference Dallas, Texas Reflect… How successfully is your agency leveraging your OES data in building your case for support? Are your OES performance results used to discuss improvements or enhancements to program operations?

2011 National Conference Dallas, Texas Next steps… Assess your obstacles and your facilitators Talk with your agency team – build your action plan