The New Outcomes System Stage I: Adopting the Youth Outcomes Survey AIM Agencies June 2009

Today’s workshop focus: Vision, goals, and components of BBBSA’s Outcomes System Development of the Youth Outcome Survey (YOS) Guidelines and tips to effectively implement the YOS

The vision… We stand for quality. We will track the length, strength and outcomes of each Little in a consistent and accurate way so we may know how many successful matches we have in an agency and nationwide. Our outcomes are central to our case for support. We seek to communicate our results in effective and powerful terms.

The goals: To track current outcomes for each Little, each agency, and nationally. To use methodologies that are compelling to supporters, funders and researchers. To improve agency program performance and build our case for community support using outcomes.

The components: Length 6 and 12 month RR average match length (AML) Strength Strength of Relationship (SoR) Outcomes Youth Outcomes Survey (YOS) School records (future) other quality processes? quality assurance ratings match support compliance %

BBBSA’s Outcome System

What is the Youth Outcomes Survey? Built on the findings of our past P/PV studies that indicate specific areas where our work yields impact Designed by BBBSA in consultation with Rhodes, Grossman, and Herrera “Stronger” than POE

Why “stronger?” Pre/post and age cohort comparisons tell us more Different levels of impacts at different lengths? What makes some matches more successful than others? Continues to validate the methodologically sound and rigorous results from the P/PV studies

And the pilot results? Sound scales Changed some questions Results showed statistical significance in key outcome areas Positive feedback from all levels of staff about feasibility of network-wide roll out Great recommendations for large- scale implementation

Summary of the Youth Outcomes Survey Academics –Scholastic Competence –Educational Expectations –Self-Assessment of Academic Performance Relationships –Social Acceptance –Parental Trust Risky Behaviors –Attitudes Towards Risk

More about the YOS Pre-test (baseline surveys) and post-test (follow-up surveys) For youth ages 9 and up 32 question survey For both CB and SB matches

MeasureRelated Long- Term Outcomes Rationale Scholastic Competence Academic achievement Increases in this scale often precede performance improvement. Higler levels of scholastic competence are associated with better mental health and high grades. Educational Expectations Academic achievement Youth’s academic expectations and plans have shown longitudinal associations with school performance and standardized test scores. Self- Assessment of Academic Performance Academic Achievement Correlation with actual academic performance. Social Acceptance Academic achievement and delinquency avoidance Improvements in peer relations have been linked to school achievement, improved conduct, and lower likelihood of dropping out. Parental TrustAcademic achievement and delinquency avoidance Improved parental relationships are associated with improved grades and attendance and reduced substance abuse. Attitudes Towards Risk Delinquency avoidance Correlation with actual risk behaviors.

Scholastic Competence I have trouble figuring out the answers in school. I feel that I am just as smart as other kids. I am very good at my schoolwork. I’m pretty slow in finishing my school work. I often forget what I learn. I do very well at my class work.

Educational Expectations How sure are you that you will finish high school? How sure are you that you will go to college? How sure are you that you will finish college?

Self-Assessment of Academic Performance Mathematics Reading or language arts Social studies Science

Social Acceptance I am always doing things with a lot of kids. I wish that more people my age liked me. I find it hard to make friends I would like to have a lot more friends. I am popular with others my age. I have a lot of friends.

Parental Trust How often do I feel that my parents respect my feelings. How often do I feel that my parents accept me as I am. How often do I feel that when I’m angry about something, my parents try to be understanding.

Attitudes Towards Risk Using tobacco (cigarettes, cigars, smokeless or chewing tobacco)? Taking drugs that aren’t given to them by a doctor or parent? Drinking alcohol without their parents knowing? Skipping school without permission? Hitting someone because they didn’t like something they said or did? Breaking rules in school? Being late for school?

Plus… 2 truancy questions –In the past 30 days, have you been absent from school? –In the past 30 days, have you been late for school? 1 “special adult” question

But should we rely on youth responses?? Widely accepted in research community Unique position of youth as most accurate source of data Potential for high response rates Valuable and essential complement to hard data Positive indicators of accuracy

Administration Guidelines for Youth Outcomes Survey Who gets it? –All Littles ages 9 and over –Both CB and SB programs What about younger Littles? –Can use youth POE if needed

Administration Guidelines for Youth Outcomes Survey When do they get it? –Baseline Surveys: before match is made (up to 30 days prior) –Follow-up Surveys: CB: at 12 month match anniversaries SB: at end of school year

Administration Guidelines for Youth Outcomes Survey Who gives it? –BBBS staff –Never our Bigs! How should it be completed? –Preferably in person – then by phone –No e-Form functionality –Staff should read survey to Littles, even if they know how to read! Where does it go? –In AIM

Consents Update your child application Passive consent form for all children/parents in current processing

What about the data? 2 methodologies: –Pre/post comparison –Age cohort comparison Value of “statistically significant” differences

Outcomes Scorecard Template provided AIM will help with data gathering Share/discuss with fund development team

Possible / Expected Program Implications? Survey fatigue? Time needed to complete baseline and follow up surveys Need to train staff How to expect higher response rates? Implications if results, at first, aren’t as positive as POE?

Suggested Implementation Inform “up” about evolving outcome evaluation system Train staff Update program manual to reflect use of new forms and new procedures Look out for continued webinars and check out resources on Agency Connection