Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

“Not everything that counts can be counted, not everything that can be counted counts.”

“So What?”

“In God we trust, all others bring data.”

“Data is the new sexy!”

Why is Outcome Measurement important?

Era of accountability Doing good is not enough Stakeholders are asking more difficult questions The Importance of Measurement

Formal Program EvaluationOutcome Measurement Does the program work? How well is it working? ProvingImproving Research toolManagement tool Applies to programsApplies across org Analyzing Learning Theory of Change Best Practices Program Evaluation vs Outcome Measurement:

Key Elements of Impact Safe, positive environment Fun Supportive relationships Opportunities and expectations Recognition

Making Every Day Count Longitudinal Study Tracked 7 th and 8 th graders for 30 months Linked levels of attendance at Boys & Girls Clubs to outcomes Finding – MORE IS BETTER

Outcome (All changes are positive changes) Minimum Level of Attendance Over 30- Months Related to Positive Change in Outcome 52 Days122 Days244 Days366 Days School effort Confident in ability Integrity Community service Initiate marijuana use Future connectedness Shyness Aggression Initiate weapon carrying Stopped by police Initiate drinking Initiate Smoking Number times skipped Initiate sexual activity Negative peers

Four things led to teen participation… 1.Fully engaged in a variety of activities 2.Friends came to the Club 3.Involved in leadership roles 4.First became involved as preteens or even younger

Making Every Day Count affirms… Attendance matters Participation matters Fun matters Relationships matter Programs matter

Annual Visits Definition: The number of times a member attends the Club over the course of the year. It’s important to track this metric times per year is the baseline Increase attend levels of all members regardless of current attendance patterns

A big thank you to… Boys & Girls Clubs of Greater Milwaukee Boys & Girls Clubs of Omaha

Planning a Measurement Strategy: Key Steps Identify key organizational priorities and select key indicators Track attendance and program participation Locate and collect data Understand and use data Communicate results

Review sources that outline your priorities including your strategic plan Review the needs of your community Involve your board by asking questions Identify Organizational Priorities

Success Equation Components A + B + C = D What are you ultimately trying to accomplish? Priority Outcomes What changes in behavior or condition are required? Indicators How do we measure progress against our outcomes? Key Strategies Which programs or activities will drive results?

Use your success equation and priority outcomes as your guide Start with the basics: attendance and program participation Continue with more complex indicators: changes in knowledge/attitude and changes in behavior Use the Outcomes Framework as a guide Select indicators that are feasible to collect and credible to communicate Select Indicators to Track Progress

Practical – Can you realistically capture this data without a lot of cost or difficulty? Actionable – Will the data allow you to make decisions in a timely manner? Relevant – Does it answer the “so what” question? Controllable – Is it within your sphere of influence or control? Characteristics of Good Indicators

more time spent going to the Club was linked to positive change research supports the importance of regular attendance and the links to positive outcomes. at least 52 times per year is a baseline Clubs should strive to increase the attendance levels of all members Average daily attendance is an important measure of Club’s operational capacity, but does not measure individual members and is therefore not an indicator of changes for youth. Track Attendance and Program Participation

Locate and Collect Data Collect on a program-by-program basis Leverage existing tools, i.e. test or surveys Use Club-wide surveys when warranted Tests or Surveys can be administered by hand or electronically Results from the tests or surveys can be tracked in the member’s management system.

OutcomeIndicatorIndicator Definition Location of data FrequencyCollect Where Report Academic Achievement Indicator 1 Indicator 2 Indicator 3 Locate and Collect Data

Improve program and/or youth results Share progress and success Share results alone 50% of members participated in community service. Share results compared past performance (growth/change) Participation in community service activities has increased by 25%. Understand and Use Results

IndicatorSample Impact StatementLinks to External Research Number of volunteer hours completed by Club members 75% of Boys & Girls Clubs of Main Street members participated in volunteer service activities in 2008 and contributed a total of 1,000,000 hours of service. Young people who volunteer are likely to do better in school, graduate, maintain positive attitudes towards society and avoid risky behaviors (4) Simply based on regular member attendance and program participation, Clubs can demonstrate the impact they are having by linking participation to external research on positive activities:

Explore and define how measurement can and will be used Setting the right tone and creating ownership and buy-in Setting the process in motion Identify the processes and plans needed to bring measurement to life Engage your Staff

Available Resources Demonstrating Impact Guide Outcome Measurement Framework BGCA - Public/Private Ventures Study Distance Learning courses Abigail Diner, Karen MacDonald, Romero Brown,