Measuring Program Outcomes Boys & Girls Clubs of America

Workshop Climate Hours Breaks Lunch Cell phones Rest Rooms Learning Credit Other?

Climate - Introductions Interview: Name Club Years working with youth Role in Club Role in measurement of program

Expectations At your tables: Appoint a note keeper Identify your expectations for this workshop

Training Objectives Be aware of the board, growing interest in outcomes; Understand the uses and limitations of outcome measurement; Be able to identify appropriate program outcomes and outcome indicators; Understand the basic steps in creating and using logic models; Learn the benefits and how to utilize the Outcomes Measurement Tool Kit. Develop a plan to begin outcome measurement at your Club.

BGCA’s Mission Statement To inspire and enable all young people, especially those from disadvantaged circumstances, to realize their full potential as productive, responsible and caring citizens.

What Makes Clubs Great? In groups of three: Discuss what makes Clubs GREAT What would you do if someone asked you to “prove it?” How would you back up the statement “We are great?” Measuring Outcomes is an effective backup. Benefits include

Benefits of Measurement Determines effectiveness of services and programs Becomes resource development tool Increases awareness of Club Enables Clubs to respond to increased levels of accountability (United Way, public dollars, parents, funders, etc.)

Overview of Topic 2: The Logic Model Terms and Definitions – A common Language The Logic Model – A look at structure Outcomes – The fourth column Logic Model Shuffle – The if. . . than. . . exercise Building a Logic Model – putting it all together

Logic Model Terminology Handout: Selected Terms Inputs - Resources a program uses to achieve program objectives. Activities - What a program does with inputs. This is our program. Outputs - Products of the program (activities), which can be counted. Outcomes – Benefits for participants during or after involvement with the program. Initial Outcomes – First benefits or changes participants experience. Intermediate Outcomes – Link the initial outcomes to the longer-term outcomes desired for participants. Long-Term Outcomes – The ultimate goals of the program. Outcome Indicators - The specific information collected to track a program’s success in achieving the targeted outcome. Outcome Targets – A numerical objective for a program’s level of achievement.

Logic Model – What do we now measure? What do you measure at your club? Flip Chart: Fit in outline of Logic Model

Logic Model Logic Model Handouts Terminology Inputs Activities Outputs Outcomes   Initial: Intermediate: Long Term: Terminology

Handout: Program Outcome Model   Inputs   Activities Outputs Outcomes Resources Services Products Benefits for Participants Money Staff Volunteers  Equipment and Supplies CConstraints   Laws  Regulations Ø     Training Ø     Education Ø     Counseling Ø     Mentoring Ø      Internships   Classes Taught   Counseling sessions conducted  Educational Materials Distributed  Hours of Service Delivered ØParticipants Served New Knowledge Increased Skills  Changed Attitudes or Values Ø Modified Behavior Ø Improved Condition Ø     Altered Status Handout: Program Outcome Model

Outcomes - are what happens to members involved in the program – how members are better off. They include changes in the following: Knowledge Skills Attitudes Values Behavior Condition Status

BGCA’s Youth Development Outcomes Positive Self-Identity Community and Civic Involvement Health and Well-Being Moral Compass Competencies: Educational Social Emotional Cultural Employment Handout: BGCA Youth Development Outcomes

Program Outcome Criteria For each outcome: It is reasonable to think the program can influence the outcome in a non-trivial way, even though it can’t control it. Would measurement of the outcome help identify program successes and pinpoint problems? Will the program’s various “publics” accept as a valid outcome of the program? For the set of outcomes: Do they reflect the program logic – the chain of changes program outputs are intended to set in motion for participants? Do the longer-term outcomes represent meaningful benefits or changes in participants’ conditions or quality of life? Are potential negative outcomes identified?

Uses of Outcome Findings INTERNAL Provides direction for staff Identify training needs Support strategic planning Suggest outcome targets Focus board members’ attention on programmatic issues EXTERNAL Recruit talented staff and volunteers Promote program to potential participants and referral sources Identify partners for collaboration Enhance program’s public image Retain and increase funding

Work with the Logic Model Handouts: Conceptual Chain Logic Model Shuffle Review Start Smart Logic Model Arrange Pieces of a Logic Model Build Logic Models: A Program you are familiar with A Core Program Area

What is the Tool Kit? Web-based Application Based on member survey administration Resource for documenting and assessing change

Development of The Tool Kit Created in 1998 Initial instrument was a paper version Contracted with Policy Studies Associates 12 Organizations piloted in Fall 2003

What’s New in the Tool Kit Reliability of Questions Improved Scoring is Simplified Charts Available with a Click of a button Handout: Outline of Outcome Measurment Tool Kit

Handout: Outcome Indicator Definitions

Three Distinct Surveys Elementary School Middle School High School

Reporting the Results Compare the Club activities against scores. Convene staff to help decide how data should be analyzed and reported. Create an action plan.

Use the 75/25 Rule A good rule of thumb is … Have 75% of all survey takers respond in the “Doing Great” or “Doing Fine” categories. Think about how to affect change if less than 25% Bonus slide

Getting Started Log on www.bgca.net and request an ID and password Designate a lead Club staff Provide Staff Training Determine what information you want to learn

Sampling More Increases Reliability Target 30 Members Per Age Group Make Random Selections Just Get Started!

Consent Youth younger than age 18 need parental consent. It’s the law! There are several means to secure consent: Passive Active Two Consent Handouts

When Should I Survey? Surveys should be conducted annually, possibly in the fall after enrollment has been stabilized. Conduct surveys at the same time each year. Surveys should be linked with the need for outcomes information.

Helpful Tips Have a survey administrator available to provide oral instructions Have survey copies available for parents Use lots of incentives Have fun!

Target Audience Chief Professional Officers Club Staff Funders Boards Regional Service Directors

Other BGCA Evaluation Resources Commitment to Quality Program Reviews Peer Reviews SOEs Proven Results: A Compendium Program Reviews: An independent assessment of the quality of Club Programming and environment. Questions drawn from CTQ Based on observations Based on five key areas Scoring similar to SOEs SOEs: Used to measure organizational development across broad categories. Led by Board Presidents, CPOs and RSDs. Focus is on areas that create sustainability Identify aspects that need improvement Create a development plan for growth Proven Results: A summary of BGCA evaluations of its national programs and services. An evaluation of other youth development organizations Additional resources available to Club professionals

B&GCA Assessment Tools Organization: Standards of Organizational Effectiveness (SOE) Club Experience: Commitment to /Quality Youth Development Outcomes Youth Development Outcome Measurement Kit

Getting Started Develop anOutcome Measurement Workgroup Create plan for the Tool Kit Identify programs to be measured Build logic models for programs Manage expectations Create data collection strategies Analyze and present results Collect praise for job well done!

Responsibilities of Outcome Measurement Work Group Create measurement strategy Develop a timeline Identify the outcomes to measure Construct a program logic model when necessary Identify outcome indicators Prepare and test data collection instruments Plan for and monitor a pilot process Analyze and share results Evaluate the results and make changes

Criteria for Choosing First Program for Outcome Measurement It has a recognizable and reasonably defined mission and clientele. It represents a substantial portion of the Club’s activity. Funders or others are interested in the program’s results. Program supervisors and staff are likely to be supportive of an outcome measurement effort.

Reporting the Results Description of goals Description of outcomes explored and Club’s goals and strategies Description of survey instruments, sample process and approach to administration Description of the sample and comparison to overall characteristics of Club members Description of results Uses of results

Who Is Interested Tool Kit Results At your tables, brainstorm and list “What’s In It” for your assigned “interested party.” Boards CPO’s Staff Funders B&GCA (RSD’s, Others?) Others?

Wrap Up What was learned? How might you use it? What will each of you do: Write what you will do: Within 30 days Within 90 days Share one of the things you will do.

Questions

(401) 253-5309 Ja.reinholt@cox.net Jack Reinholt (401) 253-5309 Ja.reinholt@cox.net