1. Measuring Program Outcomes
Boys & Girls Clubs of America

2. Workshop Climate Hours Breaks Lunch Cell phones Rest Rooms
Learning Credit
Other?

3. Climate - Introductions
Interview:
Name
Club
Years working with youth
Role in Club
Role in measurement of program

4. Expectations At your tables: Appoint a note keeper
Identify your expectations for this workshop

5. Training Objectives
Be aware of the board, growing interest in outcomes;
Understand the uses and limitations of outcome measurement;
Be able to identify appropriate program outcomes and outcome indicators;
Understand the basic steps in creating and using logic models;
Learn the benefits and how to utilize the Outcomes Measurement Tool Kit.
Develop a plan to begin outcome measurement at your Club.

6. BGCA’s Mission Statement
To inspire and enable all young people, especially those from disadvantaged circumstances, to realize their full potential as productive, responsible and caring citizens.

7. What Makes Clubs Great? In groups of three:
Discuss what makes Clubs GREAT
What would you do if someone asked you to “prove it?”
How would you back up the statement “We are great?”
Measuring Outcomes is an effective backup. Benefits include

8. Benefits of Measurement
Determines effectiveness of services and programs
Becomes resource development tool
Increases awareness of Club
Enables Clubs to respond to increased levels of accountability (United Way, public dollars, parents, funders, etc.)

9. Overview of Topic 2: The Logic Model
Terms and Definitions – A common Language
The Logic Model – A look at structure
Outcomes – The fourth column
Logic Model Shuffle – The if. . . than. . . exercise
Building a Logic Model – putting it all together

10. Logic Model Terminology
Handout: Selected Terms
Inputs - Resources a program uses to achieve program objectives.
Activities - What a program does with inputs. This is our program.
Outputs - Products of the program (activities), which can be counted.
Outcomes – Benefits for participants during or after involvement with the program.
Initial Outcomes – First benefits or changes participants experience.
Intermediate Outcomes – Link the initial outcomes to the longer-term outcomes desired for participants.
Long-Term Outcomes – The ultimate goals of the program.
Outcome Indicators - The specific information collected to track a program’s success in achieving the targeted outcome.
Outcome Targets – A numerical objective for a program’s level of achievement.

11. Logic Model – What do we now measure?
What do you measure at your club?
Flip Chart: Fit in outline of Logic Model

12. Logic Model Logic Model Handouts Terminology Inputs Activities Outputs
Outcomes
Initial:
Intermediate:
Long Term:
Terminology

13. Handout: Program Outcome Model
Inputs
Activities
Outputs
Outcomes
Resources
Services
Products
Benefits for Participants
Money
Staff
Volunteers
 Equipment and Supplies
CConstraints
  Laws
 Regulations
Ø     Training
Ø     Education
Ø     Counseling
Ø     Mentoring
Ø      Internships
  Classes Taught
  Counseling sessions conducted
 Educational Materials Distributed
 Hours of Service Delivered
ØParticipants Served
New Knowledge
Increased Skills
 Changed Attitudes or Values
Ø Modified Behavior
Ø Improved Condition
Ø     Altered Status
Handout: Program Outcome Model

14. Outcomes - are what happens to members involved in the program – how members are better off. They include changes in the following:
Knowledge
Skills
Attitudes
Values
Behavior
Condition
Status

15. BGCA’s Youth Development Outcomes
Positive Self-Identity
Community and Civic Involvement
Health and Well-Being
Moral Compass
Competencies: Educational Social Emotional Cultural Employment
Handout: BGCA Youth Development Outcomes

16. Program Outcome Criteria
For each outcome:
It is reasonable to think the program can influence the outcome in a non-trivial way, even though it can’t control it.
Would measurement of the outcome help identify program successes and pinpoint problems?
Will the program’s various “publics” accept as a valid outcome of the program?
For the set of outcomes:
Do they reflect the program logic – the chain of changes program outputs are intended to set in motion for participants?
Do the longer-term outcomes represent meaningful benefits or changes in participants’ conditions or quality of life?
Are potential negative outcomes identified?

17. Uses of Outcome Findings
INTERNAL
Provides direction for staff
Identify training needs
Support strategic planning
Suggest outcome targets
Focus board members’ attention on programmatic issues
EXTERNAL
Recruit talented staff and volunteers
Promote program to potential participants and referral sources
Identify partners for collaboration
Enhance program’s public image
Retain and increase funding

18. Work with the Logic Model
Handouts: Conceptual Chain
Logic Model Shuffle
Review Start Smart Logic Model
Arrange Pieces of a Logic Model
Build Logic Models:
A Program you are familiar with
A Core Program Area

19. What is the Tool Kit? Web-based Application
Based on member survey administration
Resource for documenting and assessing change

20. Development of The Tool Kit
Created in 1998
Initial instrument was a paper version
Contracted with Policy Studies Associates
12 Organizations piloted in Fall 2003

21. What’s New in the Tool Kit
Reliability of Questions Improved
Scoring is Simplified
Charts Available with a Click of a button
Handout: Outline of Outcome Measurment Tool Kit

22. Handout: Outcome Indicator Definitions

23. Three Distinct Surveys
Elementary School
Middle School
High School

26. Reporting the Results Compare the Club activities against scores.
Convene staff to help decide how data should be analyzed and reported.
Create an action plan.

27. Use the 75/25 Rule A good rule of thumb is …
Have 75% of all survey takers respond in the “Doing Great” or “Doing Fine” categories.
Think about how to affect change if less than 25%
Bonus slide

28. Getting Started Log on www.bgca.net and request an ID and password
Designate a lead Club staff
Provide Staff Training
Determine what information you want to learn

29. Sampling More Increases Reliability Target 30 Members Per Age Group
Make Random Selections
Just Get Started!

30. Consent
Youth younger than age 18 need parental consent. It’s the law! There are several means to secure consent:
Passive
Active
Two Consent Handouts

31. When Should I Survey?
Surveys should be conducted annually, possibly in the fall after enrollment has been stabilized.
Conduct surveys at the same time each year.
Surveys should be linked with the need for outcomes information.

32. Helpful Tips
Have a survey administrator available to provide oral instructions
Have survey copies available for parents
Use lots of incentives
Have fun!

33. Target Audience Chief Professional Officers Club Staff Funders Boards
Regional Service Directors

34. Other BGCA Evaluation Resources
Commitment to Quality
Program Reviews
Peer Reviews
SOEs
Proven Results: A Compendium
Program Reviews:
An independent assessment of the quality of Club Programming and environment.
Questions drawn from CTQ
Based on observations
Based on five key areas
Scoring similar to SOEs
SOEs:
Used to measure organizational development across broad categories. Led by Board Presidents, CPOs and RSDs.
Focus is on areas that create sustainability
Identify aspects that need improvement
Create a development plan for growth
Proven Results:
A summary of BGCA evaluations of its national programs and services.
An evaluation of other youth development organizations
Additional resources available to Club professionals

35. B&GCA Assessment Tools
Organization:
Standards of Organizational Effectiveness (SOE)
Club Experience:
Commitment to /Quality
Youth Development Outcomes
Youth Development Outcome Measurement Kit

36. Getting Started Develop anOutcome Measurement Workgroup
Create plan for the Tool Kit
Identify programs to be measured
Build logic models for programs
Manage expectations
Create data collection strategies
Analyze and present results
Collect praise for job well done!

37. Responsibilities of Outcome Measurement Work Group
Create measurement strategy
Develop a timeline
Identify the outcomes to measure
Construct a program logic model when necessary
Identify outcome indicators
Prepare and test data collection instruments
Plan for and monitor a pilot process
Analyze and share results
Evaluate the results and make changes

38. Criteria for Choosing First Program for Outcome Measurement
It has a recognizable and reasonably defined mission and clientele.
It represents a substantial portion of the Club’s activity.
Funders or others are interested in the program’s results.
Program supervisors and staff are likely to be supportive of an outcome measurement effort.

39. Reporting the Results Description of goals
Description of outcomes explored and Club’s goals and strategies
Description of survey instruments, sample process and approach to administration
Description of the sample and comparison to overall characteristics of Club members
Description of results
Uses of results

40. Who Is Interested Tool Kit Results
At your tables, brainstorm and list “What’s In It” for your assigned “interested party.”
Boards
CPO’s
Staff
Funders
B&GCA (RSD’s, Others?)
Others?

41. Wrap Up What was learned? How might you use it?
What will each of you do:
Write what you will do:
Within 30 days
Within 90 days
Share one of the things you will do.

42. Questions

43. (401) 253-5309 Ja.reinholt@cox.net
Jack Reinholt
(401)