Measuring and Using Outcomes for Volunteer Programs 2014 Greater Richmond Association for Volunteer Administration Conference July 10, 2014

Measuring and Using Outcomes for Volunteer Programs Topics Outcome measurement overview: What, why, how Identifying outcomes Identifying your volunteer program’s outcomes Identifying indicators, data sources, data collection methods Identifying indicators, data sources & data collection methods for your volunteer program Using outcome data to learn and improve Parting advice Resources Measuring and Using Outcomes for Volunteer Programs

What Measuring and Using Outcomes for Volunteer Programs

Program elements OUTCOMES INPUTS ACTIVITIES OUTPUTS Benefits or changes for participants during or after program activities e.g. new knowledge increased skills changed attitudes or values modified behavior improved condition altered status INPUTS Resources dedicated to or consumed by the program e.g. money staff & staff time volunteers & volunteer time facilities equipment & supplies ACTIVITIES What the program does with inputs to fulfill its mission e.g. feed and shelter homeless families provide job training educate teachers about signs of child abuse counsel pregnant women provide mentors for youth OUTPUTS The direct products of program activities e.g. number of classes taught number of counseling sessions conducted number of educational materials distributed number of hours of service delivered number of participants served Source: Measuring Program Outcomes: A Practical Approach. United Way of America, 1996 Measuring and Using Outcomes for Volunteer Programs

how well the fledgling flies not how many worms the bird feeds its young, but how well the fledgling flies Outcome: Measuring and Using Outcomes for Volunteer Programs

Outcome vs. indicator vs. target Measuring and Using Outcomes for Volunteer Programs Outcome vs. indicator vs. target Outcome: Benefit or change for intended beneficiaries or target group e.g., Parents will use more effective discipline methods Outcome indicator: The specific statistic used to show how fully the program is achieving the outcome e.g., The number and percent of parents who, after completing the program, explain rules and consequences and apply consequences each time a rule is broken Outcome target: Numerical objective for a program’s level of achievement on an outcome e.g., 75% of parents will explain rules and consequences and apply consequences each time a rule is broken Measuring and Using Outcomes for Volunteer Programs

Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives A note about targets Don’t set targets for outcome achievement until you have collected 1 or 2 rounds outcome data Until you know your current level of achievement you won’t know what target is reasonable; you’ll have to guess at a target Whether you meet, exceed, or fall below the target won’t reflect your effectiveness; it will reflect how well you guessed Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 7

Outcome measurement is NOT. . . Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Outcome measurement is NOT. . . Experimental research Program evaluation Performance measurement A replacement for tracking inputs, activities, and outputs Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 8

Outcome findings will NOT. . . Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Outcome findings will NOT. . . Show whether what you are measuring is the right outcome for your program’s activities Prove statistically that your program caused the outcome Explain why this level of outcome was achieved By themselves, tell how to improve the outcome Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 9

So… Measuring and Using Outcomes for Volunteer Programs

Why Measuring and Using Outcomes for Volunteer Programs

  Increase effectiveness Communicate value Why measure outcomes? Measuring and Using Outcomes for Volunteer Programs

Internal uses of program outcome findings Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Internal uses of program outcome findings Provide direction for staff Identify training needs Improve measurement and/or program Support long-range & annual planning Guide budgets & justify resource allocations Suggest outcome targets Focus board members’ attention on programmatic issues Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 13

External uses of program outcome findings Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives External uses of program outcome findings Recruit talented staff and volunteers Promote the program to potential participants and referral sources Identify partners for collaboration Enhance the program’s public image Retain and increase funding Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 14

Benefits of program outcome measurement Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Benefits of program outcome measurement Executives of 298 programs agreed or strongly agreed – outcome measurement had helped their program: Focus staff on shared goals (88%) Communicate results to stakeholders (88%) Clarify program purpose (86%) Identify effective practices (84%) Compete for resources (83%) Enhance record keeping (80%) Improve service delivery (76%) Nearly all (89%) would recommend outcome measurement to the director of a similar program Source: Agency Experiences with Outcome Measurement. United Way of America, 2000 Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 15

How Measuring and Using Outcomes for Volunteer Programs

Outcome measurement: 8 steps Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Outcome measurement: 8 steps 7. Improve your system 6. Analyze and report your findings 5. Try out your outcome measurement system 4. Prepare to collect data on your indicators 8. Use your findings 1. Get ready 2. Choose the outcomes you want to measure 3. Specify indicators for your outcomes Source: Measuring Program Outcomes: A Practical Approach. United Way of America, 1996 Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 17

In good outcome measurement … Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives In good outcome measurement … Outcomes are appropriate for the program Indicators provide useful data on outcome achievement Methods – data sources and data collection procedures – are sound Tools measure what they’re supposed to and are appropriate for the population Analysis shows level of achievement for different segments of the target audience and different program characteristics Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 18

Measuring and Using Outcomes for Volunteer Programs Topics Outcome measurement overview: What, why, how Identifying outcomes Identifying your volunteer program’s outcomes Identifying indicators, data sources, data collection methods Identifying indicators, data sources & data collection methods for your volunteer program Using outcome data to learn and improve Parting advice Resources Measuring and Using Outcomes for Volunteer Programs

“Outcome” The specific results(s) sought for a specific audience Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives “Outcome” The specific results(s) sought for a specific audience (clients, participants, visitors, patrons, volunteers, etc.) Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 20

  Identifying outcomes Not “What will we measure?” First question: Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Identifying outcomes First question:  Not “What will we measure?”  Instead “What result do we want our activities to achieve for whom?” Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 21

Volunteer programs: Results for two audiences Results for clients/ visitors/patrons e.g.: Clients receive desired/ needed information Patrons recommend the setting to others Results for volunteers e.g.: Volunteers provide accurate information Volunteers make a long-term commitment Volunteers’ activities Volunteer program administration activities Measuring and Using Outcomes for Volunteer Programs

Measuring and Using Outcomes for Volunteer Programs Topics Outcome measurement overview: What, why, how Identifying outcomes Identifying your volunteer program’s outcomes Identifying indicators, data sources, data collection methods Identifying indicators, data sources & data collection methods for your volunteer program Using outcome data to learn and improve Parting advice Resources Measuring and Using Outcomes for Volunteer Programs

Identifying your volunteer program’s outcomes Part 1 Results for clients/ visitors/patrons e.g.: Clients receive desired/ needed information Patrons recommend the setting to others Volunteers’ activities Worksheet A: Activities of your volunteers Measuring and Using Outcomes for Volunteer Programs

Identifying your volunteer program’s outcomes Part 1 Results for clients/ visitors/patrons e.g.: Clients receive desired/ needed information Patrons recommend the setting to others Worksheet B Section II: Outcomes you want your volunteers to create Volunteers’ activities Measuring and Using Outcomes for Volunteer Programs

Activities  Outcomes: An “If…then” chain THEN … this change THEN … this change THEN … this change IF … these activities Measuring and Using Outcomes for Volunteer Programs

A simple “If…then” chain THEN … this change in condition/status THEN … this change in behavior THEN … this change in knowledge IF … these activities Measuring and Using Outcomes for Volunteer Programs

Identifying your volunteer program’s outcomes Part 1 Results for clients/ visitors/patrons e.g.: Clients receive desired/ needed information Patrons recommend the setting to others Worksheet C: If … then chain linking volunteers’ activities and outcomes for target audience Volunteers’ activities Measuring and Using Outcomes for Volunteer Programs

Testing your “If … then” chain Are the outcomes really outcomes? Do they describe changes in or benefits for clients/ participants/beneficiaries, rather than something about the program? Is the longest-term outcome meaningful for program clients/participants/beneficiaries? Will the change or benefit make a real and lasting difference for them? Is the logic logical? Are the activities sufficient to produce the intended outcomes for these clients/participants/beneficiaries? Measuring and Using Outcomes for Volunteer Programs

Are the activities sufficient to achieve the outcomes? At-risk teens graduate from high school. At-risk teens meet district grade requirements. Outcomes At-risk teens complete homework regularly. At-risk teens know homework is important. Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of education, may help teens with homework. Activities Measuring and Using Outcomes for Volunteer Programs

Identifying your volunteer program’s outcomes Part 1 Worksheet D: Testing your If … then chain Outcome A Outcome C Outcome B Activities 1 & 4 3 & 5 Activity 2 Measuring and Using Outcomes for Volunteer Programs

Identifying your volunteer program’s outcomes Part 2 Results for volunteers e.g.: Volunteers provide accurate information Volunteers make a long-term commitment Worksheet E: Impact you want to have on your volunteers Volunteer program administration activities Measuring and Using Outcomes for Volunteer Programs

Identifying your volunteer program’s outcomes Part 2 Results for volunteers e.g.: Volunteers provide accurate information Volunteers make a long-term commitment Worksheet F: If … then chain linking volunteer program administration activities and outcomes for volunteers Volunteer program administration activities Measuring and Using Outcomes for Volunteer Programs

Identifying your volunteer program’s outcomes Part 2 Worksheet G: Testing your If … then chain Outcome A Outcome C Outcome B Activities 1 & 4 3 & 5 Activity 2 Measuring and Using Outcomes for Volunteer Programs

Identifying your volunteer program’s outcomes Requires time, thought, discussion Benefits from input by volunteers, clients/visitors/patrons, board members, and engaged donors as well as staff Evolves over time Provides content for your program’s “logic model” Is worth the effort ACTIVITIES OUTPUTS OUTCOMES INPUTS Measuring and Using Outcomes for Volunteer Programs

Programs report that defining outcomes and creating logic models … Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Programs report that defining outcomes and creating logic models … Stimulates discussion of the program’s purpose Creates a shared understanding of how activities are intended to link to results Often prompts adjustments in activities and intended results Helps communicate the program’s intent to stakeholders Makes sure useful data is collected – and unnecessary data is not Guides review of data to identify improvement opportunities Often leads to improved outcome performance because now everyone understands the results the program is aiming for Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 36

Measuring and Using Outcomes for Volunteer Programs Topics Outcome measurement overview: What, why, how Identifying outcomes Identifying your volunteer program’s outcomes Identifying indicators, data sources, data collection methods Identifying indicators, data sources & data collection methods for your volunteer program Using outcome data to learn and improve Parting advice Resources Measuring and Using Outcomes for Volunteer Programs

Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives “Indicator” Specific statistic used to show how fully the outcome is being achieved Often the number and percentage of clients/patrons/ volunteers who, e.g.: attain specific knowledge demonstrate a needed skill display a certain behavior achieve a desired status Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 38

Examples of indicators Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Examples of indicators Number and percentage of volunteers in the rotating gallery who can answer 25 FAQs about the current exhibit correctly Number and percentage of activity volunteers in the senior center who use techniques for engaging withdrawn seniors Number and percentage of fund-raising volunteers who renew their commitment for 1 year of service Number and percentage of children in the activity program who read at least 5 books with their volunteer reading buddy Number and percentage of patrons in the volunteer-staffed gift shop who make or add to a purchase Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 39

Identifying possible indicators Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Identifying possible indicators Questions to ask to generate options: How will we know if the outcome is achieved? What will we see? What can we count or measure? If you’re stuck, ask: How would we know if the intended result is not achieved? What would we see? If we were successful in one instance but not another, how would we know? What would we see in one instance but not the other? Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 40

Testing a possible indicator Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Testing a possible indicator Does it capture the central point of the outcome? (e.g., if the central point is behavior, it tracks behavior – not knowledge) Is it specific enough that everyone interprets it the same way? Is it unlikely to be influenced by factors others than the outcome? Do key stakeholders accept it as a measure of the outcome? Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 41

Possible sources of program outcome data Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Possible sources of program outcome data Case/client records Documents Data files The volunteers/clients/visitors/patrons themselves Others who know the volunteers/clients/visitors/patrons Other individuals Physical environment Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 42

Possible methods of collecting program outcome data Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Possible methods of collecting program outcome data Record review Document review Data file review Questionnaire Interview Observation Testing Mechanical measurement Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 43

Measuring and Using Outcomes for Volunteer Programs Topics Outcome measurement overview: What, why, how Identifying outcomes Identifying your volunteer program’s outcomes Identifying indicators, data sources, data collection methods Identifying indicators, data sources & data collection methods for your volunteer program Using outcome data to learn and improve Parting advice Resources Measuring and Using Outcomes for Volunteer Programs

Data Collection Method/Tool Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Identifying indicators, data sources & data collection methods for your volunteer program Worksheet H: Indicators, data sources, data collection methods for your outcomes Outcome Indicator(s) Data Source(s) Data Collection Method/Tool Outcome A Outcome B Outcome C Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 45

Measuring and Using Outcomes for Volunteer Programs Topics Outcome measurement overview: What, why, how Identifying outcomes Identifying your volunteer program’s outcomes Identifying indicators, data sources, data collection methods Identifying indicators, data sources & data collection methods for your volunteer program Using outcome data to learn and improve Parting advice Resources Measuring and Using Outcomes for Volunteer Programs

Using outcome data to learn and improve Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Using outcome data to learn and improve Calculate overall outcomes — # and % of target audience achieving the outcome e.g.: Total number served 1,000 Number achieving outcome 535 Percent achieving outcome 53.5% Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 47

Using outcome data to learn and improve Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Using outcome data to learn and improve Calculate overall outcomes Break out results by: level of outcome relevant characteristics of target audience relevant program characteristics combinations to see where you are having more and less success Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 48

Outcomes by level of outcome Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of education, encourage school attendance, occasionally help with homework. At-risk teens achieve passing grades. At-risk teens earn better grades. At-risk teens graduate from high school. At-risk teens meet district attendance requirements. At-risk teens complete homework regularly. At-risk teens attend school regularly. 60% 85% Measuring and Using Outcomes for Volunteer Programs

Outcomes by client characteristic: Parent attended teacher conference Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of education, encourage school attendance, occasionally help with homework. At-risk teens achieve passing grades. At-risk teens earn better grades. At-risk teens graduate from high school. At-risk teens meet district attendance requirements. At-risk teens complete homework regularly. At-risk teens attend school regularly. Yes 75% No 45% Measuring and Using Outcomes for Volunteer Programs

Outcomes by program characteristic: Mentors meet monthly with mentee’s teacher Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of education, encourage school attendance, occasionally help with homework. At-risk teens achieve passing grades. At-risk teens earn better grades. At-risk teens graduate from high school. At-risk teens meet district attendance requirements. At-risk teens complete homework regularly. At-risk teens attend school regularly. Yes 55% No 40% Measuring and Using Outcomes for Volunteer Programs

Outcomes by client & program characteristics: Boys only, by mentor’s sex Mentor is a Woman Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of education, encourage school attendance, occasionally help with homework. At-risk teens achieve passing grades. At-risk teens earn better grades. At-risk teens graduate from high school. At-risk teens meet district attendance requirements. At-risk teens complete homework regularly. At-risk teens attend school regularly. Mentor is a Man 45% 70% 80% 80% Measuring and Using Outcomes for Volunteer Programs

Using outcome data to learn and improve Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Using outcome data to learn and improve Calculate overall outcomes Break out results by target audience and program characteristics With caution, compare current outcomes with: previous outcomes outcome targets outcomes of similar programs relevant standards Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 53

Using outcome data to learn and improve Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Using outcome data to learn and improve Calculate overall outcomes Break out results by target audience and program characteristics With caution, compare current outcomes with other points of reference Analyze findings Review and discuss on a regular basis Seek input from program volunteers, clients/patrons and others Recommend improvements Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 54

In analyzing findings for improvement opportunities, look for . . . Logic (theory) problems: Assumptions about the relationship among activities and outcomes are questionable (e.g., will an hour of contact a week really change behavior?) Program problems: The logic is sound, but the program wasn’t implemented as planned (e.g., 55% of teens missed more than half of the sessions) Measurement problems: An issue with how the data was collected means the numbers are questionable (e.g., many more responses from older teens than from younger ones may skew results) Measuring and Using Outcomes for Volunteer Programs

Using outcome data to learn and improve Connecting the Dots: Global Standards, Key Business Practices, Collective Impact, Improving Lives Using outcome data to learn and improve Calculate overall outcomes Break out results by target audience and program characteristics With caution, compare current outcomes with other points of reference Analyze findings Implement improvements and track results Celebrate learning! Repeat Measuring and Using Outcomes for Volunteer Programs United Way Worldwide, 2012 56

Measuring and Using Outcomes for Volunteer Programs Topics Outcome measurement overview: What, why, how Identifying outcomes Identifying your volunteer program’s outcomes Identifying indicators, data sources, data collection methods Identifying indicators, data sources & data collection methods for your volunteer program Using outcome data to learn and improve Parting advice Resources Measuring and Using Outcomes for Volunteer Programs

Your first priority in outcome measurement Measuring and Using Outcomes for Volunteer Programs Your first priority in outcome measurement Make it work for YOU Measuring and Using Outcomes for Volunteer Programs

Don’t kill what you won’t eat Measuring and Using Outcomes for Volunteer Programs

Meg Plantz Helping organizations improve lives through collaborative community change meg.plantz@verizon.net 703-960-9680 Measuring and Using Outcomes for Volunteer Programs