Using Data to Measure and Report Program Impact Anne Basham, MFA MEMconsultants

Agenda  Introductions  Define Learning Objectives and Key Concepts  Apply Knowledge to Your Work  Conclusions  Questions?

Introductions  About Anne  Program Evaluation Consultant,  Organizations serving youth and communities  Collect data to determine what is working well about a program and to identify opportunities for improvement

Learning Objectives  become familiar with the concepts of Performance Measurement and Theory of Change ;  understand how data can be used to measure performance at your specific site;  clarify barriers to collecting quality data and strategies to overcome these barriers;  learn to interpret and report data once it has been collected;  share findings with multiple stakeholders; and  identify next steps in the continuous quality improvement cycle

Performance Measurement  Performance measurement is the process by which businesses, governments and other organizations establish criteria for determining the quality of their activities, based on organizational goals.  Performance measure: A value or characteristic that measures progress toward goals, and also used to improve progress, reduce risks, or improve cost-effectiveness.

Theory of Change  Theory of Change defines all building blocks required to bring about a given long-term goal. 1  Includes assumptions, strategies and predicted outcomes  Customized to reflect the goals and needs of an organization  AKA: pathway of change, blueprint, logic model, outcome map, “so that”chain 1 as defined at

Logic Models Resources/ Inputs ActivitiesOutputs Outcomes (short term, intermediate, long term)

Logic Model Example: WSC Academic Improvement Programs Resources: AmeriCorps Members, Funding, Program Site Structure Activities: Recruit youth, Offer program Output: 4,500 children complete participation in K-12 education programs Outcome: 70% of youth that complete program demonstrate improved academic performance.

Logic Model Example: WSC Environmental Stewardship Programs Resources: AmeriCorps Members, Funding, Tools, Land Permit Activities: Offer training, Organize volunteer opportunities Output: 3,000 individuals participate in environmental awareness and/or energy efficiency education Outcome: 70% of individuals that participate demonstrate improved knowledge or awareness.

Logic Models Resources/ Inputs ActivitiesOutputs Outcomes (short term, intermediate, long term)

“So That” chains… Strategy An organization offers this opportunity or activity; So that… This short term outcome or change can occur; So that… This intermediate outcome or change can occur; So that…This long term outcome or change can occur; So that…This organizational goal is met.

“So That” Chain Example: WSC Healthy Futures Programs Strategy Educate the public about healthy behaviors through outreach and activities So that… 2,500 individuals participate in health education activities led or supported by AmeriCorps Members So that… Individuals who participate in a health education opportunity indicate an increase in knowledge; or increased awareness; or changes in healthy behavior. End Goal Improved access to health care; Increased physical activity and improved nutrition for youth; Seniors’ increased ability to remain in their homes.

“So That” Chain Example WSC Veterans & Military Families Programs Strategy Offer volunteer opportunities and events for veterans and their families. So that… 500 veterans will be engaged in AmeriCorps volunteer projects, activities and events led or supported by AmeriCorps members So that… Veterans and their families become more aware of opportunities to be involved in their communities. So that… Veterans will indicate in increased interest and/or involvement in volunteer activities in the future. End Goal Improved quality of life for veterans Strong military families

ProgramOutcome/Goal Academic Achievement Students demonstrate improved academic performance Disaster Preparedness Individuals are able to respond in the event of a disaster Environmental Stewardship Individuals increase their knowledge and awareness of energy efficient and environmentally conscious practices Healthy Futures Individuals increase their knowledge/awareness of and/or participation in healthy behaviors School Readiness Students demonstrate gains in school readiness in terms of social/emotional development Veterans Veterans indicate increased interest and/or involvement in volunteer activities Indicators

Measuring Performance - Indicators  Indicators are the measurable evidence that demonstrate progress towards your long term goal.  To restate, they are measurable.  Examples:  Number of people participating in a program, event or activity  Number of economically disadvantaged individuals  Increased interest or prediction of involvement  Increased knowledge and/or awareness  Increased confidence to act  Demonstrated gains in school readiness or academic performance

Data Collection Methods: Tracking Indicators QuantitativeQualitative Program Records (attendance) Structured Assessment (test scores) Focus Groups Document/ Literature Review Surveys Interviews Observations

IndicatorData Collection Method Number of people participating in a program, event or activity Program Records/Tracking Log Number of economically disadvantaged individuals Program Records compared to census data, eligibility for benefits (FRL, TANF, Section 8, SCHIP, etc.) Increased interest or prediction of involvement Post Survey Increased knowledge and/or awareness Post Survey Increased confidence to actPost Survey Demonstrated gains in school readiness or academic performance Standardized Assessment Tools Pre and Post Surveys

Common Data Collection Challenges  Time  Working with others  Measurement mismatch  Access  Sample size  Comprehension  “Bad” data  Poorly designed tools  Measuring prevention  What else?

Data Analysis  Quantitative  Totals  Averages  Percentages  Pre/Post Comparison  Qualitative  Emergent  Top Down

Research-based strategies  Don’t reinvent the wheel – learn from past successes – “best practices”  Acknowledge you aren’t doing research or studies that are genearlizable  You are collecting information about your program  Build off of the research of others

Data Reporting  Other Staff or Volunteers  Program Participants  Board Members  Funders  Washington Service Corps  Who else?

Continuous Quality Improvement clarifyplandevelopcollectanalyzecommunicateuse

Learning Objectives  become familiar with the concepts of Performance Measurement and Theory of Change ;  understand how data can be used to measure performance at your specific site;  clarify barriers to collecting quality data and strategies to overcome these barriers;  learn to interpret and report data once it has been collected;  share findings with multiple stakeholders; and  identify next steps in the continuous quality improvement cycle

Questions?  Everyone will be unmuted so you can ask your questions.  You may also type questions into the chat box.

Thank you!  Slides will be available for your use  Corporation for National and Community Service (CNCS) Performance Measurement website:  gram-financial-and-grant- management/performance-measurement gram-financial-and-grant- management/performance-measurement  Contact Anne with questions: