1. Program Analysis Overview
Draft
Program Analysis Overview
Leading for Impact

2. Contents Overview and context Defining outputs and outcomes
3_84
Contents
Overview and context
Defining outputs and outcomes
Gathering data: sources and useful tips
Analyzing and synthesizing data
Revisiting your hypothesis and theory of change

3. How does this fit into the Strategic Clarity Project?
7_84
How does this fit into the Strategic Clarity Project?
Program analysis is one type of analysis you can do in the second step of the Strategic Clarity Project.
STRATEGIC CLARITY PROJECT OVERVIEW
Step
Description
Timeline
Draft theory of change and prioritize questions
Develop draft TOC
Identify and prioritize questions to test
Month 1
Conduct analysis
Answer prioritized questions using internal and external data as appropriate
Month 2-3
Refine theory of change
Make changes to TOC based on analysis
Codify learning agenda
Month 4
Prepare to implement
Develop 6-12 month implementation plan
Create presentation summarizing project outcomes
Month 4-5

4. What can I expect to do in a program analysis?
23_84
What can I expect to do in a program analysis?
PROGRAM ANALYSIS OVERVIEW 1 2 3 4 5
Scope Programs
Define Metrics
Gather Data
Analyze Results
Refine TOC
STAGE
Define outputs and intended impact
Identify metrics and indicators you need
Collect data from different sources
Analyze and synthesize data for key findings
Adjust your TOC based on your findings
GOAL
Using your theory of change draft, identify the outputs and outcomes of each program you want to assess
For each of your programs, figure out: 1) what data, metrics, and indicators best represent your intended impact, and 2) where you will gather that data
Explore existing internal program data
Conduct interviews and/or surveys with clients and/or staff
Synthesize data by program
Find key themes across data sources
Develop possible implications about program analysis
Make revisions to your TOC based on findings
Add more specificity or definition
Identify possible program adjustments or improvements
TASKS
WORK
PRODUCT
Worksheet 1
Worksheet 2
Data collection summary files for each source
Worksheet 3; data analysis slides
Revised TOC
TIMELINE
1 week
1 week
3 weeks
3 weeks
1 week

5. Why conduct a program analysis?
5_84
Why conduct a program analysis?
We believe this… …gets us to this.
But:
How can you demonstrate that to others?
Are some pockets of clients falling through the cracks?
Are there gaps in your programs or services?
This program analysis module can help you
answer these questions.
Theory of Change
Intended Impact

6. 4_84 11_84
Other organizations have used program analysis to strengthen their program offerings, delivery and impact
LIFT Boston looked at who they were serving – and how well they were serving them – to better understand where they wanted to focus their work.
LIFT Boston used data on client attendance and services accessed to clarify their target segment: those able to attend at least four sessions and in need of at least two services
The organization also used qualitative case notes to identify an important factor for client success – consistent and stable relationships between clients and their volunteer advocates
The Latin American Youth Center used performance data from its parenting program to inform some small shifts that reversed the negative outcomes they were achieving.
The initial program data showed that LAYC was moving attitudes about domestic violence in the opposite direction than intended
With some help from outside experts, LAYC adjusted its program delivery to better achieve its goals for educating parents about domestic violence

7. What are we actually doing? What is our intended impact?
4_84
You can assess your programs by looking at the how and the what of your Theory of Change
Outputs
Outcomes
Definition
Examples
What are we actually doing?
What is our intended impact?
# of students in literacy classes
# of clients who completed a job placement program
# of financial education classes offered
Improved reading proficiency
Increased income; decreased unemployment rates
Increased savings; improved financial literacy skills
The indicators or interim steps that lead you to or drive outcomes
Products of your organization's efforts (i.e. activities, services)
Your target outcomes
Change occurring during/after your program, compared to a baseline
Breakthrough results with long- lasting impact

8. Expect your analysis to feel more like this…
6_89 4_84
Triangulate data sources to piece together as much of the picture as you can
Expect your analysis to feel more like this…
…than like this
You probably won’t have “perfect” data to answer a question
Triangulation is using multiple data sources that each address a portion of the question to arrive at a well-supported answer
Even if you can’t assemble all the puzzle pieces, you should be able to good sense of what the picture looks like
Combining multiple sources can also check the biases of each and lead you to a more nuanced understanding in some areas

9. What sources of data can you look at?
30_84 29_84
What sources of data can you look at?
You probably won’t have good data from all of the below sources. Rather, these are ideas for where you can find information to creatively piece together a picture of how your programs are working.
Existing program output and outcome data
Interviews with staff and clients (current and former)
Survey of staff and clients (current and former)
Utilizes existing documentation; constrained to what already exists
Can provide full range and depth of experience; develops relationships; usually results in better response rate; selection is often not random; can be costly, time-consuming; interviewer can bias client
Requires relatively little time, cost, and effort; can be anonymous; randomness is important yet difficult to achieve; represents breadth but not depth
Data source type
Pros and cons of this source
MOST
LIKELY
SOURCES
OF DATA
External data and benchmarks
Can provide benchmark or baseline data to which you can compare client outcomes; can be difficult and time-consuming to find relevant public data
Note: Depending on your programs and on what data you need, external data may not be applicable, and is often especially time-consuming and difficult to source

10. 21_84 22_84
Once you’ve gathered data, compare results from different sources and against your expectations
Look at the negative
What groups of clients are struggling or not meeting your target outcomes?
Are there common characteristics or behaviors among those who do not succeed?
Would your impact have more magnitude or meaning if you better reached these groups of clients?
Reality vs. expectations
How does the actual implementation of your programs differ from the intended program design? Do you think this affects the program’s efficacy?
Are there any unexpected outcomes of your program(s)?
Were there any surprises in the data?
Powerful aspects of the program
What change do clients see during and after your program?
Are any program aspects commonly identified as particularly beneficial or not useful?
Patterns and key themes
Are there similar comments across data sources?
Can you group key takeaways or main ideas into common themes?
Discrepancies
Are there discrepancies across data sources? How could you explain this?
Are there differences in results across locations or populations?
Consider reviewing results in a group session to gain perspectives and input from different stakeholders

11. Use your program analysis to refine the who, what, and how of your theory of change and intended impact
Redefine target outcomes
Consider adding numerical targets for easily track-able outcomes. Be specific about the gain you want to see, especially for less tangible impacts, such as skills or confidence.
Consider adjusting or adding program offerings if needed
Confirm that you are including all the specific aspects or offerings of the program that are important in achieving your target outcomes. Eliminate aspects or programs that do not contribute to your intended impact.
Specify how programs should be delivered
Include specifications and guidelines on how programs should be implemented and delivered. Defining your program design and requirements brings clarity and accountability to those who scale the program.
Confirm who you want to serve
Make sure you accurately define the clients you want to and can serve. Your programs should ultimately align with your target clients, and vice versa.