Evaluating Financial Education Programs: A framework for measuring results Ellen Taylor-Powell, Ph.D. Evaluation Specialist American Savings Education Council April 18, 2007 Washington, D.C. University of Wisconsin-Extension Cooperative Extension

The state of affairs … Evidence tells of the situation we face: low saving rates; high debt; high bankruptcy rates, shift in policies, sophisticated financial markets and products… Partners have responded in force to build financial literacy with a myriad of investments and activities The scope and size of the financial educational effort is impressive

Our purpose today… What difference is all this effort making? Does financial education work? How do we know? Examine the logic model as a framework for planning and evaluation Explore evaluation challenges and options Discuss how we can make evaluation meaningful

Accountability age No longer good enough to report numbers of services delivered; number of participants reached Need documentation of changes: improvements for individuals, families, communities Need to be able to answer: So what?

“I think you should be more explicit here in Step Two.”

What is a Logic Model? A graphic depiction of a program that shows the relationship among activities and intended results – Linkage between and among components: Program “theory” or “program action” – Intended results: what the program is to accomplish Serves as a tool for integrating planning and evaluation – Helps with both planning and evaluation Is not an evaluation model, method or reality Is widely used: nonprofits, public sector, foundations, international agencies, evaluation community Other names: theory of change, program action, program theory, causal road map

Family Members Budget Car Camping Equipment Drive to state park Set up camp Cook, play, talk, laugh, hike Family members learn about each other; family bonds; family has a good time A Family Vacation

A youth financial literacy program Partners invest resources A high school financial planning program – 7 unit curriculum - is developed and delivered in high schools Teens gain knowledge and skills in money management Teens establish sound financial habits Teens make better decisions about the use of money

Comprehensive Cancer Control

Common graphic of logic model the “four box approach INPUTS OUTPUTS ACTIVITIES OUTCOMES

A bit more detail INPUTSOUTPUTSOUTCOMES Program investments ActivitiesParticipationShortMedium What we invest What we do Who we reach What results SO WHAT?? Long- term

Fully detailed

OUTPUTS What we do Who we reach ACTIVITIES Assess needs and assets Design and deliver curriculum Conduct workshops; teach Provide one-on-one counseling Produce products - Exhibits - Printed materials - Electronic materials Populate/maintain web sites Hold Conferences Work with the media Facilitate policy dialogue Partner – collaborate Advocate PARTICIPATION Underserved populations Low income consumers Youth: elementary, high school, post grad; college students Homebuyers/homeowners Military personnel Teachers/educators Immigrant populations Mandated audiences Financial professionals Reactions – Satisfaction

OUTCOMES What results for individuals, families, communities..… SHORT Learning Changes in Awareness Knowledge Attitudes Skills Opinion Aspirations Motivation Behavioral intent MEDIUM Action Changes in Behavior pay bills on time  savings  assets –debt achieve personal goals spend wisely track spending and income Decision making Policies Social action LONG-TERM Conditions Changes in Human well-being stability security Economic Community –civic activity C H A I N OF O U T C O M E S

Reactions Learning Actions Social-economic- environmental improvements Hierarchy of effects Source: Bennett and Rockwell, 1995, Targeting Outcomes of Programs Number and characteristics of people reached; frequency and intensity of contact Degree of satisfaction with program; level of interest; feelings toward activities, educational methods Changes in knowledge, attitudes, skills, aspirations Changes in behaviors and practices Participation

Staff Money Partners Develop parent ed curriculum Deliver series of 8 interactive sessions Parents increase knowledge of child dev Parents better understanding their own parenting style Parents use effective parenting practices Improved child- parent relations Research INPUTS OUTPUTS OUTCOMES Facilitate support groups Parents gain skills in effective parenting practices Simple logic model Parents identify appropriate actions to take Parents of 3-10 year olds SITUATION: During a county needs assessment, majority of parents reported that they were having difficulty parenting; were often uncertain about what to do and felt stressed as a result. Parents increase knowledge of community resources

Staff Time Money Volunteer Partners Materials Supplies Research- best practices Develop, deliver parent ed curriculum Do home visits;Outreach Parents/ - caregivers: Single parent Couples Divorced Teen parent Fathers Low-income Homeless Mandated Childcare providers: directors, staff Demonstrate positive parenting practices; improved family communications Children enter school ready to learn Demonstrated language, communication and social skills Use community supports/resources Increased abilities to socialize and communicate Implement quality improvement strategies Changes in knowledge, skills intent re. child dev.; parenting style Elaborated logic model - Parent support initiative Increased knowledge of community resources Increased knowledge of quality improvement strategies: developmentally appropriate practice, child assessments, learning environments Early detection of delays and referrals OUTCOMES Improved child-family relati’ships Families connected to community Facilitate parent support groups Provide parent/child Activities: Play group; Toy lending Provide individual services; counseling Train Support Do assessments Children Ages Low-income Special needs Increased ability to advocate for self and families Advocate for self and family Inclusive, appropriate services available Situation

What does a logic model look like? Graphic display of boxes and arrows; vertical or horizontal – Relationships, linkages Any shape possible – Circular, dynamic – Cultural adaptations; storyboards Level of detail – Simple – Complex Multiple models – Multi-level programs – Multi-component programs

EVALUATION: check and verify What do you want to know?How will you know it? PLANNING: start with the end in mind

EVALUATION: check and verify What do you want to know?How will you know it? PLANNING: start with the end in mind Plan backwards

Logic Model and Planning Apply to any level: national plan, statewide plan, individual plan of work, specific project/activity plan Model vs. a more detailed narrative plan or a management plan Focus on outcomes: “start with end in mind”

Integrate evaluation into program planning Think about: What do you (and other key stakeholders) want to know? What data will you want to have? When will it be appropriate to collect data? What evidence will be credible; believable by various consumers? If you told me that people are benefiting, what would I see if I came to visit? Is it your program or something else that is making this difference?

EVALUATION: check and verify What do you want to know? How will you know it?

INPUTSOUTPUTSOUTCOMES Program investments ActivitiesParticipationShortMedium Long- term Indicators : What evidence do you need to answer your questions? Match evaluation questions to program Evaluation questions : What questions do you want to answer? e.g., accomplishments at each step; expected causal links; unintended consequences or chains of events set into motion

Staff Money Partners Parents increase knowledge of child dev Parents better understand their own parenting style Parents use effective parenting practices Improved child- parent relations Research Facilitate support groups Parents gain skills in effective parenting practices Parents identify appropriate actions to take To what extent are relations improved? To what extent did behaviors change? For whom? Why? What else happened? To what extent did knowledge and skills increase? For whom? Why? What else happened? Who/how many attended/did not attend? Did they attend all sessions? Supports groups? Were they satisfied – why/why not? How many sessions were held? Quality of implementation? #, quality of support groups? What amount of $ and time were invested? Example: Evaluation questions and indicators Deliver series of 8 interactive sessions EVALUATION QUESTIONS # Staff $ used # partners # Sessions held Quality criteria INDICATORS #,% attended per session Certificate of completion #,% demonstrating increased knowledge/skills Additional outcomes #,% demonstrating changes Types of changes #,% demonstrating improvements Types of improvements Develop parent ed curriculum Parents of 3-10 year olds

Logic model and evaluation Needs/asset assessment: What are the characteristics, needs, priorities of target population? What are potential barriers/facilitators? What is most appropriate to do? Process evaluation: How is program implemented? Are activities delivered as intended? Fidelity of implementation? Are participants being reached as intended? What are participant reactions? Outcome evaluation: To what extent are desired changes occurring? Goals met? Who is benefiting/not benefiting? How? What seems to work? Not work? What are unintended outcomes? Impact evaluation: To what extent can changes be attributed to the program? What are the net effects? What are final consequences? Is program worth resources it costs?

Key considerations Match evaluation to stage of program’s development Define evaluation purpose Identify specific evaluation questions Select measurable indicators Match design to purpose, audience and resources Attend to cultural relevance

Evaluation challenges Internal Commitment Training Common language and understanding Value Leadership Resources Community context Complex issues Lagged outcomes Attrition Non-response “Noisy” systems Implementation fidelity Varied programming: contexts, content, intensity, audience and implementation

So, what can we do? Internal Build capacity Provide leadership Showcase use Institute operational and structural changes Provide resources Practice, practice, practice Community context Use logic model to show connections Match evaluation to purpose, program, questions Use multiple sources of information Mix methods Conduct formative research

Summing up Logic model links activities to results  integrates planning and evaluation  helps match evaluation to the program effort  identifies points for appropriate data collection Evaluation challenges are many but not insurmountable!  Remember that evaluation equals learning

Resources “We build the road and the road builds us.” -Sri Lankan saying Weiss, Carol. 1998, Evaluation, 2 nd Ed; Prentice-Hall, Inc.