Back to the Future: Evaluation and Measurement of Learner Outcomes in Financial Education National Endowment for Financial Education® (NEFE®) August 2-4, 2010 Quarter Century of Research in Personal Finance Colloquium Lois A. Vitt, Sharon Danes, Jeanne Hogarth, Barbara O’Neill, John Tatom, William Walstad

National Endowment for Financial Education Introduction We explore efforts—past, present, and underway  That expand financial educators’ capacity to perform better program evaluations, and  Encourage evaluation research from both  financial education researchers, and  policy-oriented research organizations. 2

National Endowment for Financial Education …Introduction First, distinguish between:  Program evaluation of financial education in the field performed by  by teachers (and/or other program stakeholders)  Evaluation research By financial educator-researchers, and By policy researchers 3

National Endowment for Financial Education Four Sections of Theme 3 Paper I. Better program evaluation:  Traditions and trends. II. Limitations and obstacles to good evaluation. III. Micro and macro measures that assess the impact of financial education, and IV. Learner outcomes:  What to measure and why. 4

National Endowment for Financial Education Program Evaluation - Traditions  Learner satisfaction with teaching, topics, materials, convenience, etc.  Adult education courses must be appealing, interesting, convenient and pleasant.  Adult learners are “customers”, want to know “what’s in it for me?”  Educators want to improve their evaluations, but need to get students in the door. 5

National Endowment for Financial Education Program Evaluations – Trends  Calls for better program evaluation by national and international stakeholders.  Theoretical frameworks that test the efficiency of programs being evaluated.  Written guides to good evaluation in preparation.  Standardized measures for a literacy “score” being suggested.  Program evaluation teacher training is on the rise. 6

National Endowment for Financial Education II. Limitations and Obstacles to Good Evaluations  Program or researcher bias.  “One size” seldom fits all problems or solutions.  Omitted variables (e.g. antecedent, intervening variables).  Pre and post testing issues vs. post-then-pre-tests.  Other statistical analysis issues.  Threats to internal validity – reliable causal relationships.  Threats to external validity – generalizability. 7

National Endowment for Financial Education Long-standing Controversies in Evaluation  Research methods:  Quantitative vs. Qualitative  Subjective vs. objective evidence (self-reporting)  Intentions vs. measurable action (behavior)  Program vs. policy research philosophy  Utilization approach to evaluation—how will the research be used? Don’t put the evaluation on a shelf! 8

National Endowment for Financial Education Quantitative vs. Qualitative Methods?  Quantitative  Randomized studies  Experimental research design  Survey research with controls  Qualitative  In-depth interviews of individuals  Focus groups  Observations  Mixed methods—adapt for information needs  Provides both explanation and understanding 9

National Endowment for Financial Education III. Micro and Macro Measures  Micro-level orientations  Individuals and families in particular situations. Making ends meet. Planning ahead. Managing financial products. Financial decision-making.  Macro-level orientations  Aggregating program results.  Financial well-being of populations.  Stable, growing economy. 10

National Endowment for Financial Education IV. Learner Outcomes: what to Measure and Why  Triangulation (multiple methods)  Money Talks: Building a Brighter Future (for ex-offenders) Pre/post tests Assessment of actions taken by learners Videotaped interviews of learners Face-to-face interviews with probation personnel Tracking of “compliance” (recidivism issue) 3-month follow-up of knowledge, attitudes, behaviors Pre-and post-test credit scores – up by at least 10% Quiz results vs. control group (67.8% vs. 85%) 11

National Endowment for Financial Education Replication & Large-Scale Impacts  To reach large populations, design programs to be easily  Taken to scale  Evaluated and Replicated  MONEY 2000  5-year Cooperative Extension Program (Rutgers)  $20 million debt-reduction and savings resulted  Inspiration, but no mechanism for replication 12

National Endowment for Financial Education What we know from past evaluations…  Aspiration indicates readiness for change.  Intentions are predictors of subsequent behavior.  Self-efficacy is having the confidence in one’s ability to deal with situation without being overwhelmed.  Incremental progress is important to recognize.  Financial learning is not always linear. 13

National Endowment for Financial Education Conclusions  Financial education programs are highly varied:  Topics and issues covered  Breadth and depth of education offered  Nature of audience  Challenge: evaluate and compare findings across programs  Proposed solutions  Complimentary program evaluation & evaluation research  Show outcomes occurring at both micro & macro levels  Focus on both short and long term outcomes  Understand the challenges of changing behavior 14

National Endowment for Financial Education Evaluation and Measures of Learner Outcomes Thank You Questions and Discussion Lois A. Vitt, Sharon Danes, Jeanne Hogarth, Barbara O’Neill, John Tatom, Bill Walstad 15