1. Evaluating a Financial Literacy Program: the Case of the Australian MoneyMinded Program Rob Brooks Monash University

2. Presentation Structure  Background / Context  Objectives  Overview of evaluation framework  Evaluation of the MoneyMinded program in the context of the framework  Conclusion

3. Background / Context The promotion of greater levels of financial inclusion is now seen to be a major public policy and corporate initiative across a variety of countries Core elements of programs to promote greater financial inclusion include matched savings programs and financial literacy education

4. Background / Context United States: American Dream Demonstration United Kingdom: Saving Gateway Australia: Saver Plus Need for effective evaluation of programs in their role in improving financial inclusion Fox, Bartholomae and Lee (2005) To date, no single framework to guide the evaluation process

5. Background / Context Fox, Bartholomae and Lee extend the framework of Jacobs (1988) to develop a 5 tiered evaluation framework against which evaluations of financial literacy education programs can be conducted

6. Objectives To explore this evaluation framework in the context of a financial literacy program designed and developed in a different country, specifically the MoneyMinded financial literacy program, offered in Australia Extension of work of Fox, Bartholomae and Lee (2005) Provides information on the generalisability of framework Results will be of interest to a range of countries given comparable initiatives relating to financial inclusion across various countries

7. The Evaluation Framework  5 tiers, based on the work of Jacobs (1988) 1.Pre-implementation 2.Accountability 3.Program clarification 4.Progress towards objectives 5.Program Impact

8. Pre-implementation  Involves a needs assessment on the financial literacy training required by participants Needs identified from more general financial literacy testing Specific needs assessment around program participants

9. Accountability  Involves the collection of data on the education and services provided as part of the program, the costs involved and demographic information on participants

10. Program Clarification  Involves ongoing assessment of the program's strengths and weaknesses, and plans to review the program in the light of such assessments

11. Progress Towards Objectives  Involves the collection of data on the impacts of the program on participants

12. Program Impact  Builds on the previous stage in terms of long and short term impacts, typically involving comparison of participants and non-participants

13. The MoneyMinded Program  General objective: To assist people to make informed financial decisions and take control of their finances for their future  6 topics separated into 17 workshops* Planning and Saving Easy Payments Understanding Paperwork Living with Debt Everyday Banking and Financial Products Rights and Responsibilities *(original version which has now been revised)

14. The MoneyMinded Program  Allows individual participants to choose which workshops they wish to complete  Developed in part as a response to the results from the first-ever major survey of consumer financial literacy in Australia (ANZ, 2003)  Results of the survey identified the following: The groups within the community that could benefit from improved financial literacy education The issues that could be addressed in such an education program  Involvement of community groups in the delivery of program

15. The MoneyMinded Program  ANZ plans to partner with around 100 community organisations during the next 5 years to deliver the program to 100,000 people nationally  Evaluation is critical for a program as comprehensive and extensive as MoneyMinded  Importance of evaluation being collected and analysed in a systematic manner

16. Evaluating the Money Minded Program  Pre-implementation needs assessment Not a formal needs analysis at the individual participant level Based on the results of the financial literacy survey reported in ANZ (2003) Telephone survey customised to particular needs and circumstances

17. Pre-implementation Stage (Cont’d)  ANZ survey identified the following groups were strongly over-represented in the lowest financial literacy quintile: Lowest education levels Not working or in unskilled work Lower income levels Lower savings levels Single Extreme ends of the age profile Females

18. Group2003 Survey (%)2005 Survey (%) Females2425 Education Less than Year 10 4243 Looking for work3231 Semi-skilled2834 Unskilled4038 Single Living Alone 26 Aged 18 – 243133 Aged 70 +3137 Renting2927 ANZ financial literacy surveys in 2003 and 2005: Groups over- represented in the lowest financial literacy quintiles

19. Pre-implementation Stage (Cont’d)  The involvement of community groups in facilitating delivery of the MoneyMinded program has assisted in targeting the over-represented groups  Community group involvement has also proven to be successful in the recruitment of participants for the Saver Plus matched savings program  Needs assessment through administration of a pre- training questionnaire, which captured demographic information and data on the areas that participants identified where they most needed financial literacy education in terms of the modules offered as part of the MoneyMinded program

20. Topic AreaNumber of Interested Participants % of Total Participants Planning and saving10171.1 Living with and managing your debt 7955.6 Your consumer rights and responsibilities 6847.9 Understanding financial paperwork 6847.9 Your everyday banking and financial products 6143.0 Your payment options5135.9 MoneyMinded modules identified by participants as being areas where they had an interest in acquiring greater financial knowledge

21. Pre-implementation Stage (Cont’d )  Together with the fact that the MoneyMinded workshop is composed of individual workshops the participation in which is chosen at the individual level suggests that the design performs well on the criteria relevant to the pre-implementation stage

22. Accountability Stage  Requires that details be collected on a range of matters including the education and services provided and base participant information  Basic participant information Pre-training questionnaire collected data on demographics, mathematical and computer literacy, Internet usage, current savings and spending behaviour, savings intentions and current usage of a range of financial products Pre-training questionnaire also captured data on the financial literacy of participants

23. Socio-demographic Characteristics of MoneyMinded Participants Number of Participants% of Total Gender Male1913.4 Female12386.6 Age 15 – 343424.3 35 – 446345.0 45 – 542920.7 55 +1410.0 Education Primary / Some Secondary3022.8 All Secondary2418.2 TAFE / Workplace Training3224.2 University4632.3 Employment Status Full time5035.4 Part time4531.9 Casual107.1 Unemployed / Not looking107.1 Student107 Home duties1510.6 Retired107.1

24. Accountability Stage (Cont’d)  The collection of such data provides the potential for a rich analysis to inform the program clarification, progress towards objectives and program outcome stages of any evaluation by allowing an exploration of the influences of these demographic and financial factors

25. Program Clarification Stage  Requires an assessment of the program strengths and weaknesses, and the use of such information in any re- assessment of program goals and objectives  This element is present in the MoneyMinded evaluation  Initial roll-out to a small number of community organisations and participants allows for the type of assessment and re-thinking that is at the core of the program clarification stage

26. Program Clarification Stage (Cont’d)  The MoneyMinded evaluation also involved the administration of post-training questionnaires to participants Required participants to provide an assessment of each workshop completed in terms of how useful they found the workshop to be, how satisfied they were with the topics and material covered in the workshop and an assessment of overall program satisfaction  Post-training evaluation data when coupled with the pre-training data provides scope for a rich analysis at the program clarification stage  High overall ratings for both satisfaction (90% reported being very satisfied or satisfied with the overall program) and usefulness (87% found the workshops to be extremely useful or useful)

27. Program Clarification Stage (Cont’d)  A richer analysis for the program clarification is provided by considering whether there are differences in the satisfaction and usefulness ratings across categories in the demographic and financial data collected in the pre-training questionnaire  There were statistically significant differences in terms of usefulness and satisfaction ratings across participant categories  Higher ratings were provided by those in categories that captured lower incomes, education levels, employment status, mathematical and computer literacy, savings levels and intentions and base levels of financial literacy

28. Groupχ 2 d.f.χ 2 valuep-value Program Satisfaction Income Source25.2820.071 Employment Status411.6920.020 Income Level612.8230.046 Savings Intentions612.6340.049 Savings Level64.7560.576 Spending Behaviour67.6990.261 Understanding Bills21.6410.440 Credit card usage25.7000.058 Workshop usefulness Income Source214.3450.001 Employment Status429.1690.000 Income Level1251.5440.000 Savings Intentions630.7330.000 Savings Level629.6560.000 Spending Behaviour674.8440.000 Understanding Bills648.9530.000 Credit card usage623.7570.001 Workshop Satisfaction Income Source26.2100.045 Employment Status426.5900.000 Income Level1222.4800.032 Savings Intentions630.4530.000 Savings Level629.3370.000 Spending Behaviour637.0840.000 Understanding Bills644.2610.000 Credit card usage632.6030.000 Selection of the results on whether the ratings of satisfaction and usefulness provided by participants vary across the socio- demographic, financial product usage and financial literacy groupings

29. Program Clarification Stage (Cont’d)  Appropriate statistical analysis can be beneficial at the program clarification stage of a structured evaluation process Bulk of data collected in the MoneyMinded evaluation was categorical in nature Statistical testing was conducted using the χ 2 testing framework appropriate to such data Obvious issue in any formal statistical testing is whether the data collected satisfies the assumptions required of the statistical procedure being used Expected cell count of 5 for the asymptotic χ 2 approximation to work Requirement not satisfied in the initial data 2 approaches available: 1)aggregate cells together and then conduct the tests 2) conduct a simulation analysis of the exact distribution of the test statistic typically involving some Monte Carlo technique

30. Program Clarification Stage (Cont’d)  Post-training questionnaire also provided useful data on some dimensions of program delivery Website of resources made available to participants 92% of participants did not access the website Strong result which suggests use of technology in delivery should be assessed, and not just assumed in the design features Also captured data on the number of individual workshops that participants attended; majority attended only a single workshop, with a small number attending 12 workshops

31. Program Clarification Stage (Cont’d) Results show that participants who attended multiple workshops had higher usefulness and satisfaction ratings than those who attended a single workshop; differences in the ratings are statistically significant Suggests consideration be given in the program design to encouraging attendance at multiple workshops via greater bundling, while recognising that workshops are already bundled by broad topic area Useful and constructive information to obtain at the program clarification stage of an evaluation

32. Program Clarification Stage (Cont’d)  Assessment of MoneyMinded also involved completion of questionnaires by facilitators who delivered the program, in terms of their expected usefulness of the workshops prior to delivery, and their perceptions of participant usefulness post-delivery  Ratings were high and were broadly consistent with the ratings provided by participants  Useful data at the program clarification stage in informing how the program goes forward

33. Program Clarification Stage (Cont’d)  Final data collected of relevance to the program clarification stage was questionnaire data from both participants and facilitators in regard to improvements that could be made to the program in terms of expanding the topic coverage  Need for workshops on investment, superannuation and retirement planning  To be expected – ties in with government policy in recent years to move funding of retirement incomes to private funding via long-term superannuation investments

34. Progress Towards Objectives Stage  Objective of a financial literacy education program is in terms of improved financial literacy, and as a consequence better financial decision-making by participants  In the context of an initial evaluation of the roll-out of what is to become a wider program, such a stage is difficult to effectively action, except in terms of interpretation of ratings of satisfaction and usefulness that have been previously utilised at a program evaluation stage

35. Program Impact Stage  Short and long-term  Looking for wider demonstrable evidence in terms of improved financial literacy and better financial decision making by those who have participated in financial literacy programs  Fox, Bartholomae and Lee (2005) recommend that this measurement of impact be done relative to a control group who have not participated in such training  This stage is not relevant in the current assessment of MoneyMinded, given the program is only at an initial roll- out stage

36. Program Impact Stage (Cont’d)  How such an evaluation might be built for use in the future Future broad surveys of financial literacy to include questions for the respondents on whether they have previously participated in a financial literacy education program Allows us to compare measures of financial literacy across groups that have/have not undertaken financial literacy education, after making appropriate adjustments for other socio-economic and demographic factors known to influence financial literacy Longer-term analysis of the savings behaviour of participants in a matched savings program would provide evidence of the joint impacts on financial decision-making of participation in both a matched savings program and financial literacy education

37. Conclusion  Exploration of the Fox, Bartholomae and Lee (2005) framework in the context of a specific program, the Australian MoneyMinded program  Framework developed in the context of US programs appears to generalise well to the Australian context, thus providing support to the general applicability of the framework  Formal statistical evaluation might be used at certain stages  Measurements of longer-term impacts i.e. behaviour change