Impact evaluation-approaches and methodologies

Pursuit for Effective Impact Evaluation Tools for Market Development Programmes Most Significant Change (MSC); Methodology and Application MSC Integration in Market Development Programs; Rationale and Case for MEDA Prosperity through Innovation Zambia Project Results, Benefits and Lessons Learned in integrating MSC in Market Development Programs Conclusion

John Maynard Keynes It’s better to be approximately right than precisely wrong.

 Impact evaluation - as the component of a project cycle that seeks to validate to what extent changes occurring to target groups can be attributed to project interventions - is a topical issue in the development sector.  Impact evaluation is a necessary part of project planning and monitoring and for improving implementation strategies regardless of type of organization  For evaluators the challenge now is how to adapt the basic approaches of M&E (qualitative and quantitative) to develop an innovative and cost effective M&E model  For evaluators of market development projects developing models is even more of a challenge due to the complexity of the market itself (supply and demand side)

1. Market intervention indicators -Relating to outputs, measuring the immediate results of project activities e.g sales performance of products, increased stock volumes, size and expanse of retail networks 2. Market development indicators -Relating to outcomes, measuring results over space and time including access to information on products location, price and benefits, impressions of specified products to drive demand, supply side development and adherence to business and marketing plans, suppliers capabilities in product stocking and marketing. Others may includes adherence to product quality, assessment of the users knowledge on how to use technology, their purchasing power and demand. 3. Market sustainability indicators -Relating to impacts, measuring results when the project intervention is withdrawn

 Market development evaluators need to go beyond measuring market intervention indicators as the success of a market development project depends on sustained product adoption  Market development indicators are difficult to track because they are qualitative  The MEDA experience has shown that the adoption of MSC as an evaluation technique can be an effective tool in assessing market development projects and specifically market development indicators

What is Most Significant Change? Most Significant Change (MSC) is a participatory, qualitative and complimentary approach to impact evaluation. MSC can be likened to newspaper development. Where stories are collected and prioritised. In this case, however, groups of beneficiaries are asked the MSC question and a panel picks the “best” MSC story. These stories are then compiled into a “catalogues/newspaper” and disseminated to decision makers. There can be many levels of sifting stories depending on the size or goal of MSC enquiry. The MSC approach was originally developed in 1994 by Rick Davies through his work on a rural development project in Bangladesh with the Christian Commission for Development Bangladesh (CCDB) and has been adopted by a number of organizations in different countries MEDA has applied MSC to measure market development indicators to be specific issues around sustainable adoption of SITT’s in the value chain. Most Significant Change (MSC) is a participatory, qualitative and complimentary approach to impact evaluation. MSC can be likened to newspaper development. Where stories are collected and prioritised. In this case, however, groups of beneficiaries are asked the MSC question and a panel picks the “best” MSC story. These stories are then compiled into a “catalogues/newspaper” and disseminated to decision makers. There can be many levels of sifting stories depending on the size or goal of MSC enquiry. The MSC approach was originally developed in 1994 by Rick Davies through his work on a rural development project in Bangladesh with the Christian Commission for Development Bangladesh (CCDB) and has been adopted by a number of organizations in different countries MEDA has applied MSC to measure market development indicators to be specific issues around sustainable adoption of SITT’s in the value chain.

Steps of MSC (Davies et al 2005)

The MEDA Project - Prosperity Through Innovation Project Goal Accelerating the demand and supply for appropriate and affordable water technologies Project Approach Using an e- discount voucher as a market catalyst Building the capacity of water technology suppliers and dealers to serve farmers better Project Strategy Pure market development facilitation Invisibility for effective market sustainability and project exit strategy

Partner Organisation ; Irrigation Trainers Smallholder Farmers Water Technology Suppliers and Retailers MEDA Provision of Training on Irrigation Business Transactions on water technologies Supply side strengthening through capacity building on marketing and cross cutting issues BUSINESS SUPPORT SERVICES Organization interventions or Member Services Market FacilitationMarket Development

 Lack of Baseline Survey from partners  Market Development program; requiring in-depth understanding of the benefits to program beneficiaries  To supplement training needs assessment tool and client satisfaction survey  To validate sustainable adoption of technologies with the view of accelerating demand. Are clients buying into the technology as appropriate, aim of the project  To check anticipated project impacts/ monitor unintended impacts  Cost

1. Defining Domains of Change 1. Changes in Business Practice and Spirit of Entrepreneurship 2. Changes in Production and Productivity 3. Changes in Business Income 4. Knowledge on Technology, Available Technologies and Suppliers 5. Household Livelihood Improvements 6. Changes in Risk Aversion, Technology Adoption and Technology Appreciation 2. Defining the Reporting Period Annually 3.Collecting SC stories 1. The stories are collected by the M & E practitioner of the project. 2. The practitioner meets a group of six farmers (maximum) at a time

4. Selection of collected SC stories 1. The panel of stakeholder used is the farmers themselves. 5. Feedback on the choices made Then the M and E practitioner develops a newspaper of these articles and circulates them to the project management at different levels within the organization 6. Verification To verify the stories the project opted to conduct the MSC approach in groups of farmers in the same locality

 Stories were compiled and at the MEDA office the number of stories on a particular domain of change were weighed with those considered MSC. In both scenarios increase in income was highest placed impact.  The application of MSC enabled the MEDA Zambia team to establish that the major impact of the intervention i.e getting the SSIT’s out to the farmers was an increase in gross income for farmers.

 The validation of sustainable adoption for most farmers pointed to benefits in terms of finance accumulated versus livelihood indicators that project monitoring activities were focused on mostly e.g health, education, housing

 In project planning terms meaning that the main information that other farmers should be furnished with to accelerate demand should be the potential for them to increase their business income.

It is not a stand alone technique;  Hit the nail on the head-get to the issue directly  Its never too late too use it  Gives a richer description of the context in which a positive change is occurring, beyond the data from the quantitative case studies  Quality is added to the data collected as the stories are collected in the beneficiaries voice(verbatim)  It is more flexible as it is not based on predefined indicators, you can measure unpredicted outcomes

 MSC can be adapted to measure specific qualitative aspects of a multi faceted project  Not all MSC domains will vary from indicators but can still give a closer look at outcomes  During story telling other demand side players learnt from their peers on best methods to employ the adopted technology  The stories can be furnished with quantitative data for depth  If the MSC is conducted in a target group with participants in the same areas they serve as a verification board to measure the truth in a story.

 Market development indicators are often a challenge to monitor due to the complexity of the market and the qualitative nature they take.  MSC as an evaluation approach can be applied to capture specific qualitative aspects of change for a multi faceted intervention.  For MEDA its application enabled the project validate sustainable adoption of SITT’s as it related to income increase. 1. There is proof farmers have not only bought the SITT, but have spotted benefits of it and realized its need (accelerating demand) 2. It was not complicated to apply and a larger number of farmers were reached to gather this specific vital information  MEDA is sure what message should be fed to farmers to accelerate demand as the supply chain is being developed even if it was not what was prioritised initially.