1. Elements of poverty reduction program monitoring and evaluation Sophisticated development context: the Kazakh experience Ayse Kudat Social Assessment 2004

2. WHAT IS OUR CHALLENGE IN KAZAKHSTAN? START A PRPM PROCESS DO SO BY TAKING MODEST SUSTAINABLE STEPS AS A FIRST STEP, INTEGRATE NATIONAL AND OBLAST LEVELS PRP ELEMENTS INTO A UNIFIED SYSTEM AND SUPPORT THIS INTEGRATION WITH REAL-TIME AND GEPOGRAPHICALLY SPECIFIC COMPUTER APPLICATION DESIGN A PROCESS OF CAPACITY BUILDING FOR PRP AND PRPM BY LINKING INPUTS, ACTIVITIES, OUTPUTS, OUTCOMES AND IMPACTS AT THE OBLAST LEVEL INITIATE A SIMPLE SYSTEM OF PRPM COORDINATION AND CREATE PRP MONITORING WORKING GROUPS IN MEBP, MLSP AND IN THE 14 OBLASTS ENSURE MONITORING CONTRIBUTIONS OF NGOS CURRENTLY COOPERATING WITH OBLASTS IN PRP PREPARATION AND IMPLEMENTATION SUPPORT THIS PROCESS IN 2 PILOT OBLASTS WITH FOCAL POINTS FINANCED BY THE TA PROJECT SUPPORT THE PARTICIPATORY PRPM GOALS OF OBLASTS BY HELPING 2 PILOT REGIONS STRENGTHEN M&E CAPACITY IN THE RAYONS---HELP THESE OBLASTS ENHANCE LOCAL M&E CAPACITY WITH MODEST RESOURCES AVAILABLE IN TA PROJECT CREATE INSTITUTIONAL ARRANGEMENTS FOR EVALUATION CONDUCT PILOT STUDIES FOR IMPACT ASSESSMENT AND INITIATIVE SURVEYS TO TRACK PUBLIC EXPENDITURES ASSESS EQUITY AND EFFICIENCY IMPLICATIONS OF PUBLIC SPENDING ON PRP EVALUATE IMPLICATIONS OF TA PROJECT ACTIVITIES FOR NATIONWIDE IMPROVEMENTS IN THE M&E OF PRPS AND PRP

3. Q: WHAT DO WE LEARN FROM INTERNATIONAL EXPERIENCE OF POVERTY REDUCTION STRATEGIES/PROGRAMS? A: PUBLIC SECTOR EXPENDITURES AND ACTIVITIES DEVOTED TO POVERTY REDUCTION PROGRAMS (PRP) CANNOT BE JUSTIFIED OR CAUSALLY LINKED TO POVERTY TARGETS UNLESS A RESULTS BASED MONITORING IS ESTABLISHED

4. In Kazakhstan we know more about outcomes than about the means of achieving them OutputActivityInput Outcome Impact Results Monitoring Implementation Monitoring (Means and Strategies) Implementation Results

5. Sources of Information for PRPM OutputActivityInputOutcome Impact Households surveys, qualitative surveys Public expenditure Tracking Surveys, Sector specific surveys Implementation Results Oblast budgets, oblast payroll, Rayon budgets, rayon payroll Oblast PRP workshops, task forces, Policy reviews, community infrastructure inventories, Rayon M&O achievements, WSS Systems repaired/constructed TA financed qualitative surveys, Public expenditure Tracking Surveys in 2 pilot oblasts,

6. 1.Conducting readiness assessment 2.Agreeing on Outcomes to monitor 3.Selecting indicators to monitor outcomes 4.Establishing baseline 5.Selecting realistic targets 6.managing for results 7.Role of evaluations 8.Reporting findings 9.Using findings 10.Sustaining M&E systems Progress made in PRPM&E in Kazakhstan

7. Much is already known What data are collected (source) When data are collected (frequency) How data are collected (methodology) Who collects the data For whom the data are collected Who reports the data Where revenues Are collected, Expenditures made And activities held, Outcome data Are not systematically gathered RAYONS

8. MEBP Inter-Ministry Steering Committee Secretariat Monitoring Working Group Two Pilot Oblasts Focal Points Oblast PRP working groups, with Oblast M&E Coordinators National Statistics Agency Local Univ., and civil society organizations Ministry of Labor and Social Protection Joint evaluation Group Rayon Administrations Other Line Ministries Steering Committee Recommendations to the Parliament and the Cabinet for Poverty Reduction Strategy Remaining Oblasts CITIZEN FEEDBACK Households

9. Stakeholders in KAZAK PRS monitoring Central Ministries (Finance, Economy) MEBP consolidate information, Line Ministries (MLSP) Provide oblast based data, helps evaluate results Reporting arrangement to be structured between MEBP and other line Ministries under the TA Universities, research institutes Local universities contribute to training and impact data gathering for small scale impact studies Rayon governments Key provider of input and output data to Oblasts National statistical agencies Conducts reliable and large scale, nationwide household surveys representative at the oblast level Develops rayon level capacity for collection of output data Civil society organizations Trade unions provide feedback information, Association of SME can contribute impact data Parliament/Cabinet MEBP through the Steering Committee to carry out Evaluation and provide strategy feedback to Parliament and Cabinet

10. Participation and Feedback Civil society involvement in policy formulation exists at Oblasts and through the TA financed focal points their contributions for M&E can be structured. Through PRPM the program planning for future PRPs (2004+) can be improved. Targets, inputs, outputs can be better defined. Sharing PRP results with public through the Newsletter of the national NGO consortium, as well as the website of the MEBP (with links to other line Ministries) can help generate citizen feedback and ensure sharing of PRP/PRS information

11. Client Feedback Limited resources available for all studies, including report cards, are extremely limited under the TA. Specific citizen feedback will be generated through the NGO Newsletter Citizen/user feedback will be assessed for 2 projects (e.g., a- SME development for the unemployed, and b- micro credit for rural areas) in 2 pilot oblasts through surveys carried out by local universities. These studies will also include budget tracking surveys Qualitative impact studies of pioneering programs in 2 pilot oblasts (a-process of social passport issuance and targeting; and b- emigration from ecological disaster areas) PRPM works shops in 2 pilot oblasts with rayon and select settlement representatives; participatory evaluation of PRP as an input to next steps of PRP preparation.

12. Types Of Evaluation Proposed within TA project Impact Evaluation of Health sector expenditures Process Implementation- commissions SME training for The unemployed Meta-Evaluation Case Study of Immigrants to 2 oblasts Pre-Implementation Assessment

13. Using the Results - Information Dissemination Results will be disseminated through MEBP website Results will be disseminated through MLSP website Targets, inputs, outputs, outcomes will be disseminated through newsletters published by civil society groups MEBP will hold regular press conferences and issue press releases on PRP progress Joint evaluation groups will be established at the central and pilot oblast levels Inter-ministry information will be assured through the Steering Committee as well as through M&E data flow yet to be structured from line ministries to the MEBP

14. Who Monitors? How should Participation in PRPM be organized? Many PRPs rely on outcome monitoring ò Such a system largely available in Kazakhstan ò But does not allow evaluation of the PRS and PRP ò Runs the risk of being a paper exercise Responsibility for PRP Monitoring is with the MEBP MLSP shares the burden Other Ministries can contribute Agency for Statistics tracks quarterly trends in poverty outcomes Oblasts, rayons contribute to M&E Civil society organizations support M&E and can do so more systematically and regularly Citizen involvement in PRPM can be improved è Public Expenditure Tracking Systems (PETS) è Quantitative Service Delivery Systems (QSDS) è Social Assessments, Citizen Report Cards è Participatory Impact Monitoring

15. Monitoring data flow In terms of institutional mechanisms, the data on poverty will flow through several different channels From RAYONS to OBLASTS to the central level MEBP for monitoring PRP From RAYONS to Statistical Agency for monitoring of the outcomes Sectoral information, MIS data and/or data from sample surveys conducted by different central agencies, including the MLSP to MEBP Data from key NGOs (e.g., trade unions, association of business and farmers’ association) to Oblast governments and transmitted to MEBP Case studies, budget tracking studies and client satisfaction studies conducted at the Oblast level and shared with MEBP

16. Input, Activity and Output Indicators: Unified and re- prioritized in accordance with the National Poverty Reduction Strategy Data obtained from the oblast level Data obtained from the implementing line ministries Outcome Indicators Data obtained from the oblasts Data obtained from the National Statistics Agency Sources of Monitoring Indicators Real-time Monitoring and Evaluation Software Analysis of specific actions underlined at the National Poverty Reduction Strategy in terms of actual outputs provided, and poverty reduction outcomes achieved Making these results available at the oblast level Enabling direct linkages between resources put through specific actions and policy outcomes of these actions, assisting policy makers in designating new priorities Expected Results from the Monitoring Software

17. Sample Indicators – How the Software Will Work Inputs Name of Activity: Primary school repairs Amount of financing (to be indicated) If financing not available, indicate estimated number of man-hours dedicated to this activity Activities and Outputs Number of schools undergoing repairs Estimated number of beneficiaries (ie the number of pupils in these schools) Outcomes Number of primary school age children studying in schools – urban and rural Percent of pupils in schools that study on the third shift Oblasts National Level Charts and Graphs for summary information and evaluation

18. Results will be made available on a geographical system (sample indicators on education) 30 schools repaired 15,000 pupils benefited 20 schools repaired 13,000 pupils benefited 10 schools repaired 9,000 pupils benefited