Managing & Measuring Progress and Impact: National Program Reviews & Evaluations LFA M&E Training February

Session objectives 1.The New Funding Model and the Evaluation Agenda 2.Program reviews: Retrospective view 3.Program reviews: Prospective view 4.Continuum of support in line with NFM 5.Joint planning and implementation of program reviews 6.Summary 2

Program reviews/evaluations inform funding decisions and implementation in the NFM -Where does the country stand, and what is the direction of the epidemic? increasing, decreasing, stabilized? -What worked well, and what did not work well so should be done differently? -Where, and which population groups, should investments be focused on? Which interventions should be prioritized for whom? -What should continue and what should be reprogrammed for better impact? From separate, commissioned studies – To a system of joint evaluations/reviews Evolve to impact and outcome measurement and improve data quality in the field (High Level Panel recommendation) 1. The new funding model and the evaluation agenda 3

Building the evaluation agenda on the Five P Principles Partner approach (build-on, collaborate and synchronize evaluations with partners, maintaining rigor and objectivity); Periodic (planned, regular evaluations rather than one-off evaluations); Plausibility design (provide evidence of program impact, positive and negative, also considering other non-program influences along the M&E chain); Country Platform (build on national systems including program reviews and strengthening of country M&E capacity) Practical for grants (recommendations for grant management, grant renewal and re-programming). 4

2. Program Reviews look both ways! Retrospective: Impact assessment –Has the program made a measurable progress and impact? –Was the observed progress and impact, if any, linked to improved service delivery coverage and increased financing? –Where did the program perform well, and where do things lag behind? –Which areas and/or population groups were disproportionately disadvantaged? Prospective: strategic plan, investment case, re-programming –How and where should limited resources be invested to achieve optimal impact? –Who should be prioritized and where should investments be focused for better impact? 5

Why impact assessment? We need to know whether and the extent to which prior investments have led to a reduction in disease burden and an increase in lives saved. –What were the drivers to the success that need to be continued and accelerated? We need to know if there are areas and groups where disease incidence, prevalence and mortality have not declined, and why. Lack of investment? Poor prioritization of interventions and/or population groups? Poor implementation? –What needs to be done differently in order to see morbidity and mortality go down? 6

Why is impact important for the Global Fund? Purpose of the Global Fund: −“a sustainable and significant contribution to the reduction of infections, illness and death” Global Fund Strategy – Investing for Impact −Targets in terms of impact: MDGs, lives saved, infections averted −Key part of programming: invest more strategically for impact Management focus of grants −Impact upfront: planning for impact to promote prioritization −PBF for impact: identifying and investing in value in the portfolio 7

What do we look at during impact assessment? Combines four key components: 1.Overall impact on the burden of cases and deaths due to the three diseases: Has there been a change in disease incidence, prevalence & mortality, positive or negative ? Has there been a change in outcomes and behaviors, positive or negative? 2.Causation and alternative explanations along the results chain: Does scale-up in interventions partly or fully explain the documented impacts? Do other competing explanations or hypotheses contributed to and resulted in these impacts, positive and negative 8

What do we look at during impact assessment? … Combines four key components: 3.Contribution of the Global Fund investments −no need for attribution of results to GF investments 4.Additional data needs for rigorous analysis: disaggregation of data by time, person and place and including comparison groups where feasible 9

Build country system of reviews to strengthen grant management and strategic investments for impact Country Platform of national program review with data investments National Program ReviewNational Program ReviewNational Program Grant Renewal/ New Funding Application Grant Management 10

LFA role in impact assessment In the capacity assessment tool (CAT), LFAs should determine whether the M&E system at the moment is capable of generating data that would enable tracking impact and progress particularly data on disease morbidity, mortality and service delivery and behavioral outcomes Deliverable: Capacity assessment report with details on currently functional data sources, observed gaps, as well as required investments to enable monitoring of progress and impact. Before grant signing, LFAs should verify that data sources and timelines are clearly defined and resources identified for measuring each impact indicator. LFAs should assess/verify impact during annual disbursements, based on indicators agreed in the respective grant documents. Deliverable: LFA verified annual progress updates with impact analysis. 11

Timeline for LFA involvement in impact assessment Grant signing Grant Implementation 12 Country dialogue Annual Disburse ment Grant Making Annual Disburse ment M&E Capacity Assessment Data sources, timelines and resources for impact data Verification of impact data, Analysis of impact Verification of impact data, Analysis of impact

Round table Discussion -1 Form small groups of 4-5 people and discuss the following 2 questions (10 min): a) Identify 1-2 examples of circumstances where it was difficult to assess impact using indicators as described in the grant agreement document. Discuss what the major obstacles were for objectively assessing the impact. When is the right time to deal with possible bottlenecks? b) Identify 1-2 examples when assessment of impact led into a programmatic and/or management action, and how the action changed grant implementation If no good example, discuss an ideal situation – how objective impact assessment could help grant implementation 13

3. Program Reviews look both ways! Retrospective: Impact assessment –Has the program made a measurable progress and impact? –Was the observed progress and impact, if any, linked to improved service delivery coverage and increased financing? –Where did the program perform well, and where do things lag behind? –Which areas and/or population groups were disproportionately disadvantaged? Prospective: strategic plan, investment case, re-programming –How and where should limited resources be invested to achieve optimal impact? –Who should be prioritized and where should investments be focused for better impact? 14

NSP: a key entry point in the new funding model NSP Support NSP 15

National strategic plans 1. Joint Assessment of National Strategies Allows governments to proceed more rapidly through the steps of the new funding model To be used for Global Fund financing, NSPs should be : –Developed using an inclusive multi-stakeholder process; and –Strong and robust following technical guidance documents such as JANS 1 criteria Ideally, NSPs will be jointly assessed through a credible, multi-stakeholder process using an internationally agreed framework The Global Fund Country Teams are not expected to support the development or evaluation of NSPs. Neither do LFAs. –Global Fund will direct countries to relevant technical resources, and will work closely with partners to support countries NSP support 16

4. Ideal approach: continuum of support in line with the new funding model Joint assessment of DQ & systems Identification of key data gaps Quantification of investment needs Strategic investment in data systems Joint review of Epi and impact for the new funding model Before the development of a Concept Note and as part of country dialogue Identifies data limitations and required actions Ambitious yet realistic goals and SMART objectives Costed M&E plan Measurable indicators, clear sources of info and means of verification Joint reviews with a particular focus on epidemiological impact & progress Recommendations to inform a revision or dev. of new NSP. Surveillance Systems and Data Quality assessment Epi analysisProgram Review Strategic Plan Development Ongoing1-2 weeks3-4 weeks2-3 months Global Fund application 17

Why support surveillance systems Assessments? We need to determine whether the system is generating quality data that will enable evaluating results and impact We need to understand whether there is capacity for analysis, review and communication of data We need to identify areas for further strengthening and the corresponding investment needs “In God we trust. All others bring data.” W. E. Deming 18

Do we have what is needed to measure progress and impact? 19

to ensure that changes at outcome and impact levels can be measured; To understand the trends and the underlying causes for the observed trends (positive or negative) to have better understanding of epidemiology of a disease, by place person and time; to focus limited resources where the impact would be higher, and to make a stronger case for the requested investment. Why support Epi analysis? Short-term Long-term 20

LFA role in program reviews Program reviews should be carried out every 2-3 yrs and will inform strategic planning and concept note development. During grant making, LFAs should check that program reviews/ evaluations are planned, budgeted for and timelines defined in the grant documents Deliverable: LFA report with comments on the timelines, budget and sources of finacing for program reviews For the Global Fund financed program reviews, PRs will submit the review protocol and budget justification for approval. LFA should review the protocol against the standard WHO review guidelines, highlight any deviations, if any, and comment on the justifications for the deviation LFA should use key findings and recommendations of program reviews to continuously update the contents of the capacity assessment tool 21

Round table Discussion -2 Reconvene in your groups of 4-5 people and discuss the following 2 questions (10 min): a)Discuss how program review reports could be used by the LFAs. What if there are visible differences, in certain areas, between the program review report and the LFA observation? b)In addition to the ones described in this presentation, in what other ways could LFAs play a role in program reviews and impact assessment? 22

5. Joint planning and implementation Program reviews should happen every 2-3 years (mid-term and end-term) Countries should include timeline and budget for upcoming program reviews in the respective grant documents LFAs should check that countries have the plan included in the respective grant documents TERG support and guidance for priority countries WHO/UNAIDS/GF Performance and Impact Group Global Fund Country Team: Regular communication with MOH and in-country technical/funding partners 23

Tracking country schedules: with internal and external partners 24

Tracking country schedules: with internal and external partners 25

6. Summary Program funding requests from countries are expected to be based on the National Disease Program Strategic Plan, and should be informed by: –Assessment of data quality and surveillance systems, that identifies gaps and investing opportunities to strengthen major components of the national health information systems –Epi analysis conducted preferably, as part of program review, or as part of Country Dialogue, prior to the development of a Concept Note –Actionable recommendations stemming from Program Reviews Global Fund will continue to work closely with countries, WHO and other technical and funding agencies, to ensure that programmatic and funding decisions are fully informed by data and guided by evidence 26

Summary: LFA role in impact assessment and program reviews As part of Capacity Assessment using capacity assessment tool (CAT), LFAs can play a crucial role in determining whether the M&E system at the moment is capable of generating data that would enable tracking impact and progress –particularly data on disease morbidity, mortality and service delivery and behavioral outcomes It is extremely importance that assessment of impact is done during annual disbursements, based on agreed indicators in the grant document –annual progress updates with impact analysis At the end of the three years of implementation, the impact will be critically assessed and will inform funding decision for the next phase of application /implementation –LFAs should check whether reviews / evaluations are planned, budgeted and timelines are defined in the grant documents 27

Thank you... TB detection (SM+) declines Questions? 28