Linking scale up theory to scale up M&E: Methodological findings from a five-country prospective study of scaling up of a reproductive health innovation Presenter1Abstract: A five year effort to achieve national scale up of new family planning method into health systems was undertaken in Guatemala, India, Democratic Republic of Congo, Mali and Rwanda from , informed by a systems and values-oriented conceptual scale up framework, ExpandNet. This framework and the program theory embodied by the framework guided development of monitoring and evaluation system as part of a multiple case study design. Variables and performance benchmarks to monitor institutionalization into health systems and availability of the new method within family planning services were identified and monitoring of processes for coverage, quality, institutionalization, sustainability, adherence to innovation values were measured. The presentation will explore how program theory informed monitoring and evaluation theory and practice, which monitoring methodologies were most used to guide strategic decisions during scale up, and recommendations for monitoring and evaluating a scale up effort within a complex system. Relevance: Scale up of health and other innovations are an important focus in many national programs to bring new services and improve health and social outcomes at national level. Scale up and monitoring of a scale up process and outcomes is often simplified and not viewed using a complex systems lens and many efforts lead only to short-term program impacts. Using a systems-oriented scale up model should lead to more sustained integration of new services. Likewise, M&E systems need to be designed to capture systems dynamics, environmental changes, and the complexity of multi-year, multi-organizational, and multi-level efforts. The presentation adds to a relatively small body of knowledge of good evaluation practice of scale up efforts and to understanding scale up as a process that occurs within complex systems that requires specific evaluation strategies. As a prospective study, an analysis of which methodologies were most valuable in measuring scale up and also most used by decision-makers adds to understanding of better evaluation practice of scale up programs.

Linking scale up theory to scale up M&E: Findings from a five-country prospective study of scaling up of a reproductive health innovation Susan Igras, Rebecka Lundgren, Victoria Jennings AEA Conference - Washington, DC – 18 October 2013

Presentation Overview 1. Program theory for scale up of health innovations and implications for M&E 2. Approach used by IRH – informed by systems-oriented WHO/ Expandnet scale-up framework 3. M&E of scale-up o Defining the systems changes necessary for sustainable SU o M&E system – variables of interest & methods o Measuring success in a complex systems context 4. Lessons learned o Most used M&E methodologies to guide strategic decisions during scale up. o Recommendations going forward.

“ Scaling-up Defined Deliberate efforts to increase the impact of health service innovations successfully tested in pilot or experimental projects so as to benefit more people and to foster policy and program development on a lasting basis.” Deliberate efforts to increase the impact of health service innovations successfully tested in pilot or experimental projects so as to benefit more people and to foster policy and program development on a lasting basis.” [Resource: Nine Steps for Developing a Scaling-Up Strategy. WHO/ExpandNet, Pre-publication version April 2009]

Goals Create significant outcomes that address a critical problem at a large scale Foster these outcomes on a lasting basis How to achieve goals Innovation Well-defined Demand Monitoring fidelity Deliberate efforts to implement Implementation supports M&E of scale up Systems change Achieving Scale-Up Goals A Fixsen

Complexity Theory Scale up does not occur in a vacuum The focus of scale up is the system (political, social, economic)  Reflect this in M&E practices And systems are complex... It’s not so linear… Technical innovation The zone of complexity! Social innovation A Fixsen

What Complexity Tells Us Expect the unexpected Some systems may move more quickly than others – tailor your approach, pay attention to local context Utilize M&E to track and react to events as they unfold – see what emerges and how it impacts scale up A Fixsen

Complexity-Informed Evaluation Evaluates from within - work with the system Collects data frequently  Capitalize on quick feedback cycles Works to understand the interactions within systems Systems are the focus of change Developmental Evaluation (Patton, 2011) A Fixsen

Implications Set the Stage Define the innovation – implementation landscape, innovation components, demand Define the scale up process – potential sources of secondary data, benchmarking M&E of Scale Up Focus M&E for the scale up period – measure scale up process and pace, as well as the coverage and fidelity of innovation Observe links between scale up strategy and innovation fidelity Be flexible – remember the zone of complexity, tailor indicators and approach to the context A Fixsen

SCALE-UP CASE STUDY ( ) 5 year prospective, multi-site, comparative study of process and outcomes of scaling up a family planning innovation (Standard Days Method) Democratic Republic of Congo, Guatemala, India-Jharkhand State, Mali, Rwanda

CASE STUDY METHODOLOGY Research questions and hypotheses Logic model, indicators, benchmarks Various data sources Triangulation Organized case study data base “Thick description” of program implementation, outputs and outcomes (Yin 2009, Stake 2006 )

COMPLEX SYSTEMS APPROACH Conceptual framework informed by systems- based ExpandNet model Team to guide complexities of multi- organization, multi- sector, and multi-level process

Monitoring & Evaluation Process and Practice

M&E Process Planning meetings to define innovation and operationalize scale up Results shared regularly with resource team and user organizations – transparency and buy-in Participation needed from national, district and local levels and from different partner organizations

Government of Jharkhand definition of scale Phase 1, Jan 2008 Pop: 3,765,983 Phase 2, Feb 2010 Pop: 2,755,023 Phase 1, Nov 2010 Pop: 5,520,869

Guatemala FAM Project Progress: Summary Table 1 (Through June 2012) FAM project accomplishments toward end of project target, by project year Guatemala End of Project Goals: By 2013, SDM will be well established in at least one-sixth of the country – three departments: Sololá, Quetzaltenango and Santa Rosa. LAM will be available in at least three departments, accordingly to feasibility for its integration along with SDM. TDM services will be established, where feasible, once integration of TDM into the MOH norms and integration possibilities are assessed. Guatemala population coverage: 825,456 (3 departments) Horizontal scale-upYear 1Year 2Year 3Year 4Year 5 project target (n) Proportion of SDPs that include FAM as part of the method mix (43%) 213 (69%) 300 (97%) Estimated number of individuals trained to counsel clients in FAM (IRH-supported) 236 (13%) 723 (40%) 1,215 (67%) 1,361 (75%) 1,895 (105%) 1, Number of organizations that have capacity to undertake FAM activities (are resource organizations) 0 7 (47%) 10 (67%) 12 (80%) 12 (80%) 15 Vertical scale-upYear 1Year 2Year 3Year 4Year 5 project target (n) SDM and LAM included in essential or key policies, norms, guidelines, and protocols a Presence of public or private training organizations that include FAM in pre-service training and/or continuing education b Presence of public or private training organizations that include FAM in in-service training 3 (27%) 9 (82%) 10 (91%) 11 (100%) 11 (100%) a Sustainable inclusion of CycleBeads into donor procurement system b Sustainable inclusion of LAM user card into donor procurement systems c Sustainable inclusion of CycleBeads into logistics systems d Sustainable inclusion of the LAM user card into logistics systems a Inclusion of SDM in HMIS/reporting systems b Inclusion of LAM in HMIS/reporting systems a Inclusion of SDM in IEC activities, materials and mass media0 4 (36%) 5 (45%) 9 (82%) 9 (82%) b Inclusion of LAM in IEC activities, materials and mass media c Inclusion of BOTH SDM and LAM in IEC activities, materials and mass media0 2 (18%) 8 (73%) 9 (82%) 10 (91%) Inclusion of FAM methods in surveys (e.g. DHS) BENCHMARKINGPROCESS

M&E Approaches and Tools by Scale-Up Domain Tools/Approaches CoverageSustain ability ProcessQualityValues Household survey & Facility assessment Provider interviews Benchmarking table Service statistics, sales, stock out reports Key events tracking Quality assurance tools In-depth interviews with stakeholders & scale-up team Most Significant Change

Key Events Tracking Rwanda, through June 2010

Quality Assurance Tools Provider supervisionClient follow up

Most Significant Change Using a participatory, story-based, ‘indicator free’ methodology to complement other M&E data

“Looking back over the last year, what do you think was the most significant change you have experienced as a result of Standard Days Method being offered in your community?” And ask ‘Why?’

Most Significant Program Manager Story - Mali Title: Increasing CYP & strengthening clients’ right to choose STORYTELLER Why most significant? SDM fosters equity and the respect of clients’ rights to choose, Has increased CYP of FP programs, Has increased the range of FP methods available to clients STORYTELLER Why most significant? SDM fosters equity and the respect of clients’ rights to choose, Has increased CYP of FP programs, Has increased the range of FP methods available to clients

Use of M&E Data

How were M&E data used to support scale-up? 1. Adaptations in innovation package 2. Strategic planning 3. Strategy adjustments 4. Supervision and problem resolution 5. Recommitment to scale up process 6. Engage new partners

Selected Indicators June 2008 June 2009 No. of resource organizations 5 of 86 of 8 Method included in key policies, norms, protocols 2 of 43 of 3 Method in pre-service training 5 of 5 Commodities in logistics & procurement systems 4 of 96 of 9 Method in IEC materials 7 of 12 Method in HMIS NoYes SDPs with method in method mix 79%100% Providers trained Mali: Monitoring Performance Benchmarks

Availability of SDM: FP Service Statistics Jharkand service data, through Jan 2011

Importance of monitoring values inherent in SDM innovation at scale Stories of Most Significant Change by program managers, service providers, and users Stakeholder interviews on appreciation of SDM and what it brings to FP programs and services Purposeful gender measures in different tools Monitoring of method bias by providers using simulated clients and supervision visits Some values inherent in Standard Days Method -Woman empowered with understanding how fertility works -Male involvement -Informed choice -Quality standards -Bringing new people with unmet need to FP (CYP)

Lessons Learned on Scale-up M&E

MOST- used M&E Approaches and Tools Tools/ApproachesCoverageSustain ability ProcessQualityValues Household survey & Facility assessment Provider interviews Benchmarking table Service statistics, sales, stock out reports Key events tracking Quality assurance tools In-depth interviews with stakeholders & scale-up team Most Significant Change

Interplay of macro-level forces influencing FP, including government and donor support Community support of and demand for innovation Community knowledge of innovation Level of political support for integration Extent of integration into guiding documents and support systems Extent of service availability TIPPING POINT

Knowing what we know now… Participatory engagement of multiple actors at multiple levels worked well, including having a resource team to guide scale-up Tools gathered important info, but how they were used should be adjusted Frequency of data collection for timely feedback  Baseline/endline measures too infrequent to provide feedback to inform process: household surveys and facility assessments  Use more frequent, perhaps less rigorous, methods to allow timely feedback, eg, sentinel surveillance sites, LQAS Collect data in actionable units, eg, district level  National level sampling of facility assessments did not provide specific feedback that could be used by district supervisors

Knowing what we know now… Values can drive innovation scale-up  Innovation: Bringing explicit gender focus midway left some holes in monitoring tools; next time would bring innovation values focus more explicitly up front  People as diffusers of ideas, attitudes: Systematize periodic values mapping / political environment scans Defining impact –a sustainability tipping point  Would focus household survey data collection more on demand factors, values, and end-use

IRH documents available on (also to be posted on eval library site) A systems approach to M&E of scale-up: Report of a technical consultation, December 2012 Monitoring and evaluating scale-up of health systems innovations: Theory and Practice, 2013 Promising scale-up practices: A monitoring and evaluation compendium (forthcoming)