Evaluating sustainability of programs in developing countries: What do we measure and how? LYNNE MILLER FRANCO, Vice President Technical Assistance and Evaluation, EnCompass LLC AEA, Nov. 2, 2011

Agenda Sustainability and institutionalization: what are they? The Case of Niger healthcare improvement Niger: Articulating the theory of change Niger: Measuring sustainability/institutionalization Deriving learning from this case Implications for future endeavors in measurement of sustainability and institutionalization 2

What is sustainability and institutionalization? And why do we care about it?

Sustainability/institutionalization Understanding results:  Benefits, continuation, institutionalization, capacity (Gruen et al, 2008) Understanding factors:  Design, organizational setting, environment (Shediac-Rizkallah and Bone, 1996) Understanding extent  None, precarious, weak, and routinized (Pluye et al, 2004) 4

Sustainability and Institutionalization 5 Sustainable results Institutionalized processes and capacities Structures, incentives, policy STRUCTURE PATTERNS EVENTS

Reducing post-partum hemorrhage through essential obstetric and newborn care (EONC) and quality improvement The Niger Case Study 6

. GOAL: reduce maternal/ newborn mortality by strengthening local health systems to implement and scale up high-impact interventions (AMTSL, ENC) SCALE: 52 health facilities (tertiary, district and peripheral maternities) COVERAGE: 3,000 births per month (27% of all public facility births) INTERVENTION PERIOD: July 2006-December 2008 Niger EONC Improvement Collaborative: focus, scale, and coverage

AMTSL not generally known among health workers Newborn care neither systematically done nor documented Little attention given to the environment in which newborns were cared for Surveillance of mother and newborn separate, if done at all AMTSL now fully integrated into care practiced in participating maternities Physical environment for delivery and post-partum improved: infection prevention/temperature Integrated mother and newborn post partum surveillance Situation before 2006 Situation in late 2008 EONC Collaborative Introduce evidence- based norms Build clinical and QI capacity Form QI teams Analyze the process Test changes Monitor results Conduct coaching Share changes Intervention

Sustained outcomes Reduction in post-partum hemorrhage Ultimate AimIntermediate AimPrimary DriversSecondary Drivers Sustained quality of EONC care provided to women coming to delivery in maternities

Sustaining gains for active management of the third stage of labor (AMTSL) in 20 sites in Niger

What does it look like? How will we know when we see it? Institutionalization 11

Secondary Drivers Sustained outcomes Sustained quality of EONC care Sustained implementation of essential QI activities Sustained effective processes of care, including implementation of effective operational changes Sustained enabling environment for quality of care: Drugs and supplies Clinical competency Staff engagement Sustained enabling environment for quality improvement: QI competency Team dynamics and functionality QI support from higher levels of the health system % of deliveries according to standards % operational changes implemented % of ongoing QI activities implemented Application of QI to new technical area % of key drugs/supplies available % providers formally trained in EONC Average worker engagement score % of QI tasks workers feel confident in Number of QI team meetings held % of original team members still present Primary Drivers Ultimate Aim Intermediate Aim

Continued implementation of QI tasks 13

But what about the “structure” and organizational support routines? We have examined the “events” and the “patterns” 14

Specific objectives for sustaining gains and QI institutionalization Health Facility level: Ensure availability of essential resources for quality health care Strengthen implementation of QI activities and application to new areas District and Regional level: Strengthen (technical and organizational) capacity of district and regional levels to provide adequate support to health facilities National Level: Create an enabling policy and strategic environment for implementation of QI at all levels of the health system

Health facility level institutionalization Objectives Ensure availability of essential resources Strengthen implementation of QI activities and application to new areas Categories of institutionalization-enhancing activities Ensure/maintain staff clinical competency (8) Ensure/maintain staff QI competency (6) Ensure availability drugs/supplies (6) Ensure availability QI resources (5) Make explicit clinical care expectations (7) Make explicit QI expectations/responsibilities (8) Standardize improved care processes (4) Monitor and analyze indicators (5) Ensure QI team functioning (6) Provide technical/moral support (5) Provide incentives aligned with performance (4) 16 Possible activities for maintaining staff clinical competency Daily staff meetings Post job aids Provide orientation to new staff Assign a “tutor” to each new staff Work with new staff until competent Observe staff quarterly Develop themes for in-service training Assign competent providers to train others

Levels of implementation at site level: institutionalization enhancing activities 17

What have we learned about sustainability and institutionalization? Most health facilities continue to implement some processes and activities that support continued sustainability of results, but many areas still weak District level support is weaker than Regional, but higher performing districts are all in higher performing regions Role of the National level needs strengthening related to accountability, motivation, and knowledge management 18

Sustainability and Institutionalization 19 Sustainable results Institutionalized processes and capacities Structures, incentives, policy

What does all this mean for measuring sustainability and institutionalization? What other aspects of sustainability and institutionalization need to be measured? What “markers” or intermediate milestones for institutionalization can we track during implementation ? 20

Parting Thoughts: We, as evaluators, can help programs: Define, in the beginning, what they want to see sustained in terms of results and activities Build a theory of change for sustainability and institutionalization along side of the project’s theory of change Monitor milestones for institutionalization during implementation Advocate for continued measurement of results post-intervention for learning and improvement 21

This work was supported by the American people through the United States Agency for International Development (USAID) and its Health Care Improvement (HCI) Project. The USAID Health Care Improvement Project (HCI) is managed by University Research Co., LLC (URC) under the terms of Contract No. GHN-I For more information, please contact: Lynne Franco at Thank you 22