A new tool for monitoring quality of care: UNHCR’s Reproductive Health Scorecard Hannah Tappis, DrPH Senior Monitoring, Evaluation & Research Advisor Jhpiego / Johns Hopkins Bloomberg School of Public Health IAWG Annual Meeting February 25-27, 2015 Jordan

Overview Background Toolkit Results of field testing Recommendations for implementation

Background Existing Quality of Care Assessment UNHCR Balanced Scorecard in use since 2011 Intended for use by UNHCR and implementing partner management, but can be adapted Designed to provide a multi-faceted measure of quality of care and identify areas for improvement The balanced scorecard approach is a common performance management tool in a variety of fields, ranging from business to education and healthcare. Balanced Scorecards have been used to measure and manage both health service and public health programs in a variety of settings. In 2010-11, UNHCR collaborated with Johns Hopkins Bloomberg School of Public Health’s Center for Refugee and Disaster Response (JHSPH) to develop a Balanced Scorecard to assess the quality of care provided in primary health care facilities in refugee camps, which has been successfully implemented in twelve countries. In 2014, UNHCR identified reproductive health as an area where additional monitoring and evaluation of services is required and partnered with JHSPH to develop a Reproductive Health Scorecard supplement to the existing Balanced Scorecard.

Background New Reproductive Health Scorecard The Reproductive Health Scorecard is an optional supplement to UNHCR’s Balanced Scorecard for assessing the quality of care at primary health care facilities. New Reproductive Health Scorecard Optional supplement to existing tools Covers multiple RH service areas Can be conducted in <2 days (including assessment of other primary health care services) User-friendly Provides actionable information to inform programming Indicators were developed based on UNHCR guidelines for reproductive health programming, and aligned to the extent possible with the World Health Organization’s Service Availability and Readiness Assessment (SARA) tool and recommendations of the 2013 Consultation on Improving Measurement of the Quality of Maternal, Newborn and Child Care in Health Facilities and 2014 Every Newborn Action Plan. Draft data collection tools were then developed, following the structure and format of established Balanced Scorecard for primary health care services, and reviewed by UNHCR Regional Reproductive Health and HIV Coordinators before field-testing with partner agencies in October-November 2014.

Toolkit Six Tools: Five Scorecard Domains: Health facility observation ANC observation ANC exit interviews PNC observation PNC exit interviews Health worker interviews Five Scorecard Domains: RH service capacity (inputs) RH service capacity (processes) RH service provision RH client satisfaction RH staff satisfaction The Reproductive Health Scorecard Toolkit is not intended to be a standalone activity. Although this presentation focuses on the new RH Scorecard contents, it should be conducted as part of the broader Balanced Scorecard assessment of primary health care services, which also includes observation of IMCI, exit interviews with parents of sick children under five, and interviews with health workers from other areas of the health facility.

Field Testing in Thailand – Oct/Nov 2014 Partner: Malteser International Locations: Mae La Oo, Mae La Ra Luang Data collectors: 4 Language: English/Karen Data collection: 2 days per camp

Field Testing in Thailand – Oct/Nov 2014 GAPS IDENTIFIED Camp 1: FP commodities, fetoscope, syphilis test kit, MVA, ARVs, PEP kit Camp 2: FP commodities, fetoscope, ARVs, PEP kit

Field Testing in Thailand – Oct/Nov 2014 GAPS IDENTIFIED Camp 1: ANC: blood testing, urine testing, TT PNC: checking temperature, breathing and eyes, counseling on thermal care Camp 2: ANC: client recognition of danger signs PNC: checking temperature and breathing, counseling on thermal care,

Field Testing in Rwanda – Nov 2014 Partners: Africa Humanitarian Action American Refugee Committee Locations: Kiziba, Gihembe Data collectors: 6 Language: English/French Data collection: <2 days per camp

Field Testing in Rwanda – Nov 2014 GAPS IDENTIFIED Camp 1: MgSO4, recent training on MNH services Camp 2: MgSO4, antenatal corticosteroids, EmONC guidelines, CCSAS guidelines, handwashing soap in delivery room, EmONC referral protocols, maternal death review system

Field Testing in Rwanda – Nov 2014 GAPS IDENTIFIED Camp 1: PNC: client ability to identify danger signs RH staff satisfaction Camp 2: ANC: client recognition of danger signs PNC: checking temperature and breathing, counseling on thermal care,

General Findings The Reproductive Health Scorecard is an effective and acceptable method for assessing the quality of services in protracted refugee settings. Overall service quality is good, but areas needing attention vary across and within countries. Detailed results (sub-items for each indicator) may not be reported in the scorecard, but are important for programming

Recommendations for Implementation Take time to review tools in advance Carefully consider selection of data collectors to minimize bias and maximize learning opportunities Practice observations! Encourage data collectors to work in pairs Consider including assessment as part of routine program review (1x/year or more)

Thank you UNHCR JHSPH Sathya Doriaswamy Shannon Doocy Nadine Cornier Catherine Lee Herve Isambert Yasmina Aules Vincent Kahi Ko Myo Eugene Paik Jonathan Akuoku Partners Other Malteser International All health facility staff and client participants American Refugee Committee Africa Humanitarian Action

For additional information about the UNHCR Balanced Scorecard Toolkit, please visit http://twine.unhcr.org (Reproductive Health Scorecard details coming soon)