1. A new tool for monitoring quality of care:
UNHCR’s Reproductive Health Scorecard
Hannah Tappis, DrPH
Senior Monitoring, Evaluation & Research Advisor
Jhpiego / Johns Hopkins Bloomberg School of Public Health
IAWG Annual Meeting
February 25-27, 2015
Jordan

2. Overview Background Toolkit Results of field testing
Recommendations for implementation

3. Background Existing Quality of Care Assessment
UNHCR Balanced Scorecard in use since 2011
Intended for use by UNHCR and implementing partner management, but can be adapted
Designed to provide a multi-faceted measure of quality of care and identify areas for improvement
The balanced scorecard approach is a common performance management tool in a variety of fields, ranging from business to education and healthcare. Balanced Scorecards have been used to measure and manage both health service and public health programs in a variety of settings. In , UNHCR collaborated with Johns Hopkins Bloomberg School of Public Health’s Center for Refugee and Disaster Response (JHSPH) to develop a Balanced Scorecard to assess the quality of care provided in primary health care facilities in refugee camps, which has been successfully implemented in twelve countries. In 2014, UNHCR identified reproductive health as an area where additional monitoring and evaluation of services is required and partnered with JHSPH to develop a Reproductive Health Scorecard supplement to the existing Balanced Scorecard.

4. Background New Reproductive Health Scorecard
The Reproductive Health Scorecard is an optional supplement to UNHCR’s Balanced Scorecard for assessing the quality of care at primary health care facilities.
New Reproductive Health Scorecard
Optional supplement to existing tools
Covers multiple RH service areas
Can be conducted in <2 days (including assessment of other primary health care services)
User-friendly
Provides actionable information to inform programming
Indicators were developed based on UNHCR guidelines for reproductive health programming, and aligned to the extent possible with the World Health Organization’s Service Availability and Readiness Assessment (SARA) tool and recommendations of the 2013 Consultation on Improving Measurement of the Quality of Maternal, Newborn and Child Care in Health Facilities and 2014 Every Newborn Action Plan. Draft
data collection tools were then developed, following the structure and format of established Balanced Scorecard for primary health care services, and reviewed by UNHCR Regional Reproductive Health and HIV Coordinators before field-testing with partner agencies in October-November 2014.

5. Toolkit Six Tools: Five Scorecard Domains: Health facility observation
ANC observation
ANC exit interviews
PNC observation
PNC exit interviews
Health worker interviews
Five Scorecard Domains:
RH service capacity (inputs)
RH service capacity (processes)
RH service provision
RH client satisfaction
RH staff satisfaction
The Reproductive Health Scorecard Toolkit is not intended to be a standalone activity. Although this presentation focuses on the new RH Scorecard contents, it should be conducted as part of the broader Balanced Scorecard assessment of primary health care services, which also includes observation of IMCI, exit interviews with parents of sick children under five, and interviews with health workers from other areas of the health facility.

6. Field Testing in Thailand – Oct/Nov 2014
Partner: Malteser International
Locations: Mae La Oo,
Mae La Ra Luang
Data collectors: 4
Language: English/Karen
Data collection: 2 days per camp

7. Field Testing in Thailand – Oct/Nov 2014
GAPS IDENTIFIED
Camp 1: FP commodities, fetoscope, syphilis test kit, MVA, ARVs, PEP kit
Camp 2: FP commodities, fetoscope, ARVs, PEP kit

8. Field Testing in Thailand – Oct/Nov 2014
GAPS IDENTIFIED
Camp 1: ANC: blood testing, urine testing, TT
PNC: checking temperature, breathing and eyes, counseling on thermal care
Camp 2: ANC: client recognition of danger signs
PNC: checking temperature and breathing, counseling on thermal care,

9. Field Testing in Rwanda – Nov 2014
Partners: Africa Humanitarian Action
American Refugee Committee
Locations: Kiziba, Gihembe
Data collectors: 6
Language: English/French
Data collection: <2 days per camp

10. Field Testing in Rwanda – Nov 2014
GAPS IDENTIFIED
Camp 1: MgSO4, recent training on MNH services
Camp 2: MgSO4, antenatal corticosteroids, EmONC guidelines, CCSAS guidelines, handwashing soap in delivery room, EmONC referral protocols, maternal death review system

11. Field Testing in Rwanda – Nov 2014
GAPS IDENTIFIED
Camp 1: PNC: client ability to identify danger signs
RH staff satisfaction
Camp 2: ANC: client recognition of danger signs
PNC: checking temperature and breathing, counseling on thermal care,

12. General Findings
The Reproductive Health Scorecard is an effective and acceptable method for assessing the quality of services in protracted refugee settings.
Overall service quality is good, but areas needing attention vary across and within countries.
Detailed results (sub-items for each indicator) may not be reported in the scorecard, but are important for programming

13. Recommendations for Implementation
Take time to review tools in advance
Carefully consider selection of data collectors to minimize bias and maximize learning opportunities
Practice observations!
Encourage data collectors to work in pairs
Consider including assessment as part of routine program review (1x/year or more)

14. Thank you UNHCR JHSPH Sathya Doriaswamy Shannon Doocy Nadine Cornier
Catherine Lee
Herve Isambert
Yasmina Aules
Vincent Kahi
Ko Myo
Eugene Paik
Jonathan Akuoku
Partners
Other
Malteser International
All health facility staff and client participants
American Refugee Committee
Africa Humanitarian Action

15. For additional information about the UNHCR Balanced Scorecard Toolkit,
please visit
(Reproductive Health Scorecard details coming soon)