1. INVESTMENT CLIMATEDEVELOPMENT IMPACT EVALUATION INITIATIVE Risk-based support and compliance reform for the Kenyan health sector KENYA Impact Evaluation Concept Note INNOVATIONS IN INVESTMENT CLIMATE REFORMS AN IMPACT EVALUATION WORKSHOP PARIS, NOVEMBER 2012

2. JAMBO!

3. Objective:  To provide transparent and cost-effective inspection and support for improving compliance with Joint Health Inspections Checklist  To reduce discretion of inspection through randomization and thereby provide a more uniform basis for a risk-based inspections regime, and increase efficiency in the process. Motivation  Kenya Constitution 2010: The right to the highest attainable level of healthcare  Fragmented inspections systems lacking a mechanism for joint inspections for regulatory groups  Official Gazettment of the IFC supported Joint Health Inspections Checklist and the drafting of the Health Bill & Health Professionals Council Bill, 2012.  The results of the IFC-WHO/ PharmAccess/ MOH Minimum Patient Safety Compliance Survey indicate a strong need to take action to improve patient safety Motivation: Health Facility Inspections Reform

4. Evaluation Question QuestionIntervention 1. What is the effect of high frequency inspections on improved quality of service for high risk facilities (measured by compliance with the Joint Health Inspections Checklist requirements) ? What is the optimal frequency for low risk facilities? Inspections 2. What is the optimal effect of distance support for high-risk facilities in improving compliance, and client satisfaction? M-health 3. What is the effect on compliance of adding a SWAT team for High Risk Facilities? M-health + SWAT

5. Evaluation Design 1: TreatmentHigh Risk FacilitiesLow Risk Facilities Inspection Frequency (% of chance inspected per year) 200% (every 6 months) 200% vs. 100% M-health support: -Call centre -SMS feedback loops -Quiz -Training manuals M-health support SWAT team -Rapid intense training -Mentoring -Follow-up support M-Health + SWAT team

6. Evaluation Design 2: Randomization Sample of High-risk health institutions High- frequency inspections M-health + SWAT M-health Control Group 8,000 registered health institutions in Kenya (sample size n=1000) Sample of Low-risk health institutions BASELINE DATA COLLECTION

7. Evaluation Design 3: Randomization Sample of High-risk health institutions High- frequency inspections Medium- frequency inspections Control Group 8,000 registered health institutions in Kenya (sample size n=1000) Sample of Low-risk health institutions BASELINE DATA COLLECTION

8.  Use of Service Provision Assessment/ Master Facility List raw data to inform sample size calculations.  Sample size approx n=1000  Sampling and randomization at institution (clinic/hospital/etc) out of 8,000 institutions in Kenya keeping up National Representative Sample. Current Background Data availability:  Existing data and records from health inspections on selected facilities using the previous separate checklists  Data from the IFC-WHO joint Minimum Patient Safety Standards Survey conducted by IPSOS and with the technical assistance of PharmAccess and MOH on a sample of 500 private and public facilities in Kenya Sampling and Data

9. Diagnostic Tools/ Data Collection: Joint Health Inspections (JHI) Checklist Tool ▪ Data collection on JHI Tool completed at facility at Baseline and at End-line ▪ Tool used to identify high risk facilities (HRF) and low risk facilities (LRF) through a set of risk-based Guidelines. ▪ Facilities self-assess every year on the JHI checklist and submit to the regulatory authorities; penalties for inaccurate reporting. ▪ Mid-Line Rapid Assessment Tool (RAT) administration: Administration of a shortened tool developed by factor analysis  RAT varied yearly on a random blinded basis Patient Exit Survey (mobile enabled)  Conducted at Base-line and End-line on all groups to determine movement in standard of service level as a result of interventions

10. Core Government Team: Policy, Implementation, Coordination Permanent SecretaryMinistry of Medical Services Director of Medical Services Dr. Judith BwonyaMinistry of Medical Services Mrs. Elizabeth OywerRegistrar, Nursing Council of Kenya Mr. Daniel YumbyaCEO, Medical Practitioners and Dentists Board Mr. Pepela WanjalaMinistries of Health, M&E coordinator CEOs/ Registrars of the four other regulatory Boards and Councils Researchers: Research Design and Oversight Dr. Tomomi TanakaArizona State University Dr. Jeffery TannerIEG – World Bank Francisco CamposWorld Bank, Africa FPD IFC personnel:In-Country Coordination and Research Input Ms. Shakirah HudaniIFC-Health in Africa Mr. Syed Estem IslamIFC- IC M&E Project Team

11. Timeline November 2012 – June 2013:Conceptual development June 2013 – November 2013:Pre-implementation phase (logistical arrangements, Risk matrix and Guidelines, RAT, training of inspectors) December 2013:Baseline collection of data on inspections performance with the Joint Health Inspections Checklist for all facilities (HRF and LRF) January 2014:Implementation begins – Y1 Begin implementation of SWAT and M- health support for HRF Differential conduction of inspections (HRF and LRF) January 2015:Implementation continues – Y2 December 2015:End-line data collection for all facilities (HRF and LRF) January 2016:Post-implementation evaluation phase

12. INVESTMENT CLIMATEDEVELOPMENT IMPACT EVALUATION INITIATIVE Asante Sana!! Thank you! Merci. KENYA Impact Evaluation Concept Note INNOVATIONS IN INVESTMENT CLIMATE REFORMS AN IMPACT EVALUATION WORKSHOP PARIS, NOVEMBER 2012