Analysis of Evaluations of Health System & Policy Interventions in India Lalit Dandona Distinguished Research Professor Public Health Foundation of India Professor, Global Health Institute for Health Metrics and Evaluation University of Washington 26 February 2014

A systematic assessment of the worth or merit of some object or activity Evaluation Definition & Types Trochim, Brownson et al, Evaluating the program or policy, OUP, 2003 o Needs o Process o Outcome o Impact

Without controls o Descriptive case study o Cross-sectional o Before and after Evaluation Designs With controls o Cross-sectional o Before and after o Quasi-experimental o Randomized controlled trial

 Extensive web searches of Indian and international organizations to identify evaluations of health system or policy interventions in India that were reported between 2001 and 2008  Standardized review of each evaluation report o Type of evaluation o Commissioning and conducting agencies o Health system/policy component evaluated o Disease or health condition evaluated o Evaluation study design, type of data, methods and analysis o Conclusion positive, neutral or negative o Quality score for each evaluation report Methods Dandona et al, National Medical Journal of India 2011; 24:

 219 evaluation reports of population health interventions identified in the public domain: 6% were needs, 22% process, 42% outcome and 30% impact evaluations  76% of the evaluations were commissioned by international organizations  43.3% of the evaluations were conducted by international organizations, 41.4% by Indian organizations and 15.3% by collaborations between international and Indian organizations Evaluations in India:

Design and Analysis of Outcome and Impact Evaluations Study designNo. (%) of outcome and impact evaluations Without controls129 (81.6) Descriptive case study30 (19.0) Cross-sectional without control 29 (18.4) Before & after without control70 (44.3) With controls29 (18.4) Cross-sectional with control7 (4.4) Before & after with control19 (12.0) Quasi-experimental design or randomised controlled trial 3 (1.9) Total158 (100) 72.1% of the outcome/impact evaluations presented only basic tabulations and only 11.6% attempted multivariate analysis

Source: WHO, 2007 Components of Health System

Topics Covered by Evaluations India Dandona et al, National Medical Journal of India 2011; 24:

Diseases/Condition Covered by Evaluations Disease/condition N (%) Reproductive and child health 92 (42.0) HIV/AIDS 30 (13.7) Tuberculosis 11 (5.0) Nutrition 9 (4.1) Injury 7 (3.2) Diabetes and cardiovascular disease 5 (2.3) Oral health 4 (1.8) Vector-borne diseases 3 (1.4) Blindness 2 (0.9) Mental health 2 (0.9) Leprosy 2 (0.9) Iodine deficiency 1 (0.5) Others* 9 (4.1) Cross-cutting† 42 (19.2) Total219 (100)

Causes of Disease Burden in India, 2010 Data Source: Global Burden of Disease Study

Quality of Evaluations Organization conducting evaluation Evaluation report quality, N (%) Total PoorFairGood Indian organizations 28 (31.5)42 (47.2)19 (21.3)89 (100) International organizations 40 (43.0) 13 (14.0)93 (100) Collaborations between Indian and international organizations 2 (6.1)17 (51.5)14 (42.4)33 (100) Total70 (32.6)99 (46.0)46 (21.4)215 (100)

Involvement of Implementing Agency in Evaluation In 54% of the evaluation the implementing agency was involved in conducting the evaluations

 Commissioning of evaluations until recently dominated by international agencies  Production of quality evaluation research inadequate though improving  Access to evaluation findings sometimes restricted by commissioning agencies  Capacity to use evaluation findings for policy making and system development poorly developed  Mechanisms to manage conflicts of interest not in place Challenges in Improving Evaluations in India

 Hasten momentum for commissioning of relevant evaluations by LMIC governments and institutions  Develop and put in place strategic mechanisms for enhancing evaluation research capacity in LMICs  Recognize and develop the key role of an adequate routine health information system in facilitating large scale evaluations  Mandate open access of evaluation findings supported by public resources  Develop dissemination mechanisms that facilitate a balanced use of evaluation findings by policy makers  Develop accountability mechanisms that make distortions by conflicts of interest difficult Key Issues that Need to be Addressed

Source: Health Metrics Network, 2008 Data Sources for Health Information System Data Sources

Disease Burden Data Source: Global Burden of Disease Study Relative DALYs Per Capita