1 The measurement and comparison of health system responsiveness Nigel Rice, Silvana Robone, Peter Smith (Centre for Health Economics, University of York) HEDG Seminar York, 4 June 2008

2 Outline Introduction, Study Aim Data Econometric model Results: Descriptive statistics Estimates within countries Estimates across-countries, HDI groups Estimates across countries, overall Responsiveness ranking Conclusions

3 Introduction (1) Patients’ views and opinions are an essential means for assessing the provision of health services, to stimulate quality improvements and to measure health systems performance. Traditionally, patients’ views were sought on the quality of care provided and satisfaction with health services. Recently the concept of responsiveness has been promoted as a more desirable measure to judge health systems. Responsiveness can be defined as the way in which individuals are treated and the environment in which they are treated encompassing the notion of patient experience with the health care system (Valentine et al., 2003)

4 Introduction (2) Outcome measurements enable institutions to compare and contrast their performance to that of others, including at a macro level, the performance obtained in other countries. Studies aimed at comparative inference have rarely taken into consideration possible variations in cultural expectations that might impact on reporting behaviour (Blendon et al., 2003). ISSUE: data on Responsiveness are self-reported. Ex: “For your [child’s] last visit, how would you rate the experience of being involved in making decisions about your health care or treatment?” Response categories: “Very good”, “Good”, “Moderate”, “Bad”, and “Very bad”. Individuals, when faced with survey questions about the functioning of health systems, may systematically interpret the meaning of the available response categories differentially across population sub- groups (Sadana et al., 2002).

5 Introduction (3) Responses will be influenced by individuals' preferences and expectations, which vary systematically across countries, or across socio-demographic groups within a country (REPORTING HETEROGENEITY). Fixed level of underlying responsiveness is unlikely to be rated equally across sub-groups of interest (see Tandon et al., 2003). EX: Reporting heterogeneity has been debated usually with regard to measures of health status (for example, Jürges, 2007, Kapteyn et al., 2007; Bago d’Uva et al., 2007; Lindeboom and van Doorslaer, 2004; Iburg et al., 2002). Few studies with regard to responsiveness (Valentine et al. (2003), Puentes Rosas et al. (2006)).

6 Introduction (4) Use of anchoring vignettes to address the issue of reporting heterogeneity. Vignettes = descriptions of fixed levels of a latent construct, such as respons. EX: “When the clinic is not busy, [Mamadou] can choose which doctor he sees. But most often it is busy and then he gets sent to whoever is free”. How would you rate [Mamadou’s] freedom to choose his health care provider? 1. Very good 2. Good 3. Moderate 4. Bad 5. Very bad - Any systematic variation across individuals in the rating of the vignettes can be attributed to reporting heterogeneity (or measurement error). Responses to the vignettes allow the researcher to model the response scales as a function of the characteristics of respondents.

7 Study Aim OUR AIM: explore the utility of using information from vignettes to adjust self-reports of health system responsiveness. Use of nine countries from the World Health Survey 1)evaluate the presence of reporting bias across socio-economic groups within countries, and how it is related to the characteristics of the individuals. Human Development IndexHDI 2) to aid cross country comparison, we aggregate these country into three groups according to their Human Development Index (HDI) level (United Nations Development Programme, 2006) and we evaluate the presence of differences in reporting among countries in the same group. 3) we assess the presence of different reporting behaviour among the three HDI groups of countries. 4) we try to evaluate if the issue or reporting bias affect the ranking of health systems responsiveness.

8 DATA: The World Health Survey (1) Overview of the Survey Launched by the World Health Organisation (WHO) in 2001 AIM: strengthen national capacity to monitor critical health outputs and outcomes, provide policy-makers with the evidences to adjust their policies (Üstün, et al., 2003) 70 countries, response rates % Survey modes: face to face interview (90-minute long for 53 countries and 30-minute long for 13 countries) and computer assisted telephone interviews (4 countries) Samples: randomly selected (+ 18 years), sizes between 600 and 10,000 Modular basis: health insurance, health expenditures, socio- demographics and income, health state valuations, health system responsiveness, and health system goals.

9 DATA: The World Health Survey (2) The Responsiveness Module: Developed from an extensive consultation process. Origin in the Multi-Country Survey Study on Health and Responsiveness ( ) Sections: Needing Health Care and General Evaluation of the Health System, Seeing Health Care Providers, Outpatients and Care at Home, Inpatient Hospital, Vignettes Responsiveness Domains: Autonomy Choice Clarity of communication Confidentiality Dignity Prompt attention, Quality of basic amenities Social support

10 DATA: Country selection 9 countries within the WHS: Mexico, Spain, Malaysia, India, Philippines, SriLanka, Burkina, Malawi and Ethiopia Selected on the basis of 4 criteria: Long version (90-minute in-household) questionnaire Satisfy well a set of psychometric properties (feasibility, reliability and validity) for the responsiveness module represent geographical areas characterized by different levels of development. Use of HDI to stratify the countries into three groups, high, medium and low HDI group High sample size in comparison to other countries belonging to the same HDI group.

11 DATA: Variables (1) Dependent Variables: Only 4 out of the 8 domains: Respect, Confidentiality, Quality of Facilities and Clear Communication Domains considered as most important by the respondents in the nine countries selected. For each domain, respondents were asked up to 2 questions about their experiences of contact with health systems. The response categories: “very good”, “good”, “moderate”, “bad” and “very bad”.

12 DATA: Variables (2) Independent Variables: Individual level - Education: categorical variable (7 categories) or continuous variable (number of years in education). - Gender: is a dummy variable, 1 if woman, 0 if man. - Income: categorical variable (quintiles of the distribution of household permanent income, 1 if in the lowest income quintile, 5 if in the highest one). Permanent income measured with the HOPIT model (Ferguson et al., 2003). - Age: continuous variable (years) Country level (data provided by the UNDP for the year 2001) -Health expenditures per capita (both private and public), in current 1,000 US$ -GDP per capita, in constant ,000 US$ (to consider PPP). Country and HDI group dummies

13 Econometric model (1) The Hierarchical Ordered Probit Model (HOPIT) Terza (1985), Tandon et al. (2003) It allows to account for heterogeneous reporting behaviour Use of the anchoring vignettes to provide a source of external information that enables the identification of the cut-points as functions of covariates. Assumptions: a) Response consistency It is assumed that individuals classify the vignettes in a way that is consistent with the rating of their own experiences of the service provided. b) Irrelevance of own provider responsiveness or vignette equivalence. It is assumed that “the level of the variable represented by any one vignette is perceived by all respondents in the same way and on the same unidimensional scale” (King et al., 2004, p.194).

14 Econometric model (2) Reporting behaviour equation (1) = underlying health system responsiveness for vignette k, rated by individual i = vector of vignettes = vector of parameters = idiosyncratic error term we observe the vignette rating, on a five point scale ranging from ‘Very bad’ to ‘Very good’. We assume the observed category of is related to through: (2) we allow the cut-points to be functions of covariates, such that: (3)

15 Econometric model (3) Responsiveness equation = health system responsiveness faced by individual = represents a set of regressors correlated with responsiveness ( might be a sub-set of ) for = observed categorical response are defined by (3) with fixed and it is assumed that and are independent for all and the probabilities associated with each of the 5 categories are given by: where is the cumulative standard normal distribution.

16 Results: Descr. Stat. (1) Summary frequencies for the reporting of vignettes: Mexico. Clarity of communication

17 Results: Descr. Stat. (2) Ratings of vig.2 for “Clarity of com”, Mexico, by: 1) Education 2) Income 3) Gender 4) Age

18 Results: within countries (1) Tests of homogenous reporting and parallel cut-point shift (p-value for Wald Test under H 0 ) AllInc.WomenAgeEduc.AllInc.WomenAgeEduc. Dignityrespect privacy Communicationclear explanations time for questions Confidentialitytalk privately confidential info Facilitiescleanliness space AllInc.WomenAgeEduc.AllInc.WomenAgeEduc. Dignityrespect privacy Communicationclear explanations time for questions Confidentialitytalk privately confidential info Facilitiescleanliness space Malaysia Mexico HeterogeneityParallel cut-point shift HeterogeneityParallel cut-point shift

19 Results: within countries(2) permanent income Estimated coefficients and standard errors of permanent income of the cut-points

20 Results: within countries(3) education Estimated coefficients and standard errors of education of the cut-points

21 Results: within countries(4) Estimated coefficients and standard errors in the OPROBIT and HOPIT model, for Mexico: Income Education OPROBITHOPIT Dignity respectcoeff st. err privacycoeff st. err Communication clear explanationcoeff st. err time for questionscoeff st. err Confidentiality talk privatelycoeff st. err confidential infocoeff st. err Facilities cleanlinesscoeff st. err spacecoeff st. err

22 Results: within countries(5) Marginal effects of education in the OPROBIT and HOPIT model (the increase in the probability of being in very good responsiveness due to an increase in one year of education) ME of edu on very good responsiveness -0.05% 0.00% 0.05% 0.10% 0.15% 0.20% 0.25% 0.30% Me_edu OPROBIT Me_edu HOPIT

23 Results: within countries(6) Ex-ante frequencies, ex-post frequencies, computed through the OPROBIT and HOPIT model, of reporting each of the five response categories Item ex ante prob ex post prob PROBIT ex post prob HOPIT ex ante prob ex post prob PROBIT ex post prob HOPITex ante prob ex post prob PROBIT ex post prob HOPIT RESPECTvery bad0.3% 0.0%0.7% 0.0%0.7% 0.0% bad2.4%2.5%2.6%3.1% 2.9%0.9% 1.7% moderate10.7%10.8%12.1%8.7% 12.4%6.2%6.3%7.7% good68.5%67.7%65.5%71.2% 67.0%61.5%61.4%59.1% very good18.2%18.8%19.8%16.3% 17.6%30.8%30.7%31.6% PRIVACYvery bad0.4% 0.0%0.4% 0.0%0.2% 0.0% bad2.1%2.2%2.4%2.3% 1.9% 1.4% moderate8.3%8.4%10.0%7.4% 10.5%5.2%5.4%7.4% good69.3%69.1%66.8%73.5%73.4%70.2%66.1%65.8%63.6% very good19.9%20.0%20.8%16.5%16.4%17.5%26.6%26.7%27.6% CLEARCOMvery bad0.2% 0.0%0.7%0.8%0.0%0.7%0.8%0.1% bad2.4%2.5%1.5%3.2% 2.5%2.6%2.7%3.4% moderate8.3% 11.6%7.9% 13.3%11.0%11.2%13.0% good70.0% 67.0%71.8% 66.2%60.3%60.1%57.2% very good19.1%19.0%19.9%16.3% 18.0%25.3%25.2%26.3% TIMEQUESTvery bad0.2% 0.0%0.7% 0.0%0.9% 0.2% bad3.5% 2.7%3.7% 2.8%4.5%4.7%5.6% moderate12.1%11.6%14.6%8.5% 14.1%13.8%13.7%15.1% good68.2%68.5%65.6%71.0% 65.3%59.9%59.7%57.3% very good16.0%16.1%17.1%16.1%16.0%17.8%21.0% 21.8% TALKPRIVvery bad0.3% 0.0%0.6% 0.0%1.0%1.1%0.0% bad2.6%2.4% 4.2% 3.9%3.7%3.8%5.2% moderate12.0%11.6%13.9%10.6% 15.8%10.5%10.7%13.6% good69.2%69.3%66.0%70.6% 64.1%62.6%62.2%57.0% very good16.0%16.4%17.7%14.0% 16.2%22.3%22.2%24.2% CONFINFOvery bad0.1% 0.0%1.2% 0.0%0.8% 0.0% bad1.0%1.1%0.7%2.7% 4.0%1.7%1.8%2.7% moderate8.3%8.7%10.4%10.0% 15.1%7.7% 11.2% good70.0%70.2%68.2%71.9% 63.8%67.8%67.7%61.9% very good20.6%20.0%20.7%14.3%14.2%17.1%22.0% 24.3% CLEANvery bad0.2% 0.0%0.7% 0.0%0.5% 0.1% bad1.3%1.4%1.3%3.3%3.2%3.1%1.3% 2.9% moderate7.7%7.8%8.9%10.0%9.9%13.9%10.6%10.5%9.0% good67.2%67.1%65.9%70.1% 65.4%60.2% 60.7% very good23.6%23.5%23.9%15.9%16.1%17.6%27.4% SPACEvery bad0.1% 0.0%0.8% 0.0%1.9%2.0%0.7% bad1.6%1.5%1.4%4.7%4.8%4.4%6.6%6.7%8.7% moderate14.2%14.5%15.0%11.8%11.7%16.8%16.3% 17.3% good67.6%67.4%67.0%68.1%68.0%62.1%54.3% 51.8% very good16.5% 16.6%14.6%14.7%16.6%20.9%20.8%21.6% SPAINMALAYSIAMEXICO

24 Results: within countries(7) RESET test

25 Results: across HDI groups Tests of homogenous reporting and parallel cut-point shift

26 Results: overall countries Tests of homogenous reporting and parallel cut-point shift

27 Results: resp ranking (1) Kendall tau rank correlation coefficients Kendall tau rank correlation coefficients and related p-values for the ranking of responsiveness in the high HDI countries, computed through the OPROBIT and HOPIT (individual country models) latent value of responsiv. prob of “very good”/“good” responsiv.

28 Results: resp ranking (2) Country ranking or responsiveness predicted through the OPROBIT and the HOPIT (individual country models) for Clarity of communication and Time for questions latent value of responsiv. prob of “very good”/“good” responsiv

29 Results: resp ranking (3) Comparison of predicted responsiveness for Clarity of communication obtained through the OPROBIT and HOPIT (high HDI group model) in Spain, Mexico and Malaysia Predicted latent responsiveness Predicted responsiveness frequencies Spain 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% OPROBIT HOPIT

30 Conclusions Heterogeneity in reporting behaviour exists, and it appears to be a function of individual socioeconomic characteristics (such as income and education), and characteristics at country and macro-regional levels (that we capture through country and HDI group dummies) Adjusting for reporting bias impacts on: 1) The estimated coefficients of the responsiveness mean function when results from the HOPIT model are compared to those from an ordered probit regression. 2) The marginal effects of individual socioeconomic variables (such as education) on responsiveness 3)The ex-post frequencies of reporting each of the five response categories Reporting heterogeneity affects the ranking of high HDI countries according to their responsiveness level. The magnitude and the level of significance in the difference of ranking with or without considering for reporting heterogeneity varies according to the way we measure responsiveness.