International comparison of health system responsiveness: the use of the hierarchical ordered probit model to adjust self-reported data Nigel Rice, Silvana Robone, Peter Smith Centre for Health Economics, University of York Project funded by the UK ESRC, Public Service Programme III Workshop FEDEA, Universidad Pompeu Fabra, 4/06/2009

Introduction (1)   International comparison has become one of the most influential levers for change in public services The benchmark of performance against relevant comparative countries enables the public sector to promote accountability to its citizens, the diffusion and uptake of innovative practice and to systematically evaluate performance. A fundamental concern of cross-national surveys is the comparability of the data collected. Many of the data that international comparisons have traditionally relied upon have been reported at broad aggregate country level Problem: difficult to disentangle the many possible reasons for observed variations between countries

Introduction (2)   Recently the focus of research has shifted towards the use of data measured at an individual level (administrative records or cross-country surveys). Measures of performance are becoming increasingly reliant on the perspective of the user. Patients’ views and opinions are recognised as an essential means for assessing the provision of health services, to stimulate quality improvements and to measure health systems performance. Traditionally, patients’ views were sought on the quality of care provided and satisfaction with health services. Recently the concept of responsiveness has been promoted as a more desirable measure to judge health systems. Responsiveness can be defined as the way in which individuals are treated and the environment in which they are treated encompassing the notion of patient experience with the health care system (Valentine et al., 2003)

Introduction (3)   ISSUE: data on Responsiveness are self-reported derived from surveys Ex: “For your [child’s] last visit, how would you rate the experience of being involved in making decisions about your health care or treatment?” Response categories: “Very good”, “Good”, “Moderate”, “Bad”, and “Very bad”. The meaning of the available response categories may be interpreted differently across population sub-groups Responses will be influenced by individuals' preferences and expectations, which vary systematically across countries, or across socio-demographic groups within a country (REPORTING HETEROGENEITY) Country A Country B

Methods (1)   Use of anchoring vignettes to address the issue of reporting heterogeneity. Vignettes = descriptions of fixed levels of a latent construct EX: “When the clinic is not busy, [Mamadou] can choose which doctor he sees. But most often it is busy and then he gets sent to whoever is free”. How would you rate [Mamadou’s] freedom to choose his health care provider? 1. Very good 2. Good 3. Moderate 4. Bad 5. Very bad Any systematic variation across individuals in the rating of the vignettes can be attributed to reporting heterogeneity (or measurement error).

Methods (2) The Hierarchical Ordered Probit Model (HOPIT)   The Hierarchical Ordered Probit Model (HOPIT) Terza (1985), Tandon et al. (2003) It allows to account for heterogeneous reporting behaviour Use of the anchoring vignettes to provide a source of external information that enables the identification of the cut-points as functions of covariates. Two parts: 1) reporting behaviour (bias) equation 2) responsiveness equation Assumptions: a) Response consistency: individuals classify the vignettes in a way that is consistent with the rating of their own experiences of the service provided. b) Vignette equivalence: “the level of the variable represented by any one vignette is perceived by all respondents in the same way and on the same unidimensional scale” (King et al., 2004, p.194).

The World Health Survey   Overview of the Survey Launched by the World Health Organisation (WHO) in 2001 AIM: strengthen national capacity to monitor critical health outputs and outcomes, provide policy-makers with the evidences to adjust their policies (Üstün, et al., 2003) 70 countries, response rates 96 - 98 % Survey modes: face to face interview (90-minute long for 53 countries and 30-minute long for 13 countries) and computer assisted telephone interviews (4 countries) Samples: randomly selected (+ 18 years), sizes between 600 and 10,000 Modular basis: health insurance, health expenditures, socio-demographics and income, health state valuations, health system responsiveness, and health system goals.

Data Dependent Variable: Responsiveness   Dependent Variable: Responsiveness Sections: Needing Health Care and General Evaluation of the Health System, Seeing Health Care Providers, Outpatients and Care at Home, Inpatient Hospital, Vignettes Domains: Autonomy, Choice, Clarity of communication, Confidentiality, Dignity, Prompt attention, Quality of basic amenities, Social support Response categories: “very good”, “good”, “moderate”, “bad” and “very bad” Independent Variables: (reporting behaviour and responsiveness equation) Education: categorical variable (7 categories) or continuous variable (number of years in education). Gender: is a dummy variable, 1 if woman, 0 if man. Income: dummy variables to indicate the tertiles of the within-country distribution of household permanent income, measured with the HOPIT model (Ferguson et al., 2003). Age: continuous variable (years) Country dummies (Mexico is the baseline)

Evidence of differential reporting behaviour (1)   Summary frequencies for the reporting of responsiveness and vignettes, World Health Survey, Clarity of Communication Mexico UK

Evidence of differential reporting behaviour (2)   Vignette ratings by socio-demographic characteristics of the respondents, World Health Survey, Mexico, Clarity of Communication Education Income Quintiles .1 .2 .3 .4 .5 .6 % primary school completed secondary school completed high school completed post graduate degree completed mean of verygood mean of good mean of moderate mean of bad mean of verybad .1 .2 .3 .4 .5 .6 % 1 2 3 4 5 mean of verygood mean of good mean of moderate mean of bad mean of verybad Gender Age .1 .2 .3 .4 .5 .6 % female male mean of verygood mean of good mean of moderate mean of bad mean of verybad .1 .2 .3 .4 .5 .6 % 10-35 36-50 51-65 66+ mean of verygood mean of good mean of moderate mean of bad mean of verybad

Comparison of OECD countries (1)   Responsiveness equation = f (socio-demographic variables + country dummy variables + interaction terms between country dummies and the socio-demographic variables) Reporting behaviour equation = f (socio-demographic variables + country dummy variables + interaction terms between country dummies and income) Coefficients and standard errors for the full set of country dummy variables used to model the cut-point thresholds, Prompt attention. Cut - P oint Thresholds Country 1 2 3 4 AUT BEL CZE DEU DNK ESP FIN FRA UK GRC HUN IRL ITA NLD PR T SVK SWE .272 (.181 ) .408 (.144) .329 (.132) .388 (.116) .533 (.138) .392 (.052) .267 (.107) .229 (.164) .342 (.110) .283 (.105) .604 (.082) .258 (.141) .565 (.227) .356 (.163) .254 (.142) .313 (.088) .352 (.109) .351 (147) .017 (.138) .043 (.122) .1 32 (.112) .394 (.131) .335 (.048) .149 (.097) .124 (.153) .062 (.102) .039 (.098) .195 (.079) .106 (.124) .273 (.218) .055 (.155) .152 (.133) .236 (.081) .056 (.103) .408 (.141) .067 (.138) .098 (.121) .178 (.112) .014 (.134) .119 (.048) .098 (.097) .051 (.155) .034 (.102) .078 (.100) .002 (.080) .146 (.124) .171 (.217) .065 (.154) .526 (.136) .079 (.082) .067 (.105) .641 (.145) .297 (.143) .435 (.124) .530 (.115) .822 (.140) .031 (.053) .282 (.102) .554 (.160) .580 (.106) .377 (.105) .539 (.083) .879 (.127) .070 (.238) .322 (.159) .588 (.169) .400 (.087) .557 (.108)

Comparison of OECD countries (2)   Ranking of OECD countries, observed and predicted probabilities of reporting “very good” responsiveness, Respect

Comparison of OECD countries (3)   Ranking of OECD countries, observed and predicted probabilities of reporting “very good” responsiveness, Confidentiality

Comparison of OECD countries (4)   Ranking of OECD countries, across all domains

Conclusions   Evidence that reporting behaviour varies systematically both across countries and across socio-demographic groups within a country Correcting for different reporting behaviour across countries affects the ranking of countries according to their health system responsiveness Once differential reporting behaviour has be accounted for and country-specific performance is benchmarked against a single country, in general, health system responsiveness is rated more favourably in OECD countries from Northern and Western Europe, than from Eastern and Southern Europe. Future research: Focus on other determinants of health system responsiveness to further aid cross-country comparison. Anchoring vignettes rely on two key assumptions: response consistency and vignette equivalence. Test the validity of these assumptions.