International comparison of health system performance and self-reported data Nigel Rice, Silvana Robone, Peter Smith Centre for Health Economics, University of York Project funded by the UK ESRC, Public Service Programme

Introduction (1)   Patients’ views and opinions are an essential means for assessing the provision of health services, to stimulate quality improvements and to measure health systems performance. Traditionally, patients’ views were sought on the quality of care provided and satisfaction with health services. Recently the concept of responsiveness has been promoted as a more desirable measure to judge health systems. Responsiveness can be defined as the way in which individuals are treated and the environment in which they are treated encompassing the notion of patient experience with the health care system (Valentine et al., 2003)

Introduction (2)   ISSUE: data on Responsiveness are self-reported derived from surveys Ex: “For your [child’s] last visit, how would you rate the experience of being involved in making decisions about your health care or treatment?” Response categories: “Very good”, “Good”, “Moderate”, “Bad”, and “Very bad”. The meaning of the available response categories may be interpreted differently across population sub-groups Responses will be influenced by individuals' preferences and expectations, which vary systematically across countries, or across socio-demographic groups within a country (REPORTING HETEROGENEITY) Country A Country B

Methods   Use of anchoring vignettes to address the issue of reporting heterogeneity. Vignettes = descriptions of fixed levels of a latent construct EX: “When the clinic is not busy, [Mamadou] can choose which doctor he sees. But most often it is busy and then he gets sent to whoever is free”. How would you rate [Mamadou’s] freedom to choose his health care provider? 1. Very good 2. Good 3. Moderate 4. Bad 5. Very bad Any systematic variation across individuals in the rating of the vignettes can be attributed to reporting heterogeneity (or measurement error). Use of the hierarchical ordered probit model (HOPIT) (Tandon et al. (2003)), that exploits the anchoring vignettes as source of external information that identify the determinants of reporting heterogeniety. Two parts: 1) reporting behaviour (bias) equation 2) responsiveness equation Assumptions: Response Consistency and Vignettes Equivalence

Data The World Health Survey   The World Health Survey Launched by the World Health Organisation (WHO) in 2001 70 countries, samples randomly selected (+ 18 years), sizes 600 - 10,000 Dependent Variable: Responsiveness Domains: Autonomy, Choice, Clarity of communication, Confidentiality, Dignity, Prompt attention, Quality of basic amenities, Social support Response categories: “very good”, “good”, “moderate”, “bad” and “very bad” Independent Variables: (reporting behaviour and responsiveness equation) Education: categorical variable (7 categories) or continuous variable (number of years in education). Gender: is a dummy variable, 1 if woman, 0 if man. Income: dummy variables to indicate the tertiles of the within-country distribution of household permanent income, measured with the HOPIT model (Ferguson et al., 2003). Age: continuous variable (years)

Evidence of differential reporting behaviour (1)   Summary frequencies for the reporting of responsiveness and vignettes, World Health Survey, Clarity of Communication Mexico UK

Evidence of differential reporting behaviour (2)   Vignette ratings by socio-demographic characteristics of the respondents, World Health Survey, Mexico, Clarity of Communication Education Income Quintiles .1 .2 .3 .4 .5 .6 % primary school completed secondary school completed high school completed post graduate degree completed mean of verygood mean of good mean of moderate mean of bad mean of verybad .1 .2 .3 .4 .5 .6 % 1 2 3 4 5 mean of verygood mean of good mean of moderate mean of bad mean of verybad Gender Age .1 .2 .3 .4 .5 .6 % female male mean of verygood mean of good mean of moderate mean of bad mean of verybad .1 .2 .3 .4 .5 .6 % 10-35 36-50 51-65 66+ mean of verygood mean of good mean of moderate mean of bad mean of verybad

Comparison of OECD countries (1)   Responsiveness equation = f (socio-demographic variables + country dummy variables + interaction terms between country dummies and the socio-demographic variables) Reporting behaviour equation = f (socio-demographic variables + country dummy variables + interaction terms between country dummies and income) Coefficients and standard errors for the full set of country dummy variables used to model the cut-point thresholds, Prompt attention. Cut - P oint Thresholds Country 1 2 3 4 AUT BEL CZE DEU DNK ESP FIN FRA UK GRC HUN IRL ITA NLD PR T SVK SWE .272 (.181 ) .408 (.144) .329 (.132) .388 (.116) .533 (.138) .392 (.052) .267 (.107) .229 (.164) .342 (.110) .283 (.105) .604 (.082) .258 (.141) .565 (.227) .356 (.163) .254 (.142) .313 (.088) .352 (.109) .351 (147) .017 (.138) .043 (.122) .1 32 (.112) .394 (.131) .335 (.048) .149 (.097) .124 (.153) .062 (.102) .039 (.098) .195 (.079) .106 (.124) .273 (.218) .055 (.155) .152 (.133) .236 (.081) .056 (.103) .408 (.141) .067 (.138) .098 (.121) .178 (.112) .014 (.134) .119 (.048) .098 (.097) .051 (.155) .034 (.102) .078 (.100) .002 (.080) .146 (.124) .171 (.217) .065 (.154) .526 (.136) .079 (.082) .067 (.105) .641 (.145) .297 (.143) .435 (.124) .530 (.115) .822 (.140) .031 (.053) .282 (.102) .554 (.160) .580 (.106) .377 (.105) .539 (.083) .879 (.127) .070 (.238) .322 (.159) .588 (.169) .400 (.087) .557 (.108)

Comparison of OECD countries (2)   Ranking of OECD countries, observed and predicted probabilities of reporting “very good” responsiveness, Respect

Conclusions   Evidence that reporting behaviour varies systematically both across countries and across socio-demographic groups within a country Correcting for different reporting behaviour across countries affects the ranking of countries according to their health system responsiveness Once differential reporting behaviour has be accounted for and country-specific performance is benchmarked against a single country, in general, health system responsiveness is rated more favourably in OECD countries from Northern and Western Europe, than from Eastern and Southern Europe. Future research: Focus on other determinants of health system responsiveness to further aid cross-country comparison. Anchoring vignettes rely on two key assumptions: response consistency and vignette equivalence. Test the validity of these assumptions.