1. International comparison of health system responsiveness: the use of the hierarchical ordered probit model to adjust self-reported data
Nigel Rice, Silvana Robone, Peter Smith
Centre for Health Economics, University of York
Project funded by the UK ESRC, Public Service Programme
III Workshop FEDEA, Universidad Pompeu Fabra, 4/06/2009

2. Introduction (1)
International comparison has become one of the most influential levers for change in public services
The benchmark of performance against relevant comparative countries enables the public sector to promote accountability to its citizens, the diffusion and uptake of innovative practice and to systematically evaluate performance.
A fundamental concern of cross-national surveys is the comparability of the data collected.
Many of the data that international comparisons have traditionally relied upon have been reported at broad aggregate country level
Problem: difficult to disentangle the many possible reasons for observed variations between countries

3. Introduction (2)
Recently the focus of research has shifted towards the use of data measured at an individual level (administrative records or cross-country surveys).
Measures of performance are becoming increasingly reliant on the perspective of the user. Patients’ views and opinions are recognised as an essential means for assessing the provision of health services, to stimulate quality improvements and to measure health systems performance.
Traditionally, patients’ views were sought on the quality of care provided and satisfaction with health services. Recently the concept of responsiveness has been promoted as a more desirable measure to judge health systems.
Responsiveness can be defined as the way in which individuals are treated and the environment in which they are treated encompassing the notion of patient experience with the health care system (Valentine et al., 2003)

4. Introduction (3)
ISSUE: data on Responsiveness are self-reported derived from surveys
Ex: “For your [child’s] last visit, how would you rate the experience of being involved in making decisions about your health care or treatment?”
Response categories: “Very good”, “Good”, “Moderate”, “Bad”, and “Very bad”.
The meaning of the available response categories may be interpreted differently across population sub-groups
Responses will be influenced by individuals' preferences and expectations, which vary systematically across countries, or across socio-demographic groups within a country (REPORTING HETEROGENEITY)
Country A
Country B

5. Methods (1)
Use of anchoring vignettes to address the issue of reporting heterogeneity.
Vignettes = descriptions of fixed levels of a latent construct
EX: “When the clinic is not busy, [Mamadou] can choose which doctor he sees. But most often it is busy and then he gets sent to whoever is free”. How would you rate [Mamadou’s] freedom to choose his health care provider? 1. Very good 2. Good 3. Moderate 4. Bad 5. Very bad
Any systematic variation across individuals in the rating of the vignettes can be attributed to reporting heterogeneity (or measurement error).

6. Methods (2) The Hierarchical Ordered Probit Model (HOPIT)
The Hierarchical Ordered Probit Model (HOPIT)
Terza (1985), Tandon et al. (2003)
It allows to account for heterogeneous reporting behaviour
Use of the anchoring vignettes to provide a source of external information that enables the identification of the cut-points as functions of covariates.
Two parts: 1) reporting behaviour (bias) equation
2) responsiveness equation
Assumptions:
a) Response consistency: individuals classify the vignettes in a way that is consistent with the rating of their own experiences of the service provided.
b) Vignette equivalence: “the level of the variable represented by any one vignette is perceived by all respondents in the same way and on the same unidimensional scale” (King et al., 2004, p.194).

7. The World Health Survey
Overview of the Survey
Launched by the World Health Organisation (WHO) in 2001
AIM: strengthen national capacity to monitor critical health outputs and outcomes, provide policy-makers with the evidences to adjust their policies (Üstün, et al., 2003)
70 countries, response rates %
Survey modes: face to face interview (90-minute long for 53 countries and 30-minute long for 13 countries) and computer assisted telephone interviews (4 countries)
Samples: randomly selected (+ 18 years), sizes between 600 and 10,000
Modular basis: health insurance, health expenditures, socio-demographics and income, health state valuations, health system responsiveness, and health system goals.

8. Data Dependent Variable: Responsiveness
Dependent Variable: Responsiveness
Sections: Needing Health Care and General Evaluation of the Health System, Seeing Health Care Providers, Outpatients and Care at Home, Inpatient Hospital, Vignettes
Domains: Autonomy, Choice, Clarity of communication, Confidentiality, Dignity, Prompt attention, Quality of basic amenities, Social support
Response categories: “very good”, “good”, “moderate”, “bad” and “very bad”
Independent Variables: (reporting behaviour and responsiveness equation)
Education: categorical variable (7 categories) or continuous variable (number of years in education).
Gender: is a dummy variable, 1 if woman, 0 if man.
Income: dummy variables to indicate the tertiles of the within-country distribution of household permanent income, measured with the HOPIT model (Ferguson et al., 2003).
Age: continuous variable (years)
Country dummies (Mexico is the baseline)

9. Evidence of differential reporting behaviour (1)
Summary frequencies for the reporting of responsiveness and vignettes, World Health Survey, Clarity of Communication
Mexico UK

10. Evidence of differential reporting behaviour (2)
Vignette ratings by socio-demographic characteristics of the respondents, World Health Survey, Mexico, Clarity of Communication
Education Income Quintiles .1 .2 .3 .4 .5 .6 %
primary school completed
secondary school completed
high school completed
post graduate degree completed
mean of verygood
mean of good
mean of moderate
mean of bad
mean of verybad .1 .2 .3 .4 .5 .6 % 1 2 3 4 5
mean of verygood
mean of good
mean of moderate
mean of bad
mean of verybad
Gender Age .1 .2 .3 .4 .5 .6 %
female
male
mean of verygood
mean of good
mean of moderate
mean of bad
mean of verybad .1 .2 .3 .4 .5 .6 %
10-35
36-50
51-65
66+
mean of verygood
mean of good
mean of moderate
mean of bad
mean of verybad

11. Comparison of OECD countries (1)
Responsiveness equation = f (socio-demographic variables + country dummy variables + interaction terms between country dummies and the socio-demographic variables)
Reporting behaviour equation = f (socio-demographic variables + country dummy variables + interaction terms between country dummies and income)
Coefficients and standard errors for the full set of country dummy variables used to model the cut-point thresholds, Prompt attention.
Cut - P
oint
Thresholds
Country 1 2 3 4
AUT
BEL
CZE
DEU
DNK
ESP
FIN
FRA UK
GRC
HUN
IRL
ITA
NLD PR T
SVK
SWE
.272 (.181 )
.408 (.144)
.329 (.132)
.388 (.116)
.533 (.138)
.392 (.052)
.267 (.107)
.229 (.164)
.342 (.110)
.283 (.105)
.604 (.082)
.258 (.141)
.565 (.227)
.356 (.163)
.254 (.142)
.313 (.088)
.352 (.109)
.351 (147)
.017 (.138)
.043 (.122) .1
32 (.112)
.394 (.131)
.335 (.048)
.149 (.097)
.124 (.153)
.062 (.102)
.039 (.098)
.195 (.079)
.106 (.124)
.273 (.218)
.055 (.155)
.152 (.133)
.236 (.081)
.056 (.103)
.408 (.141)
.067 (.138)
.098 (.121)
.178 (.112)
.014 (.134)
.119 (.048)
.098 (.097)
.051 (.155)
.034 (.102)
.078 (.100)
.002 (.080)
.146 (.124)
.171 (.217)
.065 (.154)
.526 (.136)
.079 (.082)
.067 (.105)
.641 (.145)
.297 (.143)
.435 (.124)
.530 (.115)
.822 (.140)
.031 (.053)
.282 (.102)
.554 (.160)
.580 (.106)
.377 (.105)
.539 (.083)
.879 (.127)
.070 (.238)
.322 (.159)
.588 (.169)
.400 (.087)
.557 (.108)

12. Comparison of OECD countries (2)
Ranking of OECD countries, observed and predicted probabilities of reporting “very good” responsiveness, Respect

13. Comparison of OECD countries (3)
Ranking of OECD countries, observed and predicted probabilities of reporting “very good” responsiveness, Confidentiality

14. Comparison of OECD countries (4)
Ranking of OECD countries, across all domains

15. Conclusions
Evidence that reporting behaviour varies systematically both across countries and across socio-demographic groups within a country
Correcting for different reporting behaviour across countries affects the ranking of countries according to their health system responsiveness
Once differential reporting behaviour has be accounted for and country-specific performance is benchmarked against a single country, in general, health system responsiveness is rated more favourably in OECD countries from Northern and Western Europe, than from Eastern and Southern Europe.
Future research:
Focus on other determinants of health system responsiveness to further aid cross-country comparison.
Anchoring vignettes rely on two key assumptions: response consistency and vignette equivalence. Test the validity of these assumptions.