1. International comparison of health system performance and self-reported data
Nigel Rice, Silvana Robone, Peter Smith
Centre for Health Economics, University of York
Project funded by the UK ESRC, Public Service Programme

2. Introduction (1)
Patients’ views and opinions are an essential means for assessing the provision of health services, to stimulate quality improvements and to measure health systems performance.
Traditionally, patients’ views were sought on the quality of care provided and satisfaction with health services. Recently the concept of responsiveness has been promoted as a more desirable measure to judge health systems.
Responsiveness can be defined as the way in which individuals are treated and the environment in which they are treated encompassing the notion of patient experience with the health care system (Valentine et al., 2003)

3. Introduction (2)
ISSUE: data on Responsiveness are self-reported derived from surveys
Ex: “For your [child’s] last visit, how would you rate the experience of being involved in making decisions about your health care or treatment?”
Response categories: “Very good”, “Good”, “Moderate”, “Bad”, and “Very bad”.
The meaning of the available response categories may be interpreted differently across population sub-groups
Responses will be influenced by individuals' preferences and expectations, which vary systematically across countries, or across socio-demographic groups within a country (REPORTING HETEROGENEITY)
Country A
Country B

4. Methods
Use of anchoring vignettes to address the issue of reporting heterogeneity.
Vignettes = descriptions of fixed levels of a latent construct
EX: “When the clinic is not busy, [Mamadou] can choose which doctor he sees. But most often it is busy and then he gets sent to whoever is free”. How would you rate [Mamadou’s] freedom to choose his health care provider? 1. Very good 2. Good 3. Moderate 4. Bad 5. Very bad
Any systematic variation across individuals in the rating of the vignettes can be attributed to reporting heterogeneity (or measurement error).
Use of the hierarchical ordered probit model (HOPIT) (Tandon et al. (2003)), that exploits the anchoring vignettes as source of external information that identify the determinants of reporting heterogeniety.
Two parts:
1) reporting behaviour (bias) equation
2) responsiveness equation
Assumptions: Response Consistency and Vignettes Equivalence

5. Data The World Health Survey
The World Health Survey
Launched by the World Health Organisation (WHO) in 2001
70 countries, samples randomly selected (+ 18 years), sizes ,000
Dependent Variable: Responsiveness
Domains: Autonomy, Choice, Clarity of communication, Confidentiality, Dignity, Prompt attention, Quality of basic amenities, Social support
Response categories: “very good”, “good”, “moderate”, “bad” and “very bad”
Independent Variables: (reporting behaviour and responsiveness equation)
Education: categorical variable (7 categories) or continuous variable (number of years in education).
Gender: is a dummy variable, 1 if woman, 0 if man.
Income: dummy variables to indicate the tertiles of the within-country distribution of household permanent income, measured with the HOPIT model (Ferguson et al., 2003).
Age: continuous variable (years)

6. Evidence of differential reporting behaviour (1)
Summary frequencies for the reporting of responsiveness and vignettes, World Health Survey, Clarity of Communication
Mexico UK

7. Evidence of differential reporting behaviour (2)
Vignette ratings by socio-demographic characteristics of the respondents, World Health Survey, Mexico, Clarity of Communication
Education Income Quintiles .1 .2 .3 .4 .5 .6 %
primary school completed
secondary school completed
high school completed
post graduate degree completed
mean of verygood
mean of good
mean of moderate
mean of bad
mean of verybad .1 .2 .3 .4 .5 .6 % 1 2 3 4 5
mean of verygood
mean of good
mean of moderate
mean of bad
mean of verybad
Gender Age .1 .2 .3 .4 .5 .6 %
female
male
mean of verygood
mean of good
mean of moderate
mean of bad
mean of verybad .1 .2 .3 .4 .5 .6 %
10-35
36-50
51-65
66+
mean of verygood
mean of good
mean of moderate
mean of bad
mean of verybad

8. Comparison of OECD countries (1)
Responsiveness equation = f (socio-demographic variables + country dummy variables + interaction terms between country dummies and the socio-demographic variables)
Reporting behaviour equation = f (socio-demographic variables + country dummy variables + interaction terms between country dummies and income)
Coefficients and standard errors for the full set of country dummy variables used to model the cut-point thresholds, Prompt attention.
Cut - P
oint
Thresholds
Country 1 2 3 4
AUT
BEL
CZE
DEU
DNK
ESP
FIN
FRA UK
GRC
HUN
IRL
ITA
NLD PR T
SVK
SWE
.272 (.181 )
.408 (.144)
.329 (.132)
.388 (.116)
.533 (.138)
.392 (.052)
.267 (.107)
.229 (.164)
.342 (.110)
.283 (.105)
.604 (.082)
.258 (.141)
.565 (.227)
.356 (.163)
.254 (.142)
.313 (.088)
.352 (.109)
.351 (147)
.017 (.138)
.043 (.122) .1
32 (.112)
.394 (.131)
.335 (.048)
.149 (.097)
.124 (.153)
.062 (.102)
.039 (.098)
.195 (.079)
.106 (.124)
.273 (.218)
.055 (.155)
.152 (.133)
.236 (.081)
.056 (.103)
.408 (.141)
.067 (.138)
.098 (.121)
.178 (.112)
.014 (.134)
.119 (.048)
.098 (.097)
.051 (.155)
.034 (.102)
.078 (.100)
.002 (.080)
.146 (.124)
.171 (.217)
.065 (.154)
.526 (.136)
.079 (.082)
.067 (.105)
.641 (.145)
.297 (.143)
.435 (.124)
.530 (.115)
.822 (.140)
.031 (.053)
.282 (.102)
.554 (.160)
.580 (.106)
.377 (.105)
.539 (.083)
.879 (.127)
.070 (.238)
.322 (.159)
.588 (.169)
.400 (.087)
.557 (.108)

9. Comparison of OECD countries (2)
Ranking of OECD countries, observed and predicted probabilities of reporting “very good” responsiveness, Respect

10. Conclusions
Evidence that reporting behaviour varies systematically both across countries and across socio-demographic groups within a country
Correcting for different reporting behaviour across countries affects the ranking of countries according to their health system responsiveness
Once differential reporting behaviour has be accounted for and country-specific performance is benchmarked against a single country, in general, health system responsiveness is rated more favourably in OECD countries from Northern and Western Europe, than from Eastern and Southern Europe.
Future research:
Focus on other determinants of health system responsiveness to further aid cross-country comparison.
Anchoring vignettes rely on two key assumptions: response consistency and vignette equivalence. Test the validity of these assumptions.