1 The measurement and comparison of health system responsiveness Nigel Rice, Silvana Robone, Peter Smith (Centre for Health Economics, University of York)

Slides:



Advertisements
Similar presentations
Introduction Describe what panel data is and the reasons for using it in this format Assess the importance of fixed and random effects Examine the Hausman.
Advertisements

Survey design. What is a survey?? Asking questions – questionnaires Finding out things about people Simple things – lots of people What things? What people?
Work Disability, Work, and Justification Bias in Europe and the US Arie Kapteyn (RAND) James P. Smith (RAND) Arthur van Soest (Netspar, Tilburg University)
[Part 1] 1/15 Discrete Choice Modeling Econometric Methodology Discrete Choice Modeling William Greene Stern School of Business New York University 0Introduction.
Understanding children’s well-being: A national survey of young people’s well-being 27 January 2010.
Part II Knowing How to Assess Chapter 5 Minimizing Error p115 Review of Appl 644 – Measurement Theory – Reliability – Validity Assessment is broader term.
Estimation and Reporting of Heterogeneity of Treatment Effects in Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare.
Chapter 1 The Where, Why, and How of Data Collection
Chapter 1 The Where, Why, and How of Data Collection
Measuring Disability in a Survey or Census Context: Parallel Work Advancing the Field Barbara M. Altman, Ph.D. Disability Statistics Consultant.
Clustered or Multilevel Data
Chapter 11 Multiple Regression.
FOUNDATIONS OF NURSING RESEARCH Sixth Edition CHAPTER Copyright ©2012 by Pearson Education, Inc. All rights reserved. Foundations of Nursing Research,
Variables and Measurement (2.1) Variable - Characteristic that takes on varying levels among subjects –Qualitative - Levels are unordered categories (referred.
FINAL REPORT: OUTLINE & OVERVIEW OF SURVEY ERRORS
Analysis of Clustered and Longitudinal Data
Antiretroviral Treatment Costs in Mexico WHO/UNAIDS Workshop on Strategic Information for Anti-Retroviral Therapy Programmes 30 June to 2 July, 2003 Assessment.
PowerPoint Presentation Package to Accompany:
Chapter 3 Goals After completing this chapter, you should be able to: Describe key data collection methods Know key definitions:  Population vs. Sample.
Multiple Choice Questions for discussion
Sampling. Concerns 1)Representativeness of the Sample: Does the sample accurately portray the population from which it is drawn 2)Time and Change: Was.
Discrete Choice Modeling William Greene Stern School of Business New York University.
Work Disability, Work, and Justification Bias in Europe and the US Arie Kapteyn (RAND) James P. Smith (RAND) Arthur van Soest (Netspar, Tilburg University,
Analyzing Reliability and Validity in Outcomes Assessment (Part 1) Robert W. Lingard and Deborah K. van Alphen California State University, Northridge.
Self-reported cognitive and emotional effects and lifestyle changes shortly after preventive cardiovascular consultations in general practice Dea Kehler.
Experimental Design making causal inferences Richard Lambert, Ph.D.
Instrumentation (cont.) February 28 Note: Measurement Plan Due Next Week.
[Part 4] 1/43 Discrete Choice Modeling Bivariate & Multivariate Probit Discrete Choice Modeling William Greene Stern School of Business New York University.
Introduction Biostatistics Analysis: Lecture 1 Definitions and Data Collection.
Sampling “Sampling is the process of choosing sample which is a group of people, items and objects. That are taken from population for measurement and.
Copyright  2003 by Dr. Gallimore, Wright State University Department of Biomedical, Industrial Engineering & Human Factors Engineering Human Factors Research.
How Much Do Patients’ Preferences Contribute To Resource Use? Anthony D L, Herndon M B, et al. Health Affairs, 28, no. 3 (2009):
Equity, then Social Insurance … Allen C. Goodman © 2013.
Discrete Choice Modeling William Greene Stern School of Business New York University.
Discrete Choice Modeling William Greene Stern School of Business New York University.
7th International Forum on Tourism Statistics Stockholm, Sweden, 9-11 June 2004 NEW RESEARCH LINES ON TOURISM INFORMATION SYSTEMS: POTENTIAL DEMAND ANALYSES.
A discussion of Comparing register and survey wealth data ( F. Johansson and A. Klevmarken) & The Impact of Methodological Decisions around Imputation.
1 Components of the Deterministic Portion of the Utility “Deterministic -- Observable -- Systematic” portion of the utility!  Mathematical function of.
Discrete Choice Modeling William Greene Stern School of Business New York University.
Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance.
BC Jung A Brief Introduction to Epidemiology - XIII (Critiquing the Research: Statistical Considerations) Betty C. Jung, RN, MPH, CHES.
IMPORTANCE OF STATISTICS MR.CHITHRAVEL.V ASST.PROFESSOR ACN.
Chapter 7 Measuring of data Reliability of measuring instruments The reliability* of instrument is the consistency with which it measures the target attribute.
Qualitative and Limited Dependent Variable Models ECON 6002 Econometrics Memorial University of Newfoundland Adapted from Vera Tabakova’s notes.
Basic Business Statistics, 8e © 2002 Prentice-Hall, Inc. Chap 1-1 Inferential Statistics for Forecasting Dr. Ghada Abo-zaid Inferential Statistics for.
Randomized Assignment Difference-in-Differences
1/53: Topic 3.1 – Models for Ordered Choices Microeconometric Modeling William Greene Stern School of Business New York University New York NY USA William.
The measurement and comparison of health system responsiveness Nigel Rice, Silvana Robone, Peter C. Smith Centre for Health Economics, University of York.
Measurement and Health Information Systems Working Paper No.8 11 November 2005 STATISTICAL COMMISSION andSTATISTICAL OFFICE OF THE UN ECONOMIC COMMISSION.
6. Ordered Choice Models. Ordered Choices Ordered Discrete Outcomes E.g.: Taste test, credit rating, course grade, preference scale Underlying random.
1 Collecting and Interpreting Quantitative Data Deborah K. van Alphen and Robert W. Lingard California State University, Northridge.
[Part 5] 1/43 Discrete Choice Modeling Ordered Choice Models Discrete Choice Modeling William Greene Stern School of Business New York University 0Introduction.
Monday, June 23, 2008Slide 1 KSU Females prospective on Maternity Services in PHC Maternity Services in Primary Health Care Centers : The Females Perception.
Slide 7.1 Saunders, Lewis and Thornhill, Research Methods for Business Students, 5 th Edition, © Mark Saunders, Philip Lewis and Adrian Thornhill 2009.
Modernizing Health Care Inez Bartels.  Strong focus on the provision of health care  Institutions governing health care consumption control patients.
Research proposal (Lecture 3) Dr.Rehab F Gwada. Objectives of the Lecture The student at the end of this lecture should Know Identify Target Population.
5. Evaluation of measuring tools: reliability Psychometrics. 2011/12. Group A (English)
Learning Objectives : After completing this lesson, you should be able to: Describe key data collection methods Know key definitions: Population vs. Sample.
WHO The World Health Survey General Introduction
William Greene Stern School of Business New York University
M.Sc. in Economics Econometrics Module I
12 Inferential Analysis.
Chapter Eight: Quantitative Methods
  International comparison of health system performance and self-reported data Nigel Rice, Silvana Robone, Peter Smith Centre for Health Economics, University.
Gerald Dyer, Jr., MPH October 20, 2016
12 Inferential Analysis.
The measurement and comparison of health system responsiveness
  International comparison of health system responsiveness: the use of the hierarchical ordered probit model to adjust self-reported data Nigel Rice, Silvana.
Presentation transcript:

1 The measurement and comparison of health system responsiveness Nigel Rice, Silvana Robone, Peter Smith (Centre for Health Economics, University of York) HEDG Seminar York, 4 June 2008

2 Outline Introduction, Study Aim Data Econometric model Results: Descriptive statistics Estimates within countries Estimates across-countries, HDI groups Estimates across countries, overall Responsiveness ranking Conclusions

3 Introduction (1) Patients’ views and opinions are an essential means for assessing the provision of health services, to stimulate quality improvements and to measure health systems performance. Traditionally, patients’ views were sought on the quality of care provided and satisfaction with health services. Recently the concept of responsiveness has been promoted as a more desirable measure to judge health systems. Responsiveness can be defined as the way in which individuals are treated and the environment in which they are treated encompassing the notion of patient experience with the health care system (Valentine et al., 2003)

4 Introduction (2) Outcome measurements enable institutions to compare and contrast their performance to that of others, including at a macro level, the performance obtained in other countries. Studies aimed at comparative inference have rarely taken into consideration possible variations in cultural expectations that might impact on reporting behaviour (Blendon et al., 2003). ISSUE: data on Responsiveness are self-reported. Ex: “For your [child’s] last visit, how would you rate the experience of being involved in making decisions about your health care or treatment?” Response categories: “Very good”, “Good”, “Moderate”, “Bad”, and “Very bad”. Individuals, when faced with survey questions about the functioning of health systems, may systematically interpret the meaning of the available response categories differentially across population sub- groups (Sadana et al., 2002).

5 Introduction (3) Responses will be influenced by individuals' preferences and expectations, which vary systematically across countries, or across socio-demographic groups within a country (REPORTING HETEROGENEITY). Fixed level of underlying responsiveness is unlikely to be rated equally across sub-groups of interest (see Tandon et al., 2003). EX: Reporting heterogeneity has been debated usually with regard to measures of health status (for example, Jürges, 2007, Kapteyn et al., 2007; Bago d’Uva et al., 2007; Lindeboom and van Doorslaer, 2004; Iburg et al., 2002). Few studies with regard to responsiveness (Valentine et al. (2003), Puentes Rosas et al. (2006)).

6 Introduction (4) Use of anchoring vignettes to address the issue of reporting heterogeneity. Vignettes = descriptions of fixed levels of a latent construct, such as respons. EX: “When the clinic is not busy, [Mamadou] can choose which doctor he sees. But most often it is busy and then he gets sent to whoever is free”. How would you rate [Mamadou’s] freedom to choose his health care provider? 1. Very good 2. Good 3. Moderate 4. Bad 5. Very bad - Any systematic variation across individuals in the rating of the vignettes can be attributed to reporting heterogeneity (or measurement error). Responses to the vignettes allow the researcher to model the response scales as a function of the characteristics of respondents.

7 Study Aim OUR AIM: explore the utility of using information from vignettes to adjust self-reports of health system responsiveness. Use of nine countries from the World Health Survey 1)evaluate the presence of reporting bias across socio-economic groups within countries, and how it is related to the characteristics of the individuals. Human Development IndexHDI 2) to aid cross country comparison, we aggregate these country into three groups according to their Human Development Index (HDI) level (United Nations Development Programme, 2006) and we evaluate the presence of differences in reporting among countries in the same group. 3) we assess the presence of different reporting behaviour among the three HDI groups of countries. 4) we try to evaluate if the issue or reporting bias affect the ranking of health systems responsiveness.

8 DATA: The World Health Survey (1) Overview of the Survey Launched by the World Health Organisation (WHO) in 2001 AIM: strengthen national capacity to monitor critical health outputs and outcomes, provide policy-makers with the evidences to adjust their policies (Üstün, et al., 2003) 70 countries, response rates % Survey modes: face to face interview (90-minute long for 53 countries and 30-minute long for 13 countries) and computer assisted telephone interviews (4 countries) Samples: randomly selected (+ 18 years), sizes between 600 and 10,000 Modular basis: health insurance, health expenditures, socio- demographics and income, health state valuations, health system responsiveness, and health system goals.

9 DATA: The World Health Survey (2) The Responsiveness Module: Developed from an extensive consultation process. Origin in the Multi-Country Survey Study on Health and Responsiveness ( ) Sections: Needing Health Care and General Evaluation of the Health System, Seeing Health Care Providers, Outpatients and Care at Home, Inpatient Hospital, Vignettes Responsiveness Domains: Autonomy Choice Clarity of communication Confidentiality Dignity Prompt attention, Quality of basic amenities Social support

10 DATA: Country selection 9 countries within the WHS: Mexico, Spain, Malaysia, India, Philippines, SriLanka, Burkina, Malawi and Ethiopia Selected on the basis of 4 criteria: Long version (90-minute in-household) questionnaire Satisfy well a set of psychometric properties (feasibility, reliability and validity) for the responsiveness module represent geographical areas characterized by different levels of development. Use of HDI to stratify the countries into three groups, high, medium and low HDI group High sample size in comparison to other countries belonging to the same HDI group.

11 DATA: Variables (1) Dependent Variables: Only 4 out of the 8 domains: Respect, Confidentiality, Quality of Facilities and Clear Communication Domains considered as most important by the respondents in the nine countries selected. For each domain, respondents were asked up to 2 questions about their experiences of contact with health systems. The response categories: “very good”, “good”, “moderate”, “bad” and “very bad”.

12 DATA: Variables (2) Independent Variables: Individual level - Education: categorical variable (7 categories) or continuous variable (number of years in education). - Gender: is a dummy variable, 1 if woman, 0 if man. - Income: categorical variable (quintiles of the distribution of household permanent income, 1 if in the lowest income quintile, 5 if in the highest one). Permanent income measured with the HOPIT model (Ferguson et al., 2003). - Age: continuous variable (years) Country level (data provided by the UNDP for the year 2001) -Health expenditures per capita (both private and public), in current 1,000 US$ -GDP per capita, in constant ,000 US$ (to consider PPP). Country and HDI group dummies

13 Econometric model (1) The Hierarchical Ordered Probit Model (HOPIT) Terza (1985), Tandon et al. (2003) It allows to account for heterogeneous reporting behaviour Use of the anchoring vignettes to provide a source of external information that enables the identification of the cut-points as functions of covariates. Assumptions: a) Response consistency It is assumed that individuals classify the vignettes in a way that is consistent with the rating of their own experiences of the service provided. b) Irrelevance of own provider responsiveness or vignette equivalence. It is assumed that “the level of the variable represented by any one vignette is perceived by all respondents in the same way and on the same unidimensional scale” (King et al., 2004, p.194).

14 Econometric model (2) Reporting behaviour equation (1) = underlying health system responsiveness for vignette k, rated by individual i = vector of vignettes = vector of parameters = idiosyncratic error term we observe the vignette rating, on a five point scale ranging from ‘Very bad’ to ‘Very good’. We assume the observed category of is related to through: (2) we allow the cut-points to be functions of covariates, such that: (3)

15 Econometric model (3) Responsiveness equation = health system responsiveness faced by individual = represents a set of regressors correlated with responsiveness ( might be a sub-set of ) for = observed categorical response are defined by (3) with fixed and it is assumed that and are independent for all and the probabilities associated with each of the 5 categories are given by: where is the cumulative standard normal distribution.

16 Results: Descr. Stat. (1) Summary frequencies for the reporting of vignettes: Mexico. Clarity of communication

17 Results: Descr. Stat. (2) Ratings of vig.2 for “Clarity of com”, Mexico, by: 1) Education 2) Income 3) Gender 4) Age

18 Results: within countries (1) Tests of homogenous reporting and parallel cut-point shift (p-value for Wald Test under H 0 ) AllInc.WomenAgeEduc.AllInc.WomenAgeEduc. Dignityrespect privacy Communicationclear explanations time for questions Confidentialitytalk privately confidential info Facilitiescleanliness space AllInc.WomenAgeEduc.AllInc.WomenAgeEduc. Dignityrespect privacy Communicationclear explanations time for questions Confidentialitytalk privately confidential info Facilitiescleanliness space Malaysia Mexico HeterogeneityParallel cut-point shift HeterogeneityParallel cut-point shift

19 Results: within countries(2) permanent income Estimated coefficients and standard errors of permanent income of the cut-points

20 Results: within countries(3) education Estimated coefficients and standard errors of education of the cut-points

21 Results: within countries(4) Estimated coefficients and standard errors in the OPROBIT and HOPIT model, for Mexico: Income Education OPROBITHOPIT Dignity respectcoeff st. err privacycoeff st. err Communication clear explanationcoeff st. err time for questionscoeff st. err Confidentiality talk privatelycoeff st. err confidential infocoeff st. err Facilities cleanlinesscoeff st. err spacecoeff st. err

22 Results: within countries(5) Marginal effects of education in the OPROBIT and HOPIT model (the increase in the probability of being in very good responsiveness due to an increase in one year of education) ME of edu on very good responsiveness -0.05% 0.00% 0.05% 0.10% 0.15% 0.20% 0.25% 0.30% Me_edu OPROBIT Me_edu HOPIT

23 Results: within countries(6) Ex-ante frequencies, ex-post frequencies, computed through the OPROBIT and HOPIT model, of reporting each of the five response categories Item ex ante prob ex post prob PROBIT ex post prob HOPIT ex ante prob ex post prob PROBIT ex post prob HOPITex ante prob ex post prob PROBIT ex post prob HOPIT RESPECTvery bad0.3% 0.0%0.7% 0.0%0.7% 0.0% bad2.4%2.5%2.6%3.1% 2.9%0.9% 1.7% moderate10.7%10.8%12.1%8.7% 12.4%6.2%6.3%7.7% good68.5%67.7%65.5%71.2% 67.0%61.5%61.4%59.1% very good18.2%18.8%19.8%16.3% 17.6%30.8%30.7%31.6% PRIVACYvery bad0.4% 0.0%0.4% 0.0%0.2% 0.0% bad2.1%2.2%2.4%2.3% 1.9% 1.4% moderate8.3%8.4%10.0%7.4% 10.5%5.2%5.4%7.4% good69.3%69.1%66.8%73.5%73.4%70.2%66.1%65.8%63.6% very good19.9%20.0%20.8%16.5%16.4%17.5%26.6%26.7%27.6% CLEARCOMvery bad0.2% 0.0%0.7%0.8%0.0%0.7%0.8%0.1% bad2.4%2.5%1.5%3.2% 2.5%2.6%2.7%3.4% moderate8.3% 11.6%7.9% 13.3%11.0%11.2%13.0% good70.0% 67.0%71.8% 66.2%60.3%60.1%57.2% very good19.1%19.0%19.9%16.3% 18.0%25.3%25.2%26.3% TIMEQUESTvery bad0.2% 0.0%0.7% 0.0%0.9% 0.2% bad3.5% 2.7%3.7% 2.8%4.5%4.7%5.6% moderate12.1%11.6%14.6%8.5% 14.1%13.8%13.7%15.1% good68.2%68.5%65.6%71.0% 65.3%59.9%59.7%57.3% very good16.0%16.1%17.1%16.1%16.0%17.8%21.0% 21.8% TALKPRIVvery bad0.3% 0.0%0.6% 0.0%1.0%1.1%0.0% bad2.6%2.4% 4.2% 3.9%3.7%3.8%5.2% moderate12.0%11.6%13.9%10.6% 15.8%10.5%10.7%13.6% good69.2%69.3%66.0%70.6% 64.1%62.6%62.2%57.0% very good16.0%16.4%17.7%14.0% 16.2%22.3%22.2%24.2% CONFINFOvery bad0.1% 0.0%1.2% 0.0%0.8% 0.0% bad1.0%1.1%0.7%2.7% 4.0%1.7%1.8%2.7% moderate8.3%8.7%10.4%10.0% 15.1%7.7% 11.2% good70.0%70.2%68.2%71.9% 63.8%67.8%67.7%61.9% very good20.6%20.0%20.7%14.3%14.2%17.1%22.0% 24.3% CLEANvery bad0.2% 0.0%0.7% 0.0%0.5% 0.1% bad1.3%1.4%1.3%3.3%3.2%3.1%1.3% 2.9% moderate7.7%7.8%8.9%10.0%9.9%13.9%10.6%10.5%9.0% good67.2%67.1%65.9%70.1% 65.4%60.2% 60.7% very good23.6%23.5%23.9%15.9%16.1%17.6%27.4% SPACEvery bad0.1% 0.0%0.8% 0.0%1.9%2.0%0.7% bad1.6%1.5%1.4%4.7%4.8%4.4%6.6%6.7%8.7% moderate14.2%14.5%15.0%11.8%11.7%16.8%16.3% 17.3% good67.6%67.4%67.0%68.1%68.0%62.1%54.3% 51.8% very good16.5% 16.6%14.6%14.7%16.6%20.9%20.8%21.6% SPAINMALAYSIAMEXICO

24 Results: within countries(7) RESET test

25 Results: across HDI groups Tests of homogenous reporting and parallel cut-point shift

26 Results: overall countries Tests of homogenous reporting and parallel cut-point shift

27 Results: resp ranking (1) Kendall tau rank correlation coefficients Kendall tau rank correlation coefficients and related p-values for the ranking of responsiveness in the high HDI countries, computed through the OPROBIT and HOPIT (individual country models) latent value of responsiv. prob of “very good”/“good” responsiv.

28 Results: resp ranking (2) Country ranking or responsiveness predicted through the OPROBIT and the HOPIT (individual country models) for Clarity of communication and Time for questions latent value of responsiv. prob of “very good”/“good” responsiv

29 Results: resp ranking (3) Comparison of predicted responsiveness for Clarity of communication obtained through the OPROBIT and HOPIT (high HDI group model) in Spain, Mexico and Malaysia Predicted latent responsiveness Predicted responsiveness frequencies Spain 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% OPROBIT HOPIT

30 Conclusions Heterogeneity in reporting behaviour exists, and it appears to be a function of individual socioeconomic characteristics (such as income and education), and characteristics at country and macro-regional levels (that we capture through country and HDI group dummies) Adjusting for reporting bias impacts on: 1) The estimated coefficients of the responsiveness mean function when results from the HOPIT model are compared to those from an ordered probit regression. 2) The marginal effects of individual socioeconomic variables (such as education) on responsiveness 3)The ex-post frequencies of reporting each of the five response categories Reporting heterogeneity affects the ranking of high HDI countries according to their responsiveness level. The magnitude and the level of significance in the difference of ranking with or without considering for reporting heterogeneity varies according to the way we measure responsiveness.