1. Measurement and Health Information Systems Working Paper No.8 11 November 2005 STATISTICAL COMMISSION andSTATISTICAL OFFICE OF THE UN ECONOMIC COMMISSION FOREUROPEAN COMMUNITIES EUROPE (EUROSTAT) CONFERENCE OF EUROPEAN WORLD HEALTH STATISTICIANS ORGANIZATION (WHO) Joint UNECE/WHO/Eurostat Meeting on the Measurement of Health Status (Budapest, Hungary, 14-16 November 2005) Session 4 – Invited paper Issues in international comparisons of health status measurement Somnath Chatterji WHO

2. Measurement and Health Information Systems Instrument Development review of existing standard instrumentsreview of existing standard instruments international consultationsinternational consultations review by key informantsreview by key informants pilot testingpilot testing psychometric properties – reliability and validitypsychometric properties – reliability and validity culturally sensitive ways of ensuring comparability across populationsculturally sensitive ways of ensuring comparability across populations –within and across countries

3. Measurement and Health Information Systems WHO Experience Extensive Cognitive testing in cognitive labs with cognitive debriefing protocolsExtensive Cognitive testing in cognitive labs with cognitive debriefing protocols Formal testing of recall periodsFormal testing of recall periods Comparators used in self-reportsComparators used in self-reports Response scalesResponse scales

4. Measurement and Health Information Systems WHO Experience Translation protocolsTranslation protocols –Creation of a translation workbench –Translation by bilingual expert group –Targeted back-translation –Linguistics evaluation with formal feedback on problem terms / concepts –External review by bilingual experts

5. Measurement and Health Information Systems WHO Experience Formal psychometric testingFormal psychometric testing –Test-rest reliability to establish replicability –Analysis of psychometric properties across populations –Verification of item level missing data by population –Check of IRT properties of items –Determination of the most parsimonious set for measurement

6. Measurement and Health Information Systems Cross-Population ComparabilityCross-Population Comparability –Problem of cross-population comparability: self-report categorical response data are not necessarily cross- population comparable. –Problem can be conceptualised in terms of response category cut-points shifts. For the same level of ability on an underlying domain: respondents from different socio- demographic backgrounds respond in different response categories. –This would imply, for example, that someone saying “no problems” with respect to the domain of mobility in country A may not mean the same thing as someone saying “no problems” in country B in terms of the underlying latent variable of mobility: norms or expectations may be lower in country A. WHO Experience

7. Measurement and Health Information Systems Enhancing Cross-Population ComparabilityEnhancing Cross-Population Comparability –Two strategies: use the hierarchical ordered probit (HOPIT) model with (a) vignettes, and (b) measured tests. –A vignette is a hypothetical description of a level of ability on a given domain which respondents are asked to evaluate with respect to same question -- and on the same categorical response scale -- as the main self-report for each of the domains in health and in responsiveness. –Vignettes allow us to fix the level of ability such that any variations are attributable to variations in response category cut-point shifts. –Using the HOPIT model, these variations in estimated cut-points across socio-demographic groups can then be used to calibrate self-report responses to make them cross-population comparable. WHO Experience

8. Measurement and Health Information Systems Problems with self-reported morbidity / diagnosisProblems with self-reported morbidity / diagnosis –Variations across countries lack face validity –Probably more reflective of service availability and knowledge WHO Experience

9. Measurement and Health Information Systems Comparisons across surveys

10. Measurement and Health Information Systems Survey comparisons – post adjustment

11. Measurement and Health Information Systems Prevalence of asthma Self-reported diagnosis

12. Measurement and Health Information Systems Prevalence of asthma: Probabilistic diagnosis scale