ICEPOP Programme Advantages and disadvantages of the use of best-worst scaling in the field of health Terry Flynn PhD MRC HSRC, Bristol

ICEPOP Programme Outline What is best-worst scaling? How has it been used in HSR to date? Application: dermatology trial Application: quality of life Advantages and disadvantages Areas for research

ICEPOP Programme Traditional DCEs Discrete Choice Experiments increasingly used in HSR Respondents choose a preferred specification of the good or service Aim is to obtain quantitative estimates of utility (benefit) associated with different attribute levels describing the good or service

ICEPOP Programme The issue of interest here Generally dermatology patients would prefer: Being seen by a consultant-led team rather than a GP with part-time interest in dermatology An appointment this week to one in 3 months But suppose the choice is between an appointment this week with a GP specialist and one in 3 months with a consultant. Which do patients value most? Doctor expertise or waiting time?

ICEPOP Programme An example Appointment A Appt this week GP specialist Easy to get to (S)he is thorough You pay £5 Appointment B Appt in 3 months Consultant Difficult to get to Isn’t thorough You do not pay Which appointment would you choose?

ICEPOP Programme Application 1 Estimating preferences for aspects of a dermatology appointment

ICEPOP Programme Dermatology trial example BestAppointment AWorst You will have to wait one month for your appointment Getting to your appointment is difficult and time-consuming  Consultation will be as thorough as you would like  Doctor is an expert who has been treating skin complaints for at least five years

ICEPOP Programme Best-Worst Scaling Devised by Finn & Louviere (JPPM 1992) –introduced to health care by McIntosh & Louviere (HESG 2002) –statistical proof paper Marley & Louviere (J Math Psych 2005) –‘user guide’ by Flynn et al (JHE 2006) Differs from traditional DCEs in the nature of the choice task Individuals choose the best and the worst attribute based on the levels displayed in a given specification

ICEPOP Programme Dermatology trial Patients who had been referred to secondary care for skin complaint Postal questionnaire Randomly assigned to short version (8 DCE scenarios) or long (16) 202 out of 240 q’airres returned (139 complete) Each scenario is a SINGLE consultation described by waiting time, expertise of doctor, ease of attending and thoroughness

ICEPOP Programme Attributes & levels Waiting time –3 months –2 months –1 month –1 week Doctor expertise –Part time specialist (GPSI) –Full time specialist (consultant) Ease of access –Easy –Difficult Individualised care –Thorough –Not thorough

ICEPOP Programme Attribute levels

ICEPOP Programme Attribute impacts

ICEPOP Programme BWS estimated differences

ICEPOP Programme Multinomial (conditional) logit analysis Effect of patient characteristics (clinical or sociodemographic) upon preferences Separate effects of age/sex etc upon attribute importance from effects upon level scales Independent variables are version of effects coding – epidemiological example: mean effect across both sexes is estimated, with effect code giving additional effect for one sex (the other is this multiplied by minus 1)

ICEPOP Programme Fully adjusted MNL results Estimate Std Error z p>|z| [95% confidence interval] Attributes Waiting time | Dr | Convenience | Indivcare | Levels wait3m | wait2m | wait1m | wait0m | drpttime | drfulltime | convhard | conveasy | indivno | indivyes |

ICEPOP Programme Higher education Estimate Std Error z p>|z| [95% confidence interval] Attributes educ_dr | educ_conv | educ_indiv | Levels educ_3m | * educ_2m | educ_1m | * educ_0m | educ_drpt | * educ_drft | educ_convh~d | * educ_conve~y | educ_indivno | * educ_indivye |

ICEPOP Programme Scoring 7+/30 on skin severity Estimate Std Error z p>|z| [95% confidence interval] Attributes score7_dr | * score7_conv | score7_indiv | * Levels score7_3m | score7_2m | score7_1m | score7_0m | score7_drpt | score7_drft | score7_con~d | * score7_con~y | score7_ind~n | * score7_ind~y |

ICEPOP Programme Implications for dermatology Policies to improve ‘process’ aspects of the consultation will benefit higher sociodemographic groups most Policies to improve waiting times will benefit those patients who they themselves feel most affected by their skin condition

ICEPOP Programme Statistical issues MNL is (usually) a first step –Is there heterogeneity? –Likely covariates that characterise it? More complex methods? –Mixed logit what distributional assumption? lots of parameters in BWS: 72 possible pairs here –Latent class analysis Non/semi parametric

ICEPOP Programme Application 2 Estimating tariffs for the ICECAP quality of life instrument for older people

ICEPOP Programme It’s one thing to know what the ‘average’ preference for an impaired health state is in the population……but suppose the poor/ill regard that state as being particularly dreadful – any decision to take (or not take) this into consideration requires us to find out if the poor/ill have different preferences Heterogeneity

ICEPOP Programme Heterogeneity (2) The use of population-level tariffs might mean some interventions are deemed cost-ineffective when for the poor/ill they are highly cost-effective Even if we don’t want to move away from population-level provision society should have the data to debate this

ICEPOP Programme Aim To produce a set of ‘tariffs’ for the 4 5 =1024 possible quality of life scenarios that a British older person might experience An older person could tick the box to indicate which of 4 levels (s)he is experiencing for each of 5 questions –e.g. before the meals-on-wheels service a score of 0.6 on a zero to one scale –after the meals-on-wheels service a score of 0.75 on a zero to one scale

ICEPOP Programme The ICECAP quality of life instrument Four levels –all; –a lot (many); –a little (few); –none Example: role o I am able to do all of the things that make me feel valued  I am able to do many of the things that make me feel valued o I am able to do a few of the things that make me feel valued o I am unable to do any of the things that make me feel valued

ICEPOP Programme The ICECAP quality of life instrument (contd) Similarly for: Attachment(love and friendship) Security(thinking about the future without concern) Enjoyment(enjoyment and pleasure) Control(independence)

ICEPOP Programme A complete quality of life state I can have all of the love and friendship that I want I can only think about the future with a lot of concern I am able to do many of the things that make me feel valued I can have a little of the enjoyment and pleasure that I want I am able to be completely independent

ICEPOP Programme The best-worst scaling study 315 completed interviews (478 approached to take part) 255 had complete best-worst data Average length of interview: 35 minutes Administered in older person’s own home All had participated in Health Survey for England (HSE) Data available from previous round of HSE (6-12 months previous) included sociodemographic and health (n=226)

ICEPOP Programme Statistical design Respondents randomised to: –Orthogonal main effects plan in 16 scenarios or –Its foldover

ICEPOP Programme You can have a lot of the love and friendship that you want You can only think about the future with a lot of concern You are unable to do any of the things that make you feel valued  You can have a little of the enjoyment and pleasure that you want  You are able to be independent in a few things Best Example quality of life scenario Worst

ICEPOP Programme Population-level BWS estimates (n=255)

ICEPOP Programme Heterogeneity in ICECAP

ICEPOP Programme Latent class analysis Performed on the choice data Conditional logit results for each class No adjustment for covariates –Need to know first of all if subgroups who are internally homogeneous exist –Then see if we can characterise these in terms of health/wealth/other factors Covariate-adjusted conditional logit regressions (1-class) suggested there was heterogeneity…

ICEPOP Programme LCA results

ICEPOP Programme Statistical vs policy significance

ICEPOP Programme Who are these people? Can distinguish class three easily: disproportionately: –Male –Without any qualifications –Married (but only at 10% level) But so what? Class 1 vs class 2….?

ICEPOP Programme Class 1 versus class 2 Difficult to distinguish them –Having had a total joint replacement was predictor for class 2 (more bothered about attachments than control) –Being unable to climb 12 stairs was predictor for class 1 (more bothered about control than attachments) Work with UTS researchers to investigate alternative characterisations of clustering

ICEPOP Programme Advantages of BWS All attribute levels on the same scale More data –Estimate attribute impacts –Understand heterogeneity more easily; distributional assumptions not needed when have individual respondent utilities Use as a method to get a random utility theory consistent set of rankings Easier choice task? Simpler statistical design

ICEPOP Programme Disadvantages of BWS The problem of the numeraire (money) Conditional not unconditional demand –Nest within a DCE and adjust for different random utility components Getting individual respondent models not practical in many contexts

ICEPOP Programme Future research in Best-Worst methods Individual patient preferences –clustering using other taxonomic methods –investigate decision rules (lexicographic preferences) Estimating attribute importance (rather than simply impact) –Alternative conceptualisation of utility Anchoring (the unconditional demand issue)

Investigating Choice Experiments for the Preferences of Older People (ICEPOP) Professors Joanna Coast (Birmingham) Jordan Louviere (UTS) Tim Peters (Bristol) & Dr Terry Flynn We would like to thank Dr Tony Marley for comments and assistance

ICEPOP Programme

Bristol sample 198 of the 1024 QoL states represented in Bristol Percentiles Smallest 1% % % Obs % Sum of Wgt % Mean Largest Std. Dev % % Variance % 1 1 Skewness % 1 1 Kurtosis

ICEPOP Programme ICECAP sample (313) 137 of the 1024 QoL states represented in BWS study Percentiles Smallest 1% % % Obs % Sum of Wgt % Mean Largest Std. Dev % % Variance % Skewness % 1 1 Kurtosis

ICEPOP Programme Random Utility Theory Let latent utility for item i be: U i =  i +  i U i = latent utility,  i = explainable portion &  i = unexplainable portion. Probability that i is chosen: P(i | C n ) = P[(  i +  i ) > (  j +  j )]  j  C n, if  ’ s ~ EV1 (0,  2 )  McFadden’s MNL model: P(i | C n ) = exp(  i ) /  j exp(  j )