1 CADTH Value Methods Panel Using Best Worst Scaling to elicit Values Carlo Marra

Problem 2 ?

Traditional DCEs Discrete Choice Experiments increasingly used in health services research Respondents choose a preferred specification of the good or service Aim is to obtain quantitative estimates of utility (benefit) associated with different attribute levels describing the good or service

Example of a DCE 4

Best-Worst Scaling Devised by Finn & Louviere (JPPM 1992) – introduced to health care by McIntosh & Louviere (HESG 2002) – statistical proof paper Marley & Louviere (J Math Psych 2005) –‘ user guide ’ by Flynn et al (JHE 2006) Differs from traditional DCEs in the nature of the choice task Individuals choose the best and the worst attribute based on the levels displayed in a given specification

Statistical issues MNL is (usually) a first step – Is there heterogeneity? – Likely covariates that characterise it? More complex methods? – Mixed logit – Latent class analysis Non/semi parametric

Problem 8 ?

Analyzing results To get around the dreaded BLACK BOX Best-minus-worst-scores Easy to understand Found to be linearly related to the ML estimates of the conditional logit model in virtually every empirical study to date Scores can help guide analysis of choice data – eg. LCAs which may give spurious associations 9

Stratifying and Targeting Pediatric Medulloblastoma 10

Purpose To determine preferences of the general population, parents and health professionals regarding trade-offs between treatment intensity and survival including test characteristics, functional outcomes, psychological outcomes and economic burden. 11

12 Parents: N=76 participants Health professionals: N=193 participants General population: N= 3006 participants

13 100% accuracy of test 95% accuracy of test 90% accuracy of test 85% accuracy of test 1. Accuracy of test: The possible levels of test accuracy in this survey are:

14 2. QoL/ Functional ability (Side effects of the radiotherapy): The possible health states in this survey are: Child will have normal healthy life. Child will experience mild disability. Child will experience partial disability. Child will experience severe disability.

3. Ten year survival rates: The possible levels of survival in this survey are: 15 Good prognosis Intermediat e prognosis Poor prognosis Baseline Survival Rate 90%70%40% Levels 100%85%55% 95%70%40% 90%55%25% 80%40%10%

Best and Worst Survey Design Clinicians’ Survey 16

Best-Worst estimated parameters (paired model) for general public good prognosisintermediate prognosispoor prognosis AttributesEstimateProbAttributesEstimateProbAttributesEstimateProb Accuracy of the test 100% 2.15< % 3.81< % 4.32< % 1.08< % 3.12< % 3.57< % 0.52< % 2.45< % 2.89< % -0.26< % 1.89< % 2.37<.0001 Quality of life Normal life 2.97<.0001 Normal life 4.35<.0001 Normal life 4.92<.0001 Mild disability -0.98<.0001 Mild disability 1.07<.0001 Mild disability 1.78<.0001 Partial disability -1.58<.0001 Partial disability 0.59<.0001 Partial disability 1.29<.0001 Severe disability -3.21<.0001 Severe disability -1.28<.0001 Severe disability -0.53<.0001 Survival rate 100% 3.28< % 3.01< % 2.09< % 2.29< % 2.10< % 1.40< % 1.37< % 0.56< % 0.55< % % % 0.00

Good prognosis Number of respondent 901 Attribute Times Shown Times Selected Best Best Count Proportion Times Selected Worst Worst Count Proportion Best - Worst score Accuracy of the test 100% %1283.5% % %2146.0%401 90% %3309.2%-47 85% % %-539 Quality of life Normal life %1554.3%1939 Mild disability % %-1689 Partial disability % %-2681 Severe disability % %-3074 Survival rate 100% %1173.3% % %1454.0% % % %554 80% % %-326 Baseline survival rate is 90%.

Best-Worst estimated parameters (paired model) for parents and clinicians – intermediate prognosis Attributes ParentsClinicians Estimate Prob Estimate Prob Accuracy of the test 100% 4.79< < % 4.25< < % 3.35< < % 3.11< <.0001 Quality of life Normal life 5.49< <.0001 Mild disability 3.08< <.0001 Partial disability 1.53< <.0001 Severe disability <.0001 Survival rate 85% 4.44< < % 2.89< < % < % 0.00 Normal life, 85% survival rate and 100% accuracy of the test are more favorable attributes for parents and clinicians. Severe disability is the only attribute that is less favorable than 40% survival rate. Comparing coefficients of mild disability for intermediate prognosis and good prognosis shows that parents and clinicians prefer mild disability over low of survival rate. For parents mild disability is more favorable than 70% survival rate.

Summary of results for clinicians in different prognosis 20 good prognosisintermediate prognosispoor prognosis AttributesTimes shown Times Selected Best Times Selected Worst AttributesTimes shown Times Selected Best Times Selected Worst AttributesTimes shown Times Selected Best Times Selected Worst Accuracy of the test 100% % % % % % % % % % % % Quality of life Normal life Normal life Normal life Mild disability Mild disability Mild disability Partial disability Partial disability Partial disability Severe disability Severe disability Severe disability Survival rate 100% % % % % % % % % % % % Best-Worst count score is equal to difference of times selected best and worst divided by times shown.

Best-Worst count score for clinicians’ preferences Quality of life has the most impact on clinicians’ decision making for good prognosis. Severe, partial and mild disability are least favorable attributes, respectively. Good prognosis attributes Best - Worst score Intermediate prognosis attributes Best - Worst score Poor prognosis attributes Best - Worst score Accuracy of the test 100% 25.8% 100% 47.5% 100% 53.9% 95% 9.1% 95% 37.5% 95% 45.8% 90% -11.0% 90% 5.7% 90% 15.2% 85% -25.6% 85% -9.0% 85% -0.2% Quality of life Normal life 33.9% Normal life 51.0% Normal life 56.4% Mild disability -42.3% Mild disability -12.3% Mild disability -2.9% Partial disability -64.5% Partial disability -36.1% Partial disability -18.4% Severe disability -74.1% Severe disability -63.5% Severe disability -49.6% Survival rate 100% 76.2% 85% 51.5% 55% 16.1% 95% 64.1% 70% 15.1% 40% -12.4% 90% 21.0% 55% -38.0% 25% -50.6% 80% -13.0% 40% -49.2% 10% -53.8%

Summary Strengths – Easy – Simple to calculate - no black box – Can be done online – Use of scores might give average applied researcher more confidence in results Weaknesses – Does not meet economists definition of a trade-off – Cannot on its own produce QALYs