1. 1 CADTH Value Methods Panel Using Best Worst Scaling to elicit Values Carlo Marra

2. Problem 2 ?

3. Traditional DCEs Discrete Choice Experiments increasingly used in health services research Respondents choose a preferred specification of the good or service Aim is to obtain quantitative estimates of utility (benefit) associated with different attribute levels describing the good or service

4. Example of a DCE 4

6. Best-Worst Scaling Devised by Finn & Louviere (JPPM 1992) – introduced to health care by McIntosh & Louviere (HESG 2002) – statistical proof paper Marley & Louviere (J Math Psych 2005) –‘ user guide ’ by Flynn et al (JHE 2006) Differs from traditional DCEs in the nature of the choice task Individuals choose the best and the worst attribute based on the levels displayed in a given specification

7. Statistical issues MNL is (usually) a first step – Is there heterogeneity? – Likely covariates that characterise it? More complex methods? – Mixed logit – Latent class analysis Non/semi parametric

8. Problem 8 ?

9. Analyzing results To get around the dreaded BLACK BOX Best-minus-worst-scores Easy to understand Found to be linearly related to the ML estimates of the conditional logit model in virtually every empirical study to date Scores can help guide analysis of choice data – eg. LCAs which may give spurious associations 9

10. Stratifying and Targeting Pediatric Medulloblastoma 10

11. Purpose To determine preferences of the general population, parents and health professionals regarding trade-offs between treatment intensity and survival including test characteristics, functional outcomes, psychological outcomes and economic burden. 11

12. 12 Parents: N=76 participants Health professionals: N=193 participants General population: N= 3006 participants

13. 13 100% accuracy of test 95% accuracy of test 90% accuracy of test 85% accuracy of test 1. Accuracy of test: The possible levels of test accuracy in this survey are:

14. 14 2. QoL/ Functional ability (Side effects of the radiotherapy): The possible health states in this survey are: Child will have normal healthy life. Child will experience mild disability. Child will experience partial disability. Child will experience severe disability.

15. 3. Ten year survival rates: The possible levels of survival in this survey are: 15 Good prognosis Intermediat e prognosis Poor prognosis Baseline Survival Rate 90%70%40% Levels 100%85%55% 95%70%40% 90%55%25% 80%40%10%

16. Best and Worst Survey Design Clinicians’ Survey 16

17. Best-Worst estimated parameters (paired model) for general public good prognosisintermediate prognosispoor prognosis AttributesEstimateProbAttributesEstimateProbAttributesEstimateProb Accuracy of the test 100% 2.15<.0001 100% 3.81<.0001 100% 4.32<.0001 95% 1.08<.0001 95% 3.12<.0001 95% 3.57<.0001 90% 0.52<.0001 90% 2.45<.0001 90% 2.89<.0001 85% -0.26<.0001 85% 1.89<.0001 85% 2.37<.0001 Quality of life Normal life 2.97<.0001 Normal life 4.35<.0001 Normal life 4.92<.0001 Mild disability -0.98<.0001 Mild disability 1.07<.0001 Mild disability 1.78<.0001 Partial disability -1.58<.0001 Partial disability 0.59<.0001 Partial disability 1.29<.0001 Severe disability -3.21<.0001 Severe disability -1.28<.0001 Severe disability -0.53<.0001 Survival rate 100% 3.28<.0001 85% 3.01<.0001 55% 2.09<.0001 95% 2.29<.0001 70% 2.10<.0001 40% 1.40<.0001 90% 1.37<.0001 55% 0.56<.0001 25% 0.55<.0001 80% 0.00 40% 0.00 10% 0.00

18. Good prognosis Number of respondent 901 Attribute Times Shown Times Selected Best Best Count Proportion Times Selected Worst Worst Count Proportion Best - Worst score Accuracy of the test 100% 3612130736.2%1283.5%1179 95% 358861517.1%2146.0%401 90% 35972837.9%3309.2%-47 85% 36041634.5%70219.5%-539 Quality of life Normal life 3642209457.5%1554.3%1939 Mild disability 35972145.9%190352.9%-1689 Partial disability 36051413.9%282278.3%-2681 Severe disability 3604882.4%316287.7%-3074 Survival rate 100% 3596259172.1%1173.3%2474 95% 3596195454.3%1454.0%1809 90% 359591325.4%35910.0%554 80% 361244912.4%77521.5%-326 Baseline survival rate is 90%.

19. Best-Worst estimated parameters (paired model) for parents and clinicians – intermediate prognosis Attributes ParentsClinicians Estimate Prob Estimate Prob Accuracy of the test 100% 4.79<.00013.49<.0001 95% 4.25<.00012.94<.0001 90% 3.35<.00012.14<.0001 85% 3.11<.00011.54<.0001 Quality of life Normal life 5.49<.00013.67<.0001 Mild disability 3.08<.00011.38<.0001 Partial disability 1.53<.00010.90<.0001 Severe disability -0.460.035-0.68<.0001 Survival rate 85% 4.44<.00013.44<.0001 70% 2.89<.00012.19<.0001 55% 0.760.0030.61<.0001 40% 0.00 Normal life, 85% survival rate and 100% accuracy of the test are more favorable attributes for parents and clinicians. Severe disability is the only attribute that is less favorable than 40% survival rate. Comparing coefficients of mild disability for intermediate prognosis and good prognosis shows that parents and clinicians prefer mild disability over low of survival rate. For parents mild disability is more favorable than 70% survival rate.

20. Summary of results for clinicians in different prognosis 20 good prognosisintermediate prognosispoor prognosis AttributesTimes shown Times Selected Best Times Selected Worst AttributesTimes shown Times Selected Best Times Selected Worst AttributesTimes shown Times Selected Best Times Selected Worst Accuracy of the test 100% 92828546 100% 91347541 100% 90153751 95% 93014560 95% 90938241 95% 90146653 90% 93162164 90% 908157105 90% 90022083 85% 92639276 85% 91481163 85% 898125127 Quality of life Normal life 92635642 Normal life 91450842 Normal life 90355041 Mild disability 92837430 Mild disability 912104216 Mild disability 897132158 Partial disability 92737635 Partial disability 91151380 Partial disability 90172238 Severe disability 92759746 Severe disability 91465645 Severe disability 89964510 Survival rate 100% 92973628 85% 91252151 55% 901275130 95% 93063135 70% 910277140 40% 902134246 90% 92828893 55% 91373420 25% 89953508 80% 926109229 40% 91442492 10% 89872555 Best-Worst count score is equal to difference of times selected best and worst divided by times shown.

21. Best-Worst count score for clinicians’ preferences Quality of life has the most impact on clinicians’ decision making for good prognosis. Severe, partial and mild disability are least favorable attributes, respectively. Good prognosis attributes Best - Worst score Intermediate prognosis attributes Best - Worst score Poor prognosis attributes Best - Worst score Accuracy of the test 100% 25.8% 100% 47.5% 100% 53.9% 95% 9.1% 95% 37.5% 95% 45.8% 90% -11.0% 90% 5.7% 90% 15.2% 85% -25.6% 85% -9.0% 85% -0.2% Quality of life Normal life 33.9% Normal life 51.0% Normal life 56.4% Mild disability -42.3% Mild disability -12.3% Mild disability -2.9% Partial disability -64.5% Partial disability -36.1% Partial disability -18.4% Severe disability -74.1% Severe disability -63.5% Severe disability -49.6% Survival rate 100% 76.2% 85% 51.5% 55% 16.1% 95% 64.1% 70% 15.1% 40% -12.4% 90% 21.0% 55% -38.0% 25% -50.6% 80% -13.0% 40% -49.2% 10% -53.8%

22. Summary Strengths – Easy – Simple to calculate - no black box – Can be done online – Use of scores might give average applied researcher more confidence in results Weaknesses – Does not meet economists definition of a trade-off – Cannot on its own produce QALYs