1. Decision and cost-effectiveness analysis: Understanding sensitivity analysis Advanced Training in Clinical Research Lecture 5 UCSF Department of Epidemiology and Biostatistics Feb 4, 2003

2. Objectives  To understand the purposes of sensitivity analysis.  To understand techniques used for sensitivity analysis.

3. Why do sensitivity analyses? All CEAs have substantial uncertainty All CEAs have substantial uncertainty Sensitivity analyses deal with that uncertainty systematically Sensitivity analyses deal with that uncertainty systematically Convince audience results are ‘robust’ – qualitative findings don’t change with small changes in inputs Convince audience results are ‘robust’ – qualitative findings don’t change with small changes in inputs

4. Sensitivity analysis Prior lectures reviewed how inputs are determined, plus a few simple sensitivity/threshold analyses. This lecture will cover four topics: This lecture will cover four topics: 1. Types of uncertainty 2. Deterministic sensitivity analyses (one-way, multi-way, scenario) 3. Probabilistic sensitivity analysis (Monte Carlo) 4. Uses of sensitivity analysis

5. Types of uncertainty Truth uncertainty – What’s are the correct input Truth uncertainty – What’s are the correct input values? values? Trait uncertainty – What if population characteristics Trait uncertainty – What if population characteristics or other circumstances change? or other circumstances change? Methodological uncertainty – What if the analysis Methodological uncertainty – What if the analysis were done differently? were done differently?

6. Deterministic Sensitivity Analyses How does assigning specific different values to inputs change output?  One-way (‘univariate’): Vary 1 input at a time  Multi-way (‘multivariate’): Vary 2+ inputs at a time  Scenario (variant of multi-way): Tests set of relevant conditions.  Threshold analysis (one-way or multi-way): Input values beyond which cost-effectiveness achieved ( or lost).

7. One-way SA – Aneurysm management:

8. Automating one-way SAs: Cotrimoxazole prophylaxis for HIV+ employees in Uganda

9. Two-way SA: CE of Mmpowerment program

10. Savings in parentheses “( )”; Cost per HIV case averted in brackets “[ ]” Low HIV prevalence setting: 25% in CSWs; 16.4% in clients Low HIV/AIDS treatment cost: $1,433Med. HIV/AIDS treatment cost: $2,507High HIV/AIDS treatment cost: $3,582 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 ($3,864)($1,824) $2,076 [$509] ($7,426)($5,386)($1,486)($10,989)($8,949)($5,059) Medium HIV prevalence setting: 50.3% in CSWs; 33.0% in clients Low HIV/AIDS treatment cost: $1,433Med. HIV/AIDS treatment cost: $2,507High HIV/AIDS treatment cost: $3,582 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 ($6,023)($3,983)($83)($11,203($9,163)($5,263) ($16,386) ($14,346)($10,446) Multivariate SA on female condom promotion: Net costs by HIV prevalence and key cost inputs for 1,000 CSWs

12. Threshold Analysis: NVP for prevention of vertical transmission of HIV in sub-Saharan Africa Input values needed for $50/DALY 15% HIV prevalence 30% HIV prevalence Regimen efficacy (47%) 18.0%10.6% VCT cost ($7.30)$18.50$36.00 HIV transmission (25.1%)9.6%5.6% HIV prevalence for $50/DALY4.5%

13. Probabilistic sensitivity analysis What is it? What is it good for?

14. Probabilistic sensitivity analysis Operational Definition : Outputs are calculated based on random assignment of values to inputs drawn from user- selected probability distribution. Examples: Monte Carlo, Latin Hypercube Software: Crystal Ball ®

15. The problem with deterministic SAs No estimate of the probability of achieving a particular outcome (Probabilistic SAs are the remedy)

16. Probabilistic sensitivity analyses Value Returns the likelihood of attaining particular outcome or outcome range. Everything known about each input expressed all at once. Particularly valuable when many inputs important. Drawback Need to know, or be able to make decent estimates of, the underlying probability distribution.

17. Monte Carlo simulation output

18. (the inner teachings) Other uses of sensitivity analysis (the inner teachings) Planning the analysis Debugging the model Documenting relationships between inputs and outputs Identifying thresholds Influencing policy

19. Other uses: Planning the analysis Program software to permit SAs on likely SA variables. SA curves provide a check on integrity of model. Identify candidates for more data collection early.

20. Other uses: Debugging the model Tricks of the trade One-ways best because simple and intuitive. Plug in extreme values. Separate diagnosis of numerator from denominator. Break outputs down further if necessary (intervention versus control arms).

21. Other uses: Documenting relationships between inputs and outputs Distinguish between ‘bugs’ and insights. Examples of insights: Slowing disease progression can increase costs. Higher disease prevalence can mean lower benefits. Benefits decrease with age - competing mortality risks.

22. Unexpected dynamic uncovered by SA: Female condoms study

23. Other uses: Identify thresholds – Influence Policy Preventing HIV vertical transmission in sub-Saharan Africa Cost of ARVs to prevent vertical transmission. Universal versus targeted provision of NVP.

24. Cost per DALY of HIVNET 012 NVP regimen as function of HIV seroprevalence and type of counseling/testing regimen

25. Summary SA is a set of techniques for the explicit management of uncertainty. Essential part of establishing key findings. Indispensable for convincing your audience that your results are technically sound and policy- relevant.