A Two Level Monte Carlo Approach To Calculating Expected Value of Sample Information:- How to Value a Research Design. Alan Brennan and Jim Chilcott, Samer Kharroubi, Tony O’Hagan University of Sheffield IHEA June 2003

What are EVPI and EVSI? Adoption Decision Choose policy with greatest expected ‘payoff’ Uncertainty Confidence interval e.g. |--------[]---------| implies Adoption Decision could be wrong Perfect Information perfect knowledge about parameter e.g. [] Value of Information quantify additional payoff obtained by switching adoption decision after obtaining additional data Expected Value of Information quantify average additional payoff obtained for range of possible collected data EVPI Expected value of obtaining perfect knowledge EVSI Expected value of obtaining a specific sample size

Typical Process Model Net benefit of alternative policies Probabilistic Sensitivity Analysis e.g. 1,000 samples … C-E plane CEACs Partial EVPIs identify single or groups of parameters Partial EVSIs propose specific data collections (n) calculate expected value of different samples

 = uncertain model parameters d = set of possible decisions NB(d, ) = net benefit (λ*QALY – Cost) for decision d, parameters  i = parameters of interest – possible data collection -i = other parameters (not of interest, remaining uncertainty) Expected net benefit (1) Baseline decision = (2) Perfect Information on i = two expectations Partial EVPI = (2) – (1) (3) Sample Information on i = Partial EVSI = (3) – (1)

Expected Value of Sample Information 0)Decision model, threshold, priors for uncertain parameters 1) Simulate data collection: sample parameter(s) of interest once ~ prior decide on sample size (ni) (1st level) sample the simulated data | parameter of interest 2) combine prior + simulated data --> simulated posterior 3) now simulate 1000 times parameters of interest ~ simulated posterior unknown parameters ~ prior uncertainty (2nd level) 4) calculate best strategy = highest mean net benefit 5) Loop 1 to 4 say 1,000 times Calculate average net benefits 6) EVSI parameter set = (5) - (mean net benefit | current information) e.g. 1000 * 1000 simulations

Bayesian Updating: Normal Prior 0= mean, 0= uncertainty in mean (standard deviation) = precision of the prior mean 2pop = patient level uncertainty ( needed for update formula) Simulated Data   = sample mean (further data collection e.g. clinical trial ). = sample variance = precision of the sample mean . Simulated Posterior N(1, 1 )

Normal Posterior Variance – Implications . 1 always < 0 If n is very small, 1 = almost 0 If n is very large, 1 = almost 0

Bayesian Updating: Beta / Binomial e.g. % responders to a treatment Prior % responders ~ Beta (a,b) Simulated Data n cases, y successful responders Simulated Posterior % responders ~ Beta (a+y,b+n-y)

Bayesian Updating: Gamma / Poisson e.g. no. of side effects a patient experiences in a year Prior side effects per person ~ Gamma (a,b) Simulated Data n samples, (y1, y2, … yn) from a Poisson distribution Simulated Posterior mean side effects per person ~ Gamma (a+ yi , b+n)

Bayesian Updating without a Formula WinBUGS Put in prior distribution Put in data MCMC gives posterior (‘000s of iterations) Use posterior in model Loop to next data sample Other approximation methods Conjugate Distributions

EVSI Results for Illustrative Model

Common Properties of EVSI curve Fixed at zero if no sample is collected Bounded above by EVPI, monotonic, diminishing returns ? EVSI (n) = EVPI * [1 – exp -a*sqrt(n) ]

Correct 2 level EVPI Algorithm 0)Decision model, threshold, priors for uncertain parameters 1) Simulate data collection: sample parameter(s) of interest once ~ prior decide on sample size (ni) (1st level) sample the simulated data | parameter of interest 2) combine prior + simulated data --> simulated posterior 3) now simulate 1000 times parameters of interest ~ simulated posterior unknown parameters ~ prior uncertainty (2nd level) 4) calculate best strategy = highest mean net benefit 5) Loop 1 to 4 say 1,000 times Calculate average net benefits 6) EVPI parameter set = (5) - (mean net benefit | current information) fixed at sampled value e.g. 1000 * 1000 simulations

Shortcut 1 level EVPI Algorithm 1) Simulate data collection: sample parameter(s) of interest once ~ prior decide on sample size (ni) (1st level) sample the simulated data | parameter of interest 2) combine prior + simulated data --> simulated posterior 3) now simulate 1000 times parameters of interest ~ simulated posterior unknown parameters ~ prior uncertainty (2nd level) 4) calculate best strategy = highest mean net benefit 5) Loop 1 to 4 say 1,000 times Calculate average net benefits 6) EVPI parameter set = (5) - (mean net benefit | current information) 2) fix remaining unknown parameters constant at prior mean value Accurate if .. (a) net benefit functions are linear functions of the -i for all d,i, and (b) i and -i are independent.

How many samples for convergence ? outer level - over 500 samples, converges to within 1% inner level - required 10,000 samples, to converge to within 2%.

How wrong is US 1 level EVPI approach for a non-linear model? E.g – squared every model parameter Adjusted R2 for simple linear regression = 0.86 Linear Model Non-linear Model

Maximum of Monte Carlo Expectations:- Upward Bias simple Monte Carlo estimates are unbiased But, bias occurs when use maximum of Monte Carlo estimates. E{max(X,Y)} > max{E(X),E(Y)}. E.g. X ~ N(0,1), Y ~ N(0,1) E{max(X,Y)} = 1.2 , max{E(X),E(Y)} = 1.0 If variances of X and Y reduces to 0.5, E{max(X,Y)} = 1.08 E{max(X,Y)} continues to fall variance reduces, but stays > 1.0 Illustrative model:- 1000*1000 not enough to eliminate bias. Increasing no. of samples reduces bias, since it reduces the variances of the Monte Carlo estimates.

Computation Issues: Emulators Gaussian Processes (Jeremy Oakley, CHEBS) F() ~ NB(d, ) Bayesian non-linear regression (complicated maths) Assumes only a smooth functional form to NB(d, ) Benefits Can emulate complex time consuming models with formula i.e. speed up each sample Can produce a quick approximation to inner expectation for partial EVPI Similar quick approximation for partial EVSI but only for one parameter

Summary 2 level algorithm for EVPI and EVSI is correct approach Bayesian Updating for Normal, Beta, Gamma OK Others – WinBUGS / approximations There are issues of computation Shortcut 1 level EVPI algorithm is accurate if … net benefit functions are linear and the parameters are independent. Emulators (e.g. Gaussian Processes) can be helpful