Markov Models: Overview Gerald F. Kominski, Ph.D. Professor, Department of Health Services

Markov Models: Why Are They Necessary? n Conventional decision analysis models assume: -Chance events -Limited time horizon -Events that do not recur n What happens if we have a problem with: -An extended time horizon, say, over a lifetime -Events can reoccur throughout a lifetime

Decision Tree for Atrial Fibrillation

State-Transition Diagram for Atrial Fibrillation Well Post-StrokeDead p 12 =0.2 p 22 =0.9 p 33 =1.0 p 11 =0.7 p 23 =0.1 p 13 =0.1 The probabilities for all paths out of a state must sum to 1.0. Death is known as an absorbing state, because individuals who enter that state cannot transition out of it.

Transition Probabilities Well Post- Stroke Dead Well Dead State of Current Cycle State of Next Cycle Transition probabilities that remain constant over time are characteristic of stationary Markov models, aka Markov chains

Markov Model Definitions n Any process evolving over time with uncertainty is a stochastic process, and models based on such processes are stochastic or probabilistic models n If the process is both stochastic and the behavior of the model in one time period (i.e., cycle) does not depend on the previous time period, the process is Markovian -The process has “lack of memory” -Even processes where the previous state does matter can be made Markovian through definition of temporary states know as tunnel states

Tunnel States Well Post-Stoke 1 Post-Stroke 2Post-Stroke 3 Post-Stroke Dead

Defining a Markov Model n Define the initial states n Determine the cycle length n Consider possible transitions among states n Determine transition probabilities n Determine utilities, and costs (if cost-effectiveness analysis), for each state

Evaluating Markov Models: Cohort Simulation State CycleWell Post- Stroke Dead Sum of Years Lived Survival 010, ,0002,0001,0009, ,9003,2001,9008, ,4303,8602,7107, ,4014,1603,4396, ,6814,2244,0955, ,1764,1384,6865, ,9595,2174, , , The data in the last column is used to produce a survival curve, aka a Markov trace.

Estimating Markov Models: Monte Carlo Simulation n Instead of processing an entire cohort and applying probabilities to the cohort, simulate a large number (e.g., 10,000) cases proceeding through the transition matrix -Monte Carlo simulation -TreeAge will do this for you quickly, without programming n The advantage of this approach is that it provides estimates of variation around the mean n Monte Carlo simulation is most valuable because it permits efficient modeling of complex prior history -Such variables are known as tracker variables

Example of a 5-State Markov Source: Kominski GF, Varon SF, Morisky DE, Malotte CK, Ebin VJ, Coly A, Chiao C. Costs and cost- effectiveness of adolescent compliance with treatment for latent tuberculosis infection: results from a randomized trial. Journal of Adolescent Health 2007;40(1):61-68.

Key Assumptions of the Markov Model VariableValue (Range)Reference Efficacy of IPT0.85 ( )19 Cost of treating active TB$22,500 ($17,000-$30,000)17 Cost of IPTVaries by study group and whether 6-month IPT is completed Current study TB cases per 100, ( )20 TB case fatality rate (varies with age)17 All-cause mortality rate per 100, ,476 (varies with age)National Center for Health Statistics, 1999 mortality tables Hepatotoxicity of IPT (age<35, started IPT) (age<35, completed IPT) 21 Hepatitis fatality rate Cost of treating IPT-induced hepatitis $11,250 ($8,500-$15,000)Authors’ assumption QALY – Healthy1.00 ( )Authors’ assumption QALY – Positive Skin Test, but Incomplete IPT0.90 ( )Authors’ assumption QALY – Active TB0.50 ( )Harvard Center for Risk Analysis QALY – IPT-induced hepatitis0.75 ( )Harvard Center for Risk Analysis Discount rate0.03 ( )Panel on Cost-Effectiveness