This presentation is made available through a Creative Commons Attribution- Noncommercial license. Details of the license and permitted uses are available at © 2013, 2014 Juliet Pulliam, Steve Bellan, Clinic on the Meaningful Modeling of Epidemiological Data, ICI3D Programhttp://creativecommons.org/licenses/by-nc/3.0/ Likelihood Fitting and dynamic models, Part I: Dynamic Model Fitting And Inference Robustness (Updated: 8 June 2014) Attribution: JRC Pulliam and SE Bellan, Clinic on the Meaningful Modeling of Epidemiological Data Source URL: For further information please contact

Non-exponential Waiting Times Public Health, Epidemiology, and Models Introduction to Dynamics of Infectious Diseases Foundations of Dynamic Modeling (Hidden) Assumptions of Simple ODE’s Breaking Assumptions! Discrete Individuals and Finite Populations Consequences of Heterogeneity Difference Equations and Discrete Time Intervals Heterogeneity tutorial Gillespie, Stochastic R-F, Chain Binomials Introduction to Thinking about Data Introduction to Infectious Disease Data Thinking about Data II: Confounding, bias, and noise Data Cleaning and Database Management Formulating Research Questions Creating a Model World Variability, Sampling, and Simulation Introduction to Statistical Philosophy Introduction to Likelihood HIV Spreadsheets Study Design and Analysis: Epi methods and RCT’s Integration!

Non-exponential Waiting Times Public Health, Epidemiology, and Models Introduction to Dynamics of Infectious Diseases Foundations of Dynamic Modeling (Hidden) Assumptions of Simple ODE’s Breaking Assumptions! Discrete Individuals and Finite Populations Consequences of Heterogeneity Difference Equations and Discrete Time Intervals Heterogeneity tutorial Gillespie, Stochastic R-F, Chain Binomials Introduction to Thinking about Data Introduction to Infectious Disease Data Thinking about Data II: Confounding, bias, and noise Data Cleaning and Database Management Formulating Research Questions Creating a Model World Variability, Sampling, and Simulation Introduction to Statistical Philosophy Introduction to Likelihood HIV Spreadsheets Fitting Dynamic Models I: A conceptual framework Study Design and Analysis: Epi methods and RCT’s Integration!

Likelihood fitting and dynamic models, Part 1 Juliet Pulliam, PhD Department of Biology and Emerging Pathogens Institute University of Florida and RAPIDD Program, DIEPS Fogarty International Center US National Institutes of Health Clinic on the Meaningful Modeling of Epidemiological Data ICI3D Program and AIMS - South Arica June 6, 2014

Process Model S I

S I Parameters

Process Model S I Parameters Where do parameters come from?

A priori parameterization  Use external data to determine values for the parameters in your model CASCADE study

A priori parameterization  Use external data to determine values for the parameters in your model  eg, time from seroconversion to death  Plug estimates into models to determine expected dynamics

A priori parameterization  Long-term time series are not available  Designing a new study  Data are limited and your goal is to estimate a particular quantity that has not been directly measured  Comparing model structures, especially when multiple long-term time series are not available for validation

Fitting models to data  A priori parameterization  Use external data to determine values for the parameters in your model  Rarely possible for all model parameters

Fitting models to data  A priori parameterization  Use external data to determine values for the parameters in your model  Rarely possible for all model parameters  Trajectory matching  Feature matching

Process Model S I Parameters some (possibly) fixed and others to be fitted

Time series Process Model S I Parameters some (possibly) fixed and others to be fitted

Process Model S I Parameters some (possibly) fixed and others to be fitted Time series expectation or distribution of latent variables

Process Model S I Parameters some (possibly) fixed and others to be fitted Time series expectation or distribution of latent variables Deterministic models

Process Model S I Parameters some (possibly) fixed and others to be fitted Time series expectation or distribution of latent variables Deterministic models Stochastic models

Data

n = 3092

Data Observation model n = 3092 PDF:

Likelihood of prevalence (given data) Data Observation model n = 3092 LIKELIHOOD: PDF:

Data

Observation model PDF:

Likelihood of prevalence trajectory (given data) Data Observation model LIKELIHOOD: PDF:

Likelihood of parameters (given data) Data Observation model Time series expectation or distribution of latent variables Process Model S I Parameters some (possibly) fixed and others to be fitted

Likelihood of the model (given data) Data Observation model Time series expectation or distribution of latent variables Process Model S I Parameters some (possibly) fixed and others to be fitted

Process Model S I Why do we fit models to data in infectious disease epidemiology?

Koopman’s Inference Robustness Assessment Framework Koopman et al Transmission modeling to enhance surveillance system function.

Koopman’s Inference Robustness Assessment Framework 1. Select the policy inference to be pursued Koopman et al Transmission modeling to enhance surveillance system function.

Koopman’s Inference Robustness Assessment Framework 2. Construct & Analyze Simple Models 1. Select the policy inference to be pursued Koopman et al Transmission modeling to enhance surveillance system function.

Koopman’s Inference Robustness Assessment Framework 2. Construct & Analyze Simple Models 3. Constrain Parameter Space with Data 1. Select the policy inference to be pursued Koopman et al Transmission modeling to enhance surveillance system function.

Koopman’s Inference Robustness Assessment Framework 2. Construct & Analyze Simple Models 3. Constrain Parameter Space with Data 4. Make inference across parameter space 1. Select the policy inference to be pursued Koopman et al Transmission modeling to enhance surveillance system function.

Koopman’s Inference Robustness Assessment Framework 2. Construct & Analyze Simple Models 3. Constrain Parameter Space with Data 4. Make inference across parameter space Inference Same Across Parameter Space Inference Differs Across Parameter Space 1. Select the policy inference to be pursued Koopman et al Transmission modeling to enhance surveillance system function.

Koopman’s Inference Robustness Assessment Framework 2. Construct & Analyze Simple Models 3. Constrain Parameter Space with Data 4. Make inference across parameter space Inference Same Across Parameter Space 5. Relax Assumptions with More Realistic Model Inference Differs Across Parameter Space 1. Select the policy inference to be pursued Koopman et al Transmission modeling to enhance surveillance system function.

Koopman’s Inference Robustness Assessment Framework 2. Construct & Analyze Simple Models 3. Constrain Parameter Space with Data 4. Make inference across parameter space Inference Same Across Parameter Space 5. Relax Assumptions with More Realistic Model Inference Differs Across Parameter Space 6. Find other types and sources of available data OR study cost benefit of new data collection to justify getting new data. 1. Select the policy inference to be pursued Koopman et al Transmission modeling to enhance surveillance system function.

Koopman’s Inference Robustness Assessment Framework 4. Make inference across parameter space Inference Same Across Parameter Space 5. Relax Assumptions with More Realistic Model Koopman et al Transmission modeling to enhance surveillance system function. Inference Robustness Assessment Loop

Koopman’s Inference Robustness Assessment Framework 3. Constrain Parameter Space with Data 4. Make inference across parameter space Inference Differs Across Parameter Space 6. Find other types and sources of available data OR study cost benefit of new data collection to justify getting new data. Koopman et al Transmission modeling to enhance surveillance system function. Inference Identifiability Assessment Loop

Assess inference robustness to realistic relaxation of simplifying model assumptions Koopman’s Inference Robustness Assessment Framework Koopman et al Slide courtesy of JS Koopman (with modification).

Assess inference robustness to realistic relaxation of simplifying model assumptions Pursue complexity that matters by keeping models as simple as possible but not so simple that they lead to an incorrect inference Koopman’s Inference Robustness Assessment Framework Koopman et al Slide courtesy of JS Koopman (with modification).

Assess inference robustness to realistic relaxation of simplifying model assumptions Pursue complexity that matters by keeping models as simple as possible but not so simple that they lead to an incorrect inference Koopman’s Inference Robustness Assessment Framework Koopman et al Slide courtesy of JS Koopman (with modification). Validate the inference!

Assess inference robustness to realistic relaxation of simplifying model assumptions Pursue complexity that matters by keeping models as simple as possible but not so simple that they lead to an incorrect inference Koopman’s Inference Robustness Assessment Framework Koopman et al Slide courtesy of JS Koopman (with modification). Validate the inference! not the model or method you’re working with

Non-exponential Waiting Times Public Health, Epidemiology, and Models Introduction to Dynamics of Infectious Diseases Foundations of Dynamic Modeling (Hidden) Assumptions of Simple ODE’s Breaking Assumptions! Discrete Individuals and Finite Populations Consequences of Heterogeneity Difference Equations and Discrete Time Intervals Heterogeneity tutorial Gillespie, Stochastic R-F, Chain Binomials Introduction to Thinking about Data Introduction to Infectious Disease Data Thinking about Data II: Confounding, bias, and noise Data Cleaning and Database Management Formulating Research Questions Creating a Model World Variability, Sampling, and Simulation Introduction to Statistical Philosophy Introduction to Likelihood HIV Spreadsheets Fitting Dynamic Models I: A conceptual framework Study Design and Analysis: Epi methods and RCT’s Integration!

Non-exponential Waiting Times Public Health, Epidemiology, and Models Introduction to Dynamics of Infectious Diseases Foundations of Dynamic Modeling (Hidden) Assumptions of Simple ODE’s Breaking Assumptions! Discrete Individuals and Finite Populations Consequences of Heterogeneity Difference Equations and Discrete Time Intervals Heterogeneity tutorial Gillespie, Stochastic R-F, Chain Binomials Introduction to Thinking about Data Introduction to Infectious Disease Data Thinking about Data II: Confounding, bias, and noise Data Cleaning and Database Management Formulating Research Questions Creating a Model World Variability, Sampling, and Simulation Introduction to Statistical Philosophy Introduction to Likelihood HIV Spreadsheets Fitting Dynamic Models I: A conceptual framework Monte Carlo Markov Chains Demographic Stochasticity and Ensemble Models Study Design and Analysis: Epi methods and RCT’s Integration! Fitting Dynamic Models II: Practical considerations