CIHR Team in Microsimulation Simulation Technology for Applied Research (STAR) Montreal, November 24-25, 2009

Goal of the Meeting To develop a detailed plan for achieving the objectives of the project, specifically: –To identify the investigators responsible for each objective/subproject –To specify the final products (software, publications, course syllabus, etc.) –To specify milestones and deadlines for each product

NET Vision The end-product of this 5-year research program will be –a set of integrated, validated, transparent, and user-friendly disease simulation models –widely known and accessible to policy-makers and researchers within Canada and internationally –supported by extensive documentation and novel substantive results published in highly respected scientific journals

NET subprojects Model-specific subprojects General subprojects KTE-related subprojects Training-related subprojects The book subproject

Model-specific subprojects Breast cancer model: Validation? Colon cancer model: Validation? Diabetes model: Development/validation CHD model: Modification/Validation OA model: Further development/validation sensitivity analysis, applications For each subproject: –Final product from the NET –Person responsible –Milestones and deadlines –KTE aspects

General subprojects Validation framework and methods review Macrosimulation ontology Software development Integration of POHEM Application of integrated model: obesity Application of integrated model: health inequalities For each subproject: –Final product from the NET –Person responsible –Milestones and deadlines –KTE aspects

KTE-related subprojects NET website NET repository POHEM model documentation Advisory committee (policy makers) For each subproject: –Final product from the NET –Person responsible –Milestones and deadlines

Training Trainees and trainee awards –New investigators –Post-doc fellows –PhD students –Master’s students –Summer students –Other Course on microsimulation –Final product from the NET –Person responsible –Milestones and deadlines –KTE aspects

The Book Introduction to epidemiological simulation models Types of models, examples of models Microsimulation ontology Model development and validation framework Sensitivity analysis Statistical issues General description of POHEM Disease-specific POHEM models Multi-disease POHEM model Examples of applications The future of modeling

POHEM-OA

Age & Sex (CCHS 01) BMI (CCHS) OA Diagnosis HRQOL (CCHS 2001) Region Income Education (CCHS) Crude rates BCLHD Regression model NPHS Incidence model NPHS Tobit model CCHS 2001 OA Surgery OA Drugs Tobit model VGH 2007 Regression NPHS Crude rates BCLHD Crude rates BCLHD OA stage BCLHD Effect of drugs Literature Direct Costs Cost model Side-effects Co- morbidity Death Other risk factors Indirect costs No model at this time No model at this time Literature No model at this time No model at this time No model at this time POHEM-OA Literature

POHEM-OA – update Admin database linked to CCHS (for parameter validation): data for received, preliminary analyses initiated, waiting for data update (to 2008) Parameter validation in Ontario and Quebec: preliminary discussions completed, SAS codes will be ready to send out in December/January Sensitivity analysis: methodology partially developed, will continue next year Cost module: work will start in January Obesity application: detailed plan developed, actual simulations to start in December OA treatment application: preliminary plan developed, simulations to start next year (January/February)

POHEM-OA publication update Journal publications Kopec JA, Sayre EC, Flanagan W, Fines P, et al. Cibere J, Rahman M, Bansback N, Anis AH, Jordan JM, Sobolev B, Aghajanian J, Kang W, Greidanus NV, Garbuz DS, Hawker GA, Badley EM. Development of a population-based microsimulation model of osteoarthritis in Canada. Osteoarthritis Cartilage (in press) Presentations and abstracts Sayre EC, Finès R, Flanagan WM, Rahman MM, Kang W, Cibere J, Anis AH, Badley EM, Kopec JA. A Tobit model for predicting Health Utilities Index Mark 3 from osteoarthritis disease duration: a population-based study. To be presented at the Annual Scientific Meeting of the American College of Rheumatology, Philadelphia, October 16-21, 2009 Kopec JA, Finès P, Flanagan WM, Sayre EC, Rahman M, Bansback N, Cibere J, Anis H, Jordan JM, Badley EM. Projecting the burden of osteoarthritis in Canada using microsimulation. Presented at the Annual Meeting of the European League Against Rheumatism, Copenhagen, June 10-13, Finès P, Kopec JA, Flanagan WM, Sayre EC, Rahman M. Microsimulation of osteoarthritis in Canada – Case study of a chronic disease in Canada. Presented at the Meeting of the International Microsimulation Association, Ottawa, June 8-10, 2009.

Model validation: Conceptual issues

Agenda Validation framework Validation principles Question re: validity evidence from examining model development Questions re: validity evidence from examining model output Rating of validity evidence Validation-related subprojects

Validation framework: Sources of validity evidence Evidence from examining model development Conceptual model validity (theories, definitions, content, structure); Parameter validity (parameters based on expert opinion, literature, data analysis, databases, calibration); Computer program validity (type of simulation, software, code, internal organization) Evidence from examining model output Plausibility (face validity); Internal consistency; Parameter sensitivity; Between-model comparisons; Comparisons with external data; Evidence from examining the consequences of model-based decisions

Validation principles Models gain credibility through thorough development, extensive validation, and use Full and complete model validation is never possible, validation never ends A model can be valid (and validated) for one application and not valid (or validated) for another; for example, a model may be valid as an aid to decision making, but not as a forecasting tool, and vice versa Epidemiological microsimulation models such as POHEM models are developed for multiple purposes and should be validated accordingly

Validation principles – cont. Model validation studies are of relatively high interest and should not be too difficult to publish Published validation studies tend to increase model uptake by researchers Validity evidence based on examining model development process can/should be part of model description Probably the most powerful validation studies (but also the most difficult to do) are sensitivity analysis, between-model comparisons, and validations against external data Given the time and other constrains, we need to strike the right balance between model validation and applications

Evidence from examining model development Question re: Evidence of conceptual model validity, parameter validity and computer program validity –Should we try to include this type of validity evidence in all papers describing POHEM models? –How extensive should this evidence be? Should we follow our own guidelines/framework? –What other documentation should we develop to include those results that are not published or publishable?

Evidence from examining model output Questions re: parameter sensitivity, between-model comparisons, and comparisons with external data –Should this type of validation be part of the NET? –Is this type of validation equally important for all models? –Are all these sources of validity evidence equally important and feasible? –Do we know exactly how to do it? –If we do it, should we aim to publish all the results and if not, what other documentation should we develop to include those results that are not published?

Rating of validity evidence How many aspects have been validated? How detailed and transparent is the description of the validation of each aspect? How extensive is the validation of each aspect (many different approaches)? How quantitative is the validation of each quantitative aspect?

Validation-related subprojects Model validation framework Sensitivity analysis – a review Disease-specific POHEM models – further development/validation (breast, colon, CHD, diabetes, OA) Multi-disease POHEM model – description and validation

Example: Model validation framework Final product: paper/chapter Person responsible: Jacek Deadline: February 2010 (submission) KTE: publication

Training

Agenda Trainee awards Course development

Trainee awards How many What type How much When For how long For what subprojects Review of proposals

Course development Audience Content Level Delivery Availability Persons responsible Milestones and deadline KTE aspect