Chapter 5: AI Model Development and Validation Nigam Shah Stanford University Hongfang Liu Mayo Clinic Anna Goldenberg University of Toronto Hossein Estiri Harvard University Suchi Saria John Hopkins University Jenna Wiens University of Michigan

About Introduction: Healthcare tasks by AI or augmented AI (700 words) Bulk of the chapter: The AI model development and validation process (2700 words) Closing: Discussion and recommendation (1300 words)

Section 5 A: Healthcare tasks by AI or augmented AI We describe a hypothetical case, to motivate how predictions and classification can be used at the point of care. Then zoom out to the table below:

Section 5 B: Good Practices in Development and Validation 5.B.1 Establishing clinical utility (400 words) 5.B.2 Considerations for model development (1000 words) Expert-driven vs data-driven Augmenting vs replacing Selection of AI modeling framework Supervised, Unsupervised, Reinforcement 5.B.3 Model development process (1300 words) Problem formulation, Study population, Data extraction, Experimental pipeline this is the ‘technical validation’.

Challenges in developing and validating AI models for healthcare Unmet healthcare needs Data availability Data quality Data reproducibility Data FAIRness Model fairness and ethics (not yet covered) Workflow

Section 5 X: Validation Should we go beyond technical validation? Utility validation External validation

Section 5 C: Discussion and recommendations Data Quality Education

Known missing items Training data/ Gold standards Surrogates, Crowdsourcing, NLP, Silver Standards Advance learning techniques Active learning, distant supervision, boosting/bagging Model fairness and ethics Handling of text, images

Questions for the group Could we drop the ‘Reproducibility crisis’ content? What else can we drop? Flow related suggestions: For example, should we change chapter order, and place the current chapter 4 to be after the current chapter 5, and 6.