1. Chapter 5: AI Model Development and Validation
Nigam Shah
Stanford University
Hongfang Liu
Mayo Clinic
Anna Goldenberg
University of Toronto
Hossein Estiri
Harvard University
Suchi Saria
John Hopkins University
Jenna Wiens University of Michigan

2. About Introduction: Healthcare tasks by AI or augmented AI (700 words)
Bulk of the chapter: The AI model development and validation process (2700 words)
Closing: Discussion and recommendation (1300 words)

3. Section 5 A: Healthcare tasks by AI or augmented AI
We describe a hypothetical case, to motivate how predictions and classification can be used at the point of care.
Then zoom out to the table below:

4. Section 5 B: Good Practices in Development and Validation
5.B.1 Establishing clinical utility (400 words)
5.B.2 Considerations for model development (1000 words)
Expert-driven vs data-driven
Augmenting vs replacing
Selection of AI modeling framework
Supervised, Unsupervised, Reinforcement
5.B.3 Model development process (1300 words)
Problem formulation, Study population, Data extraction, Experimental pipeline this is the ‘technical validation’.

5. Challenges in developing and validating AI models for healthcare
Unmet healthcare needs
Data availability
Data quality
Data reproducibility
Data FAIRness
Model fairness and ethics (not yet covered)
Workflow

6. Section 5 X: Validation Should we go beyond technical validation?
Utility validation
External validation

7. Section 5 C: Discussion and recommendations
Data Quality
Education

8. Known missing items Training data/ Gold standards
Surrogates, Crowdsourcing, NLP, Silver Standards
Advance learning techniques
Active learning, distant supervision, boosting/bagging
Model fairness and ethics
Handling of text, images

9. Questions for the group
Could we drop the ‘Reproducibility crisis’ content?
What else can we drop?
Flow related suggestions:
For example, should we change chapter order, and place the current chapter 4 to be after the current chapter 5, and 6.