BIRS 2016: Opening the analysis black box: Improving robustness and interpretation Matthew Brown, PhD University of Alberta, Canada

Overview 1.About us 2.Preprocessing quality assurance 3.Interpretation of group vs. individual differences 4.Trial type fMRI signatures

Dept. Psychiatry Dept. Computing Science Computational Psychiatry Group

Diagnosis – What disease? Prognosis – Predict patient response to treatment options Clinical decision-making

What are we detecting? 10 psychosis patients, 10 controls, fMRI Highly diagnostic Fourier power distribution from voxels IN THE EYES Eye movement disturbances in psychosis

ADHD-200 and ABIDE datasets n=1000 approx. ADHD patients or autism patients Structural MRI, resting state fMRI Simple diagnosis – Classify patients vs. controls – Accuracy 50-70% in various papers Some papers reported higher 75%+ accuracy BUT cherry-picking sites?

ADHD-200 Global Competition Best-performing algorithm, but did not win Used only non-imaging features: – Age, gender, handedness, IQ, site of scan – 3-class classification (ADHD-c, ADHD-i, control) – 63% hold-out accuracy (vs. 54% chance) Using non-imaging features Brown et al Chance accuracy Validation Accuracy (%)

Histogram of oriented gradient (HOG) features Image from Ghiassian et al. under review. Also see Dalal and Triggs IEEE Computer Society Conference on. vol. 1. IEEE, p.886–893.

ADHD-200 and ABIDE datasets Ghiassian et al. under review State of the art (as of 1.5 years ago) 2-class classification (patients vs. controls) ADHD-200ABIDE Chance55%51% Non-imaging69%60% Non-imaging + Structural MRI 70%64% Non-imaging + Functional MRI 64%65%

Overview 1.About us 2.Preprocessing quality assurance 3.Interpretation of group vs. individual differences 4.Trial type fMRI signatures

Registration failure Subject 1Subject 15 Fixed -> Standard preprocessing methods failed for 1 of 21 subjects.

Inter-site variability Sen et al. in preparation PCA Component 1 PCA Component 2 ADHD-200 Subjects Projected onto PCA component space Each colour is a different scanning site. Even with standard normalization procedures, inter-site structure remains in the data.

Overview 1.About us 2.Preprocessing quality assurance 3.Interpretation of group vs. individual differences 4.Trial type fMRI signatures

Clinical research Huntington’s Image from Wikipedia Healthy One goal: Associate disease with biological features

ADHD-200 resting state fMRI functional connectivity analysis ICA Brown et al. 2012

ADHD patients vs. controls “Default mode” networkPatients vs. controls Brown et al “Desired” simple interpretation: “Patients are different from controls. This difference tells us something about the disease.”

Group vs. individual differences Patients Controls Statistically significant group differences, but substantial overlap between individual patients and controls. Brown et al. 2012

Interpretation Simple interpretation “patients are different from controls” Overlap precludes simple interpretation Yet many papers provide precisely and only the simple interpretation Patients Controls Brown et al. 2012

Overview 1.About us 2.Preprocessing quality assurance 3.Interpretation of group vs. individual differences 4.Trial type fMRI signatures

Black box analysis Analysis Software Analysis Software

General linear model regression Model voxel i’s timecourse Model matrix for trial type k

Two different models for hemodynamic response function SPM canonical model Finite impulse response model

Check deconvolved timecourses Basically agree on shape (but not statistical differences in this case) SPM canonical model Finite impulse response model, same region

Check deconvolved timecourses SPM canonical model Finite impulse response model, same region Noise in deconvolved timecourses

Another example SPM canonical model Finite impulse response model, same region Noise in deconvolved timecourses

GLM analysis Check deconvolved timecourses What is the model fitting – Noise vs. signal Model selection – regularization

Summary Quality check everything Visualization Intermediate steps and final results Particularly important for non-technical end-users

Acknowledgements People: Azad, Benoit, Dursun, Ghiassian, Greenshaw, Greiner, Juhas, Purdon, Ramasubbu, Rish, Sen, Silverstone Funding: AICML, AIHS, CIHR, Norlien Foundation, AHS, AMHB, UAlberta Questions?

Invitation Continue informing other researchers about analysis pitfalls and caveats. Questions?

Title