Agenda: - Certification Platform: performance for training - I round - II round -Feedback from naives - Beta testers proposals - Congress presentations - Paper check for axes XIV PMT meeting – Dec 5, 2012

Platform: Training Phase 2 Images (Scheltens=0,3) Feedback 6 Images (Scheltens=0,1,2,3,4,4) Feedback +-+- Certification set 10 Images (2 x Scheltens=0,1,2,3,4)Feedback +-+- Certification set

Platform: Training round I

Tracers received a verbal feedback in order to minimize misunderstanding possibly originated by the incomplete functionality of the platform at this stage, or due to the underlying statistical operations for the visual feedback. No verbal feedback for Round III Platform: Training rounds I and II

A qualitative check (QC) was carried out slice by slice, same procedure as for the benchmark images, but with the help of the platform: 1) Platform usually reflects properly the correctness of segmentation. Limitations lie in CSF pools (a.: correct segmentation is not properly rendered with green), and in the minimum variability allowed to the naïve where masters were in great agreement with each other (b.) a.b. Platform: Training round I

Platform Training round I: Performance * Minor lacks of compliance with HP; § Major lacks of compliance with HP; §§§ More relevant segmentation mistakes than just compliance to HP

Very atrophic sbj (Scheltens=4). Better distribute Sch=4 across training rounds next time? Platform Training round II: Performance

Training Platform Tracer Area (mm^2) Area delimited by the tracer contour for the current slice. Master Min Area (mm^2), Master Mean Area (mm^2), Master Max Area (mm^2) Area delimited by the master minimum/mean/maximum contour for the current slice. Distance Integral: Line integral of the distance ratio (see definition above for distance ratio) values along the tracer contour, normalized by the contours number of point, for the current slice. Hausdorff Distance (mm): Hausdorff distance (see definition above) between the tracer contour and the master mean contour for the current slice. Dice Index w.r.t. Min, Dice Index w.r.t. Mean, Dice Index w.r.t. Max Dice similarity index (see definition above) between the areas delimited by the tracer and master minimum/mean/maximum contours for the current slice. Jaccard Index w.r.t. Min, Jaccard Index w.r.t. Mean, Jaccard Index w.r.t. Max Jaccard similarity index (see definition above) between the areas delimited by the tracer and master minimum/mean/maximum contours for the current slice. True Positive Area w.r.t. Min (mm^2), True Positive Area w.r.t. Mean (mm^2), True Positive Area w.r.t. Max (mm^2) Tracer true positive (see definition above) area with respect to the area delimited by the minimum/mean/maximum master contour for the current slice. False Negative Area w.r.t. Min (mm^2), False Negative Area w.r.t. Mean (mm^2), False Negative Area w.r.t. Max (mm^2) Tracer false negative (see definition above) area with respect to the area delimited by the minimum/mean/maximum master contour for the current slice. False Positive Area w.r.t. Min (mm^2), False Positive Area w.r.t. Mean (mm^2), False Positive Area w.r.t. Max (mm^2) Tracer false positive (see definition above) area with respect to the area delimited by the minimum/mean/maximum master contour for the current slice.

Certification Criteria (to be defined) Hypothesis so far (generated versus quality check) is that: Jaccard < 70 denotes major problems in hippocampal segmentation Jaccard < 80 denotes incomplete compliance with the HP criteria A hypothesis might be that at least Dice 90 and Jaccard 80 be required for qualifying. Anyway we first evaluate performance at third round (Wide statistical analysis may be required for adequate criteria selection…)

Would find it easier to follow a video, replacing in person communication (video, and also inter-linked pop-up mesg -maybe for the first two training images?- OK but later) For now, legends to figures and table of contents are being added to the HP to facilitate reading HP edited based on appropriate feedback from tracers, and re-sent to tracers HP will be resent to panelists after we received comments from all naïves Visualization improvements (are being implemented already) Feedback from tracers

-testing Masami Nishikawa: Asks: access to protocol for validating VSRAD (common algorithm for hippo segmentation in Japan) Offers: Further validation branch for the SOPs project, on E-ADNI human phantoms (verbal offer by Matzuda of repeating segmentations on J-ADNI MRIs, having over 300 hippos segmented based on local protocol already)

- CTAD (Montecarlo, October ) Definition of the Harmonized Protocol for Hippocampal Segmentation - DGPPN Symposium (Berlin, November ) (Andreas Fellgiebel, Stefan Teipel) Segmentation of the hippocampus: Towards a joint EADC-ADNI harmonized protocol - Biomarkers for Brain Disorders (Cambridge, February ) Definition of harmonized protocol for hippocampal segmentation - AD/PD (Florence, March ) Definition of Harmonized Protocol for Hippocampal Segmentation - AAN (San Diego, March ) EADC-ADNI Benchmark Images of Harmonized Hippocampal Segmentation Congress Presentations

Papers describing the project Survey of protocols (preliminary phase; Published, JAD 2011) Operationalization (preliminary phase; Accepted, Alzheimers & Dementia, MS n. ADJ-D ) Axes check short report (Brescia Team, to be shared with coauthors) Delphi consensus (Brescia Team, in progress) Master tracers practice and reliability (Brescia Team, in progr) Development of certification platform (Duchesne and coll) Validation data (Brescia Team – companion paper 1) Protocol definition (Brescia Team – companion paper 2) Validation vs pathology (TBD) DONE IN PROGRESS PLANNED

VALIDATION VS CURRENT PROTOCOLS ASSESSMENT OF SOURCES OF VARIANCE TRAINING SET DEVELOPMENT VALIDATION VS PATHOLOGY GOLD STANDARD Harmonized Protocol ADNI scans: 2 x 5 Scheltenss atrophy score x 2 sides x 2 magnet strengths (1.5-3T) Total per rater: 40 hippos Harmonized Protocol ADNI scans: 2 sides x 5 Scheltenss atrophy scores x 3 time points (bl-1y-2y) x 3 scanners (+ bl) x 2 magnet strengths (1.5-3T) Total per rater: 240 hippos Assessment of variance due to rater and center Local Protocol ADNI scans: 2 x 5 Scheltenss atrophy scores x 2 sides x 2 magnet strengths (1.5-3T) Harmonized Protocol ADNI scans: 2 x 5 Scheltenss atrophy score x 2 sides x 2 magnet strength (1.5-3T) Total per rater: 40 hippos Harmonized Protocol: Pathological datasets: Mayo Clinic and NYU Total: about 40 hippos Training ADNI scans: 10 at 1.5T x 2 sides x 7 SUs x 2 tracing rounds Total per rater: 40 hippos 20 naïve tracers5 master tracers 1 tracer REFERENCE PROBABILISTIC MASKS with 95% C.I. QUALIFICATION Best 5 naïve tracers Assessment of variance due to side, trace-retrace, atrophy, time, scanner, rater TRAINING SET Assessment of agreement with volume on pathology or ex vivo MRI and correlation with neuronal density

GANTT