NA-MIC National Alliance for Medical Image Computing AHM 2006 Continuing Discussion of Validation

National Alliance for Medical Image Computing Thanks- Workshop & Today Ron Kikinis (BWH) Tina Kapur (Epiphany) David Tuch (MGH) Ross Whitaker (Utah) Jim Miller (GE Research) Sylvain Bouix (BWH) Dan Blezek (GE Research) Neculai Archip (BWH) Vincent A. Magnotta (Iowa) Nicole Grosland (Iowa) Guido Gerig (UNC) Peter Lyster (NIH) Group discussion But- We are not done yet!

National Alliance for Medical Image Computing Immediate Goal Define a coherent NA-MIC understanding of the issues, our responsibilities and implementation plan

National Alliance for Medical Image Computing Verification and Validation validation n 1: the act of validating; finding or testing the truth of something [syn: proof, substantiation] 2: the cognitive process of establishing a valid proof [syn: establishment]proofsubstantiationestablishment verification n 1: additional proof that something that was believed (some fact or hypothesis or theory) is correct; "fossils provided further confirmation of the evolutionary theory" [syn: confirmation, check, substantiation] 2: (law) an affidavit attached to a statement confirming the truth of that statementconfirmationchecksubstantiation validation Are we building the right product? To establish the fitness or worth of a software product for its intended mission (valere = to be worth) Do we have the right requirements? verification Are we building the product right? To establish the truth of correspondence between a software product and its specification (veritas = truth) Did we implement the requirements correctly? WordNet ® 2.0, © 2003 Princeton University Performance Design

National Alliance for Medical Image Computing Reminder It’s the biology that counts!

National Alliance for Medical Image Computing Summary- clarification of issues What is validation for NA-MIC? Level 1- machine performance Level 2- algorithm performanceLevel 2- acquisition parameters Level 3 - Workflow performance

National Alliance for Medical Image Computing Summary- examples What is validation for NA-MIC? Level 1- machine performance/platform Compare results of computer processors Dependence on compiler, number of mathematical operations, machine precision Level 2- algorithm performance Impact of “improvements” Relation between comparable algorithms Level 2- acquisition parameters Impact of “improvements” Level 3 - Workflow performance Stability of results in clinical studies Impact of sequence of algorithms

National Alliance for Medical Image Computing Summary- where we are now What is validation for NA-MIC? Level 1- machine performance/platform Core 2 has tools in place, Deployed to Core 1, implementation in process Level 2- algorithm performance *Core 1 study their own w/Core 3 data Cross algorithm study w/in NA-MIC in process Level 2- acquisition parameters ?collaboration w/M-BIRN ?collaboration w/others Level 3 - Workflow performance Initial efforts in progress *Methods/metrics for how to do this is research in and of itself

National Alliance for Medical Image Computing Proposal for NA-MIC Plan Level 1- machine performance Core 2 support & develop tools Core 1 to define metrics & tolerance levels Core 5 to develop training materials Level 2- algorithm performance/acquisition parameters Core 2 support & develop tools Core 1 or who? to develop provenance metadata Core 1 & Core 3 to define metrics* & tolerance levels Core 5 to develop training materials Level 3 - Workflow performance- NA-MIC role? Level 2-Cross algorithm study- NA-MIC role?

National Alliance for Medical Image Computing Proposal for NA-MIC Plan *Methods/metrics for how to validate the algorithms is research, how to summarize results how to visualize results how to relate results to clinical meaning Core 5 to coordinate lectures/workshops on this topic for upcoming NA-MIC Programmers Weeks and AHM

National Alliance for Medical Image Computing Proposed NA-MIC Deliverables Make available to the scientific community: Data sets for testing algorithms Core 2 Generic testing tools w/training materials to customize them for others to use in parallel manner to NA-MIC efforts