1. September 2012 Update September 13, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench WITH FUNDING SUPPORT PROVIDED BY NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

2. Agenda for Today Update on statistical analysis library modules, including conceptual development of aggregate uncertainty (Jovanna) Overview of functionality in Reference Data Set Manager staged for the development iteration (Patrick) 22222222

3. Unifying Goal of 2 nd Development Iteration Perform end-to-end characterization of vCT including meta-analysis of literature, incorporation of QIBA results, and "scaled up" using automated detection and reference volumetry method. Integrated characterization across QIBA, FDA, LIDC/RIDER, Give-a-scan, Open Science sets (e.g., biopsy cases), through analysis modules and rolling up to an i_ file in zip archive. Specifically have people like Jovanna, Ganesh, and Adele to use it (as opposed to only Gary, Mike/Patrick, and Kjell) 333333333

4. Analyze: Update on Library Modules 444 ModuleSpecificationRepositoryStatus Meta-analysis extraction Extract observation data from literature reports CalculateReadingsFromStatistics.R CalculateReadingsAnalytically.m CalculateReadingsFromMeanStdev.m Three reporting styles are supported Method Comparison Radar plots and related methodology based on readings from multiple methods on data set with ground truth CMI3A_1 5 PercentErr v1_13_sent11May2012.R Currently have 3A pilot in R, not yet generalized but straightforward to do so. Plan to refine based on Metrology Workshop results and include case of comparison without truth also. Bias and Linearity According to Metrology Workshop specifications AnalyzeBiasAndLinearity.RComplete except for minor code improvements. Variability and Variance Components Analysis Accepts as input fractional factorial data of cross-sectional biomarker estimates with range of fixed and random factors, produces mixed effects model PerformBlandAltmanAndCCC.R ModelLinearMixedEffects.R Complete at this stage of development, but presently the support for multiple timepoints does not use repeated measures analysis which it probably should. Treatment Effect and Variance Components Assessment Accepts as input longitudinal change data, estimates variance due to treatment and non-treatment factors ModelLinearMixedEffectsCHG.RPrototyped, presently being refined. Aggregate Uncertainty Accepts as input multiple s_ files, produce intermediate i_ file result, and then use it to derive aggregate based on components CalculateAggregateUncertainty.mPrototyped (with team discussion today)

5. Analyze: Validation Go to Jovanna’s desktop 555

6. Analyze: Aggregate Uncertainty Objective: comprehensively characterize the performance of an imaging biomarker. Two orthogonal considerations: – Breadth of data used: use as much data as you can, regardless of where it comes from! – Nature of study designs that result in determination of uncertainty components Approach: – Utilize common analytical pipeline to place literature and heterogeneous study results onto a common plane (this motivates the file conventions that drive the library design) – Roll-up separate components into an aggregate: current WIP for discussion 666

7. Analyze: Aggregate Uncertainty Go back to Jovanna’s desktop 777

8. Execute: Basic Plan Generalize processing framework from the previous development year Support user-in-the-loop processing workflows for certain data-processing tasks Refine input and output formats to adhere to newer standards (AIM 4.0, DICOM Segmentation Objects, etc.) 888

9. Execute: Implementation Support for Radiological Worklists and DICOM Query and Retrieve directly from Execute – Batchmake scripts initiate worklist item delegation and reader stations can retrieve those datasets from Midas as they would a PACS Generalize processing API harness to allow arbitrary algorithm runs on arbitrary datasets. Optimize the web API and scripting interface to allow more seamless interaction with other QI- Bench applications 999

10. 10

11. Value proposition of QI-Bench Efficiently collect and exploit evidence establishing standards for optimized quantitative imaging: – Users want confidence in the read-outs – Pharma wants to use them as endpoints – Device/SW companies want to market products that produce them without huge costs – Public wants to trust the decisions that they contribute to By providing a verification framework to develop precompetitive specifications and support test harnesses to curate and utilize reference data Doing so as an accessible and open resource facilitates collaboration among diverse stakeholders 11

12. Summary: QI-Bench Contributions We make it practical to increase the magnitude of data for increased statistical significance. We provide practical means to grapple with massive data sets. We address the problem of efficient use of resources to assess limits of generalizability. We make formal specification accessible to diverse groups of experts that are not skilled or interested in knowledge engineering. We map both medical as well as technical domain expertise into representations well suited to emerging capabilities of the semantic web. We enable a mechanism to assess compliance with standards or requirements within specific contexts for use. We take a “toolbox” approach to statistical analysis. We provide the capability in a manner which is accessible to varying levels of collaborative models, from individual companies or institutions to larger consortia or public-private partnerships to fully open public access. 12

13. QI-Bench Structure / Acknowledgements Prime: BBMSC (Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette, Kjell Johnson, Jovanna Danagoulian) Co-Investigators – Kitware (Rick Avila, Patrick Reynolds, Julien Jomier, Mike Grauer) – Stanford (David Paik) Financial support as well as technical content: NIST (Mary Brady, Alden Dima, John Lu) Collaborators / Colleagues / Idea Contributors – Georgetown (Baris Suzek) – FDA (Nick Petrick, Marios Gavrielides) – UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad, Yelena Yesha) – Northwestern (Pat Mongkolwat) – UCLA (Grace Kim) – VUmc (Otto Hoekstra) Industry – Pharma: Novartis (Stefan Baumann), Merck (Richard Baumgartner) – Device/Software: Definiens, Median, Intio, GE, Siemens, Mevis, Claron Technologies, … Coordinating Programs – RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao) – Under consideration: CTMM TraIT (Andre Dekker, Jeroen Belien) 13