Annotation and Image Markup : Take AIM at Images! David S. Channin M.D. Associate Professor of Radiology Chief, Imaging Informatics Northwestern University Feinberg School of Medicine Department of Radiology Daniel Rubin M.D. Clinical Assistant Professor of Radiology Research Scientist Stanford University Department of Medical Informatics Principal Investigators: Pat Mongkolwat PhD, Vladimir Kleper, Kaustubh Supekar

What is an Image Annotation? Annotations are explanatory or descriptive information, generated by humans or machines, directly related to the content of a referenced image or images Image annotations let us capture information about the meaning of pixel information in images such that similar meaning in other images can be found and used.

What is an Image Markup An image markup is the graphical symbols associated with an image and optionally with one or more annotations of that same image.

An Image

An Image and an Image Markup

An Image, an Image Markup and an Annotation The pixel at the tip of the arrow [coordinates (x,y)] in this image [DICOM: ] represents the Ascending Thoracic Aorta [SNOMED:A ]

What’s the problem? No agreed upon syntax for annotation and markup. No agreed upon semantics to describe annotations. No standard format (DICOM, XML, HL7, etc.) for annotations and markup.

Why is this important?

What is the solution? The caBIG AIM Project An ontology of image annotations An ontology of image markups An ontology defines concepts in a domain and the relationships between those concepts Use of controlled terminologies EVS, RadLex, SNOMED, LOINC, UCUM A set of translatable, standards-based representations

The Deliverables An ontology of both annotation and markup A UML model of AIM Software to instantiate AIM XML Software to generate DICOM S/R AIM Object (from AIM XML) Software to generate HL7 CDA (xml) (forthcoming) An XIP Builder SceneGraph and XIP modules to validate and transcode AIM annotations (ANIVATR) An AIM library to create and render AIM annotations and markups (Integrated into the eXtensible Imaging Platform)

The Deliverables 1Q 2007 Project Management Plan Communication Plan Draft IOSA Document Draft Reconciliation Document Final IOSA Document Final Reconciliation Document Quarterly Progress Report 2Q 2007 Draft Mechanism for Free Text Draft Mechanism for Arbitrary Calculation Final Mechanism for Free Text Final Mechanism for Arbitrary Calculation Sample InstancesSample Instances Quarterly Status Report 3Q 2007 Software DemonstrationSoftware Demonstration Import, validate, create, transcode, etc. Quarterly Status Report 4Q 2007 RSNA 2007 Demonstration Quarterly Status Report Lessons Learned

Logical Diagram AIM

Logical Diagram AIM enlarged

What is an AIM Annotation? One creator (machine or human) at one instant in time Annotating one series of images from one patient 9 Types of Image Annotations, 5 Types of Annotation of Annotations An annotation is assigned a unique identifier and can be assigned a name An annotation has one or more anatomic entity that may be related to each other An annotation has one or more imaging observations An imaging observation has more or more characteristics An annotation has one or more geometric shapes Geometric shapes are: point, multipoint, circle, ellipse, polyline An annotation has calculation(s) Calculations have result(s) Results have data, dimensions, ordinates Calculations can be defined or arbitrary Text can be intended for presentation (Text Annotation) Or not (intended just for reference) (Comments)

API Give me the XML schema for an XYZ AIM (Driven by Protégé) Constrain vocabulary choices Set the … Get the … Save As.. DICOM S/R XML/CDA

What does this look like in practice?

Select anatomic entity

Select imaging observation

Select calculation type

Select output type

Summary Image annotations and markups are critical to “tagging” content in medical images Such that images containing similar content can be identified The AIM project will deliver an information model and encoding standards for the structure and content of image annotations AIM annotations will be critical components of future image based research.