Integrating the Healthcare Enterprise Image Fusion Integration Profile Todd Kantchev Siemens Molecular Imaging IHE TC and Nuclear Medicine Meeting RSNA

IHE Nuclear Medicine – RSNA 2005 Terminology Image Registration- A process of changing the pixel spatial coordinates of a TDS so that the patient’s anatomical features are aligned with those of the SDS Rigid Registration- as defined in SRO Deformable Registration- out of scope Image Fusion- The process of superimposing (overlaying) two DS so that they can be seen as one combined image with the intention to complement features, present in both. DS- Data Set- a series of images or set of frames with a specified image position and orientation in the patient’s space, from which a 3-D image volume can be built and displayed for some clinical purpose. They may or may not have same orientation or be uniformly spaced in x-y-z. The requirements to build a volume from the DS are application specific and are outside the scope of this profile. SDS- Source data Set- a DS which is used as a base to which a TDS can be registered. TDS- Target Data Set- a DS, which is spatially transformed to the SDS by the process of Image Registration SRO- Spatial Registration Object- as defined in DICOM PS 3 Section A.39 BSPS- Blending Softcopy Presentation State as defined in DICOM Supplement 100. IHE Nuclear Medicine – RSNA 2005

Fusion Brain CT Brain MR Registration of medical images usually deals with two or more images or image sets, taken from the same patient anatomical area, for example brain CT and MR The goal is to have them aligned, superimposed and blended, so that we could see clinical information from both complementing modalities: CT (bone) and MR (soft tissue). The simplistic approach of aligning, without taking care for the imaging geometry does not work.

3-D Spatial Registration X Y Z X Y Z Source Data Set (SDS) Two Patient-based Coordinate Systems, defined by two Frames of Reference Target Data Set (TDS) Image Orientation (Patient)- x Image Position (Patient) Image Orientation (Patient)- y CT and MR series of images have been taken In 3D space on two acquisition devices at different time, hence their geometry in terms of image position/orientation (x,y) is encoded in DICOM in two different coordinate systems, identified by two Frames of Reference UIDs. Both coordinate systems have their origin in different points of the patient body. The goal of registration is to align the patient anatomical region of the Target Data Set (TDS)- chosen for some reason, with the same anatomical region of the Source Data Set (SDS), through translation, rotation, and scaling, to the best match. This process changes the coordinates of the original TDS, which is recorded as a transformation Once calculated and stored the transformation can be applied at any time. Align the patient anatomic region of the Target Data Set (TDS) to match the same anatomical region in the Source Data Set (SDS).

Registration=>Fusion (CT-MR) Translate Rotate Rigid Registration Scale Warp- optional (Deformable Registration)- not in scope Fuse (display superimposed) Translation, Rotation, Scaling (or generally affine) operations encompass Rigid Registration. The result can be complemented with an additional step of “warping” to match local misalignments. This is called Deformable Registration We use the term Fusion for the combined process of registration followed by superimposing and blending of the image sets for display.

IHE Nuclear Medicine – RSNA 2005 Normal Workflow On Evidence Creator Load Source Data Set (SDS) and Target Data Set (TDS). Find best anatomical matching of TDS to the SDS (usually pixel intensity statistical methods are used or manual). The methods are out of scope! Compute Transformation for the TDS (translate, rotate, scale) and apply Re-sample the TDS (take new axial slices in the new image orientation). The resolution usually is to the x-y-z resolution of the SDS. Fuse and display. Store Transformation On Image Display (possibly via IM) Load original SDS and TDS Apply the Transformation to the TDS Fuse and display Export Transformation Matrix: Pros: No need to store and communicate DTDS- save storage space and bandwidth. (Transformation Matrix (Rigid) is small compared to DTDS) Cons. Workstations which do not have capability to apply registration and re-sample image data cannot be used in the workflow. Ex. Basic Review WS Applying the registration is an additional step after loading the image in the display pipeline- extended time for loading Export Derived TDS: The receiving application may be inexpensive 3-D viewing workstation. It may not even have a 3-D viewing capabilities. Quick DTDS loading (quicker than loading OTDS and re-applying the registration) Cons: Re-sampled data may be too big (ex. PET res. to CT res.) IHE Nuclear Medicine – RSNA 2005

Use Case- Brain CT-MR Store-Retrieve-Apply Transformation Storage Post Processing Workstation or CAD Review Workstation Store CT Store MR Retrieve CT, MR Register MR to CT get T Fuse CT and MRR get BSPS Store T An example workflow for the first method is shown on this diagram. The workflow is managed by a centralized archive (PACS) and the use of DICOM Query Retrieve Service. The CT and the MR image data sets are stored on PACS after they are acquired and quality assured by a technologist. A physician on the Post-processing Workstation (PPWS) uses the Q/R mechanism to access the data at any convenient time later (assuming no electronic work list is in place). The data sets are registered and fused on the PPWS and the physician is able to perform a diagnosis. Then the Transformation Matrix object and the Blending Softcopy Presentation State are stored on the archive system for future use. A clinician uses a high-end Review Workstation to retrieve the CT, MR, the transformation matrix and the presentation state, apply the transformation and perform the image fusion. The clinician is then able to review the case. Store BSPS Retrieve CT, MR, T, BSPS Apply T to MR Fuse CT and MR

IHE Nuclear Medicine – RSNA 2005 Alternative Workflow On Evidence Creator Load Source Data Set (SDS) and Target Data Set (TDS) Find best anatomical matching of TDS to the SDS (usually pixel intensity statistical methods are used or manual). The methods are out of scope! Compute Transformation for the TDS (translate, rotate, scale) and apply Re-sample the TDS (take new axial slices in the new image orientation). The resolution usually is to the x-y-z resolution of the SDS. Fuse and display. Store the re-sampled TDS (resolution may differ) On Image Display (possibly via IM) Load original SDS and the re-sampled TDS Apply the Transformation to the TDS Fuse and display Export Transformation Matrix: Pros: No need to store and communicate DTDS- save storage space and bandwidth. (Transformation Matrix (Rigid) is small compared to DTDS) Cons. Workstations which do not have capability to apply registration and re-sample image data cannot be used in the workflow. Ex. Basic Review WS Applying the registration is an additional step after loading the image in the display pipeline- extended time for loading Export Derived TDS: The receiving application may be inexpensive 3-D viewing workstation. It may not even have a 3-D viewing capabilities. Quick DTDS loading (quicker than loading OTDS and re-applying the registration) Cons: Re-sampled data may be too big (ex. PET res. to CT res.) IHE Nuclear Medicine – RSNA 2005

Use Case- Brain CT-MR Store-Retrieve- Display Transformed TDS Storage Review Workstation Store CT Post Processing Workstation or CAD Store MR Retrieve CT, MR Register MR to CT get MRR Store MRR Fuse CT and MRR get BSPS Store BSPS An example workflow for the second method is shown on this diagram. The workflow is managed by a centralized archive (PACS) and the use of DICOM Query Retrieve Service. The CT and the MR image data sets are stored on PACS after they are acquired and quality assured by a technologist. A physician on the Post-processing Workstation (PPWS) uses the Q/R mechanism to access the data at any convenient time later (assuming no electronic work list is in place). The data sets are registered and fused on the PPWS and the physician is able to perform a diagnosis. Then the Transformed image data set (MRR) and the Blending Softcopy Presentation State are stored on the archive system for future use. A clinician uses a low-end Review Workstation to retrieve the CT, MRR, and the presentation state and perform the image fusion. The clinician is then able to review the case. Retrieve CT and MRR Retrieve BSPS Fuse CT and MRR

Pros and Cons Export Transformation Matrix: Pros: Save storage space and bandwidth (SRO is small compared with TDS) Cons. More expensive (Image Display higher cost) Slower Export Derived TDS: Less expensive (Image Display lower cost) Quicker Cons: Re-sampled data may be too big (ex. PET res. to CT res.) Export Transformation Matrix: Pros: No need to store and communicate DTDS- save storage space and bandwidth. (Transformation Matrix (Rigid) is small compared to DTDS) Cons. Workstations which do not have capability to apply registration and re-sample image data cannot be used in the workflow. Ex. Basic Review WS Applying the registration is an additional step after loading the image in the display pipeline- extended time for loading Export Derived TDS: The receiving application may be inexpensive 3-D viewing workstation. It may not even have a 3-D viewing capabilities. Quick DTDS loading (quicker than loading OTDS and re-applying the registration) Cons: Re-sampled data may be too big (ex. PET res. to CT res.)

Blending Presentation State SOP Class DICOM Standard provides a solution for storage of presentation of fused data (Supplement 100) The SDS would be used as an underlying image data set The TDS would be used as superimposed data set The MLUT and VOI LUT transformations are first applied as normal to both data sets The stored pseudo color will then be applied to the superimposed image data set. Both data sets will then be blended with the stored blending factor. The PCS color transformation would normally be Identity in this case (bypassed). DICOM Standard, Supplement 100

IHE Nuclear Medicine – RSNA 2005 Slide provided by David Clunie Blending for CT-PET select underlying select superimposed IHE Nuclear Medicine – RSNA 2005

IHE Nuclear Medicine – RSNA 2005 Slide provided by David Clunie Blending for CT-PET select underlying [register] select superimposed IHE Nuclear Medicine – RSNA 2005

IHE Nuclear Medicine – RSNA 2005 Slide provided by David Clunie Blending for CT-PET select underlying [register] select superimposed resample IHE Nuclear Medicine – RSNA 2005

IHE Nuclear Medicine – RSNA 2005 Slide provided by David Clunie Blending for CT-PET select underlying [register] within slices select superimposed resample IHE Nuclear Medicine – RSNA 2005

IHE Nuclear Medicine – RSNA 2005 Slide provided by David Clunie Blending for CT-PET select underlying [register] within slices select superimposed resample [between slices] IHE Nuclear Medicine – RSNA 2005

IHE Nuclear Medicine – RSNA 2005 Slide provided by David Clunie Blending for CT-PET select underlying rescale and window [register] within slices select superimposed resample [between slices] IHE Nuclear Medicine – RSNA 2005

IHE Nuclear Medicine – RSNA 2005 Slide provided by David Clunie Blending for CT-PET select underlying rescale and window [register] within slices select superimposed resample pseudo-color [between slices] IHE Nuclear Medicine – RSNA 2005

IHE Nuclear Medicine – RSNA 2005 Slide provided by David Clunie Blending for CT-PET select underlying rescale and window [register] blend within slices select superimposed resample pseudo-color [between slices] IHE Nuclear Medicine – RSNA 2005

Blending Presentation overlay transparency (blending factor) 20% 50% 80% 50% 50% 100% 50% IHE Nuclear Medicine – RSNA 2005

Interoperability for Image Registration We want to display fused data, generated from an acquisition device or workstation on another vendor’s system. Two possible solutions: Export the alignment solution (transformation matrix) and reproduce the transformation on the receiving workstation. (Supplement 73, Spatial Registration Storage SOP Classes - in the Standard since 2003) Export the transformed TDS and display it on the receiving workstation. Both methods are complemented by the use of Blending Softcopy Presentation State (BSPS) (Supplement 100: Color Softcopy Presentation State Storage SOP Classes- in the 2005 Standard) We want to display fused data, generated from one workstation on another vendor’s system. Two possible methods: Export the alignment solution (transformation matrix) and reproduce the transformation on the receiving workstation. (Supplement 73, Spatial Registration Storage SOP Classes - in the Standard since 2003) Export the transformed/re-sampled (derived) TDS and display it on the receiving workstation. Both methods are complemented by the use of Blending Softcopy Presentation State (BSPS) (Supplement 100: Color Softcopy Presentation State Storage SOP Classes- in the 2005 Standard)

The Problem DICOM Spatial Registration object enables a very broad context and does not guarantee consistency in terms of data referencing, workflow transactions etc. Blending Softcopy Presentation state object can be used to reference image series for fusion, but referencing can sometimes be ambiguous and vendors may choose not to support it or interpret it differently. Existing DICOM objects need to be constrained and further clarified in order to avoid misinterpretation and ambiguity. Require optional attributes to be present. Display capabilities such as multi-planar reformatting, color blending etc. which are essential to reading many types of fused studies, are either not supported by some display devices or are not consistently applied to achieve interoperability Sending fused data from a workstation to another system for further processing, display, and interpretation is currently loosely defined in standards to achieve the necessary multi-vendor inter-operability. Vendors are not accurately representing their support for such features

The Solution New Profile No New Actors or Transactions. Additional requirements on existing actors and transactions Clarifications of Fusion Workflow mapping to SWF Constraints on Fusion related Data Requirements/Recommendations on Fusion Clinical Display concepts, facilitating interoperability Connectathon to help test and confirm

Image Fusion Integration Profile Charge Posting Scheduled Workflow - Presentation of Grouped Procedures Reporting Workflow Patient Info. Recon-ciliation Post-Processing Workflow Image Fusion Image Fusion Consistent Present- ation of Images Evidence Docs Key Image Notes Simple Image & Numeric Reports Access to Radiology Information Portable Data for Imaging Basic Security

Affected Actors and Transactions 1: Patient Registration ADT 1: Patient Registration 12: Patient Update 12: Patient Update 2: Placer Order Management Order Filler Order Placer 3: Filler Order Management 6: Modality PPS in Progress 4: Procedure Scheduled 7: Modality PPS Completed 11: Image Availability Query 20: Creator PPS in Progress 12: Patient Update 21: Creator PPS Completed 13: Procedure Update C-FIND Evidence Creator Image Display Storage Commitment 20: Creator PPS in Progress 21: Creator PPS Completed 14: Query Images ­ 10: Storage Commitment ¯ 18: Creator Image Stored ¯ Performed Procedure Step Manager 16: Retrieve Images ­ C-STORE C-MOVE Image Manager Image Archive 6: Modality PPS in Progress 7: Modality PPS Completed 20: Creator PPS in Progress 10: Storage Commitment ­ 21: Creator PPS Completed 8: Modality Image Stored ­ Storage Commitment 6: Modality PPS in Progress C-STORE 7: Modality PPS Completed Acquisition Modality 5: Modality Worklist Provided

Acquisition Modality (Modality Images Stored [8]) Shall create image objects as specified (additional attributes mandatory, same Frame of Reference for hybrid modalities, etc.) Fusion related objects- SRO, SBPS, shall be consistent (referencing, blending etc.) Registration object shall be stored. The SRO shall be in the same Study as either SRS or TDS. If no additional alignment will be required before presentation, the SRO shall encode IDENTITY transformation. Blending PS object shall be required if the use of specific blending factor and color are essential to the clinical workflow and interpretation. The BSPS object shall be in the same Study as either SRS or TDS.

Evidence Creators- Required Creator Images Stored [18]) Shall be able to co-register and fuse data sets with images from multiple modalities (PET, CT, MR, NM in different combinations, including same modalities). The user shall be able to specify one of the input datasets to be the primary, so that registration uses that volume as the source (typically CT or MR). The user shall be able to specify which of the target volumes (typically PET or MR) to register to the source. Registration object shall be stored. The SRO shall be in the same Study as either SRS or TDS. If the transformed TDS is also stored and no additional alignment will be required before presentation, the SRO shall encode IDENTITY transformation. Storage Commitment- not required

Evidence Creators- Recommended Creator Images Stored [18]) Transformed TDS, if stored, may or may not be re-sampled to the same pixel and slice spacing, as SDS. The transformed target data set, if exported, shall be in the same frame of reference as the source data set and all other attributes shall be consistent with the new pixel data. Blending PS object shall be required if the use of specific blending factor and pseudo color are essential to the clinical workflow and interpretation. The BSPS object shall be in the same Study as either SRS or TDS.

Image Display- Required (Query Images [14], Retrieve Images [16]) Shall be able to: Fuse (superimpose) and display at least two previously registered data sets (for example a structural CT and a registered PET scan re-sampled to the same pixel and slice spacing). If Spatial Registration object for the pair is available it shall be used. Note: The ID may load and display data without an SRO present. However the fact that both DS are in the same Frame of Reference is not enough to guarantee that they can be fused. Display the fused data either color blended, or as a checker board (grayscale squares of the “checkerboard” are one image set, and the pseudo color squares are the other image set. ). If Blending SP object for the pair is available, it shall be used for initial presentation. Enable fusion-specific user interaction like transparency of fusion overlay (blending factor); color map selection; Window Width/Level control. Display MPR (Multi Planar Reconstruction) so that coronal and sagittal planes could be displayed from available transaxial data Display specific values like Derivation Code Sequence (0008,9215) or Derivation Description (0008, 2111), Series Description (0008,103E)

Image Display- Recommended (Query Images [14], Retrieve Images [16]) Should be able to: Read a SRO and apply transformation, fuse and display two data sets, which may have not been previously re-sampled to the same pixel and slice spacing. The pixel intensity interpolation method associated with this will not be specified, but may be recommended.

Image Manager/Archive (Modality Images Stored [8], Creator Images Stored [18] Query Images [14], Retrieve Images [16]) Shall be able to: Store necessary objects like PET, CT, MR, NM, Spatial Registration, Pseudo Color and Blending SP State, Real World Value Mapping etc. Support query and retrieval of necessary objects at series level with required specific keys

General Framework (from discussions so far) It will be a general Image Fusion profile (applicable to CT, MR, PET and NM), rather than be specific for PET-CT. SWF may not be required Fusion profile is not intended at this stage to be all-encompassing. For example, a reasonable limitation would be that it would deal with paired 3D data sets (no 2D, image stitching etc. ). No elaboration will be made at this stage on multidimensional presentation. Only generic features like 3D fusion/blending. Explain how SR and BSPS object reference TDS and SDS etc.

Open Issues/ Questions Do we need specific matching and return keys to be specified for query transactions? Shall we consider MPPS support for EC (Transactions 20,21 for Implicit Post-Processing- see Vol. 1- 3.3.5)? IHE Nuclear Medicine – RSNA 2005

IHE Nuclear Medicine – RSNA 2005 Table 1- Attributes Name Tag Acquisition Modality Evidence Creator Comment Related Series Sequence (0008,1250) May be R+ for hybrid PET-CT If so, may help ID to link SDS and TDS in case it cannot read SRO May extend CID 7210, Part 16. Mind NM which is MF Series Description (0008,103E) ?? yes Return key for ID/IM Derivation Code Sequence or Derivation Description (0008,9215) (0008, 2111) no Yes (if re-sampled) Matching key for ID/IM??? Request Attribute Sequence (0040,0275) Yes (if SWF supported) yes (if SWF) Already in SWF IHE Nuclear Medicine – RSNA 2005

IHE Nuclear Medicine – RSNA 2005 Expected Timeline Public comment preparation: December- January (TCON for Jan 6) Draft approved on a face to face meeting: Jan 30-Feb 03 2006 Released for public comment: Feb 17 2006 Public Comment closed : March 20 2006 Face to Face Meeting: April 3-7 2006, Chicago Publish for Trial Implementation: April 21 2006 Connectathon: End of 2006-June 2007 (possibly SNM) IHE Nuclear Medicine – RSNA 2005