Cheuk Yiu Ip Amitabh Varshney Joseph JaJa Hierarchical Exploration of Volumes Using Multilevel Segmentation of the Intensity-Gradient Histograms Cheuk Yiu Ip Amitabh Varshney Joseph JaJa

Volume Exploration Challenge Raw Volume Data Cube Meaningful Visualization [Voreen CG&A 09] Similar to other papers in this session Given a volumetric data cube on the left. We want to generate a meaningful visualization on the right.

Transfer Function Evolution How do we find the “right” transfer function? RGBA Traditionally, we use a transfer function to control the appearance of a visualization, the transfer function may look pretty complex. Finding the right transfer function has always been a challenge in scientific visualization Intensity

Histogram Helps Transfer Function Design To assist transfer function design, one of the first volume rendering papers shows a histogram may be helpful. [Drebin et al. SIGGRAPH 88]

Histogram Helps Transfer Function Design The peaks of the histogram are composed by the distributions of different constituents [Drebin et al. SIGGRAPH 88]

Histogram Helps Transfer Function Design By assigning different colors and opacities to these components. we can visualize different materials [Drebin et al. SIGGRAPH 88]

Histogram Helps Transfer Function Design For example, by following the peak in the histogram we can obtain this visualization of the cadaver head dataset

Histograms May Not Always Help However, the separation of the distributions may not be always clear

Volume Exploration Exhaustively explore the dataset In this case, we have to exhaustively explore the dataset by modifying the transfer function

Volume Exploration Seek for salient features We seek for the salient features one by one

Volume Exploration Seek for salient features

Volume Exploration Seek for salient features

Volume Exploration Feature locations can be arbitrary The meaningful feature locations can be pretty arbitrary

2D Transfer Functions Gradient ( f’(x) ) captures boundaries Histogram shapes ⇒ Volume segments To assist the search for meaningful and salient features, the transfer function has been extended to 2D by incorporating gradient. We plot a histogram in 2D with the intensity on the x axis, gradient on the y axis, the darkness of the shade represents the frequency The gradient helps capturing material boundaries. The amazing insight here is shapes on the 2D intensity-gradient histogram correspond to meaningful volumetric segments. This has been implemented in popular visualization packages such as voreen Imagevis3d and VisIt [Kindlmann & Durkin VolVis98, Kniss et al. TVCG 02] Gradient Intensity

Advances on New Attributes Higher Derivatives: f”(x) [Kindlmann & Durkin VolVis98] Specific features: Size [Correa and Ma TVCG 08] LH-transform [Sereda et al. TVCG 06], Domain specific semantic attributes [Salama et al. TVCG 06] Select good views: Visibility [Correa and Ma TVCG 10], Information divergence [Ruiz et al. TVCG 11] Many subsequent work has introduced new attributes such as higher derivatives, size, LH Transform, and even domain specific semantic attributes Visibility and information divergence have been used to select good views. For this work we will go back to intensity and gradient

Challenges of the 2D Transfer Function Search for separated meaningful features 1 Region 1 Minute Let’s take a look at the use of a 2D transfer function Users draw shapes on the 2D histogram to extract meaningful components This video shows how a user works with the histogram in ImageVis3D The arc structures can clearly be seen on the intensity gradient histogram The user is trying to use a polygon widget to highlight the feature that corresponds to the top arc. As we can see, manipulating the widget to separate the crown from the root takes quite some effort. The challenge here is the histogram does not show where do the good features separate. Extracting this crown takes almost 1 min

Histogram and Volume Features Let’s take a look at the vismale dataset

Histogram and Volume Features Skin

Histogram and Volume Features Flesh

Histogram and Volume Features Skull

Histogram and Volume Features Sinus

Histogram and Volume Features Teeth We can see that meaningful components do correspond to histogram segments. In this talk, my goal is to show you how to discover these regions in a systematic manner.

Approximate Histogram Transfer Functions Existing approaches directly or indirectly fit the histogram User Specified [Kniss et al. TVCG 02, Fogal et al. 2010] [Wang et al. TVCG 2011] Existing approaches directly or indirectly fit the 2D histogram to assist transfer function design These primitives may not cover the whole histogram completely to guarantee exhaustive exploration

Reduce Search to Classification Recursive histogram classification Tight coverage with a few segments Exhaustive exploration In this work, we attempt to reduce this search to classification by recursively segmenting the histogram. We introduce a visual-inspired approach that tightly fits the histogram with a few segments. We will show how to interactively and exhaustively explore the whole dataset

Overview Segment the histogram statistics Given a volume dataset, we mimic the user behavior by computationally segmenting the histogram statistics of the volume, we build an exhaustive multilevel hierarchy and use it to provide interactive exploration. Our approach provides a few advantages Visual segmentation matches user intuition, we guarantee tight and complete coverage, Our approach augments the intensity-gradient feature space without requiring the users to learn any new features Segment the histogram statistics Build an exhaustive multilevel hierarchy User interactive exploration

Overview Given a volume dataset, we mimic the user behavior by computationally segmenting the histogram statistics of the volume, we build an exhaustive multilevel hierarchy and use it to provide interactive exploration. Our approach provides a few advantages Visual segmentation matches user intuition, we guarantee tight and complete coverage, Our approach augments the intensity-gradient feature space without requiring the users to learn any new features Visual segmentation matches user intuition Complete coverage Users stay in a familiar feature space

Intensity-gradient Histogram The intensity gradient histogram is very effective because Human users can extract volume segments by recognizing the arcs and blobs Given these histograms, can we extract these shapes automatically? We mimic this user behavior by applying computer vision techniques to segment the histogram as an image in a top-down fashion Intensity Users implicitly recognize shapes from the histogram Segment this histogram as an image

Normalized-cut Image Segmentation Normalized-cut (ncut) image segmentation [Shi & Malik PAMI 98] Min ncut produces balanced segments Eigenanalysis approximates the min ncut for k segments B Instead of directly segmenting the volume data. We segment the histogram by using normalized-cut to find shapes This example shows the normalized cut provides intuitive segments on natural images. Normalize cut models the image as a graph, with the similarly of pixels as edge weights The weight of a cut in the graph is the total weights along the separating edges. shi and malik normalize this cut by the weights of the associated segments in the graph. The minimum normalize cut favors balanced segments We find the top k eigenvectors to approximate this minimum cut for k different segments [Wang et al. PATTERN RECOGN LETT 06]

Normalized-cut on Intensity-gradient Histogram We apply normalized-cut on 2562 8-bit histograms k=2 separates the tooth from the volume box k=10 shows segments of the tooth crown and root k=20 shows different material boundaries We apply normalized cut to segment 256 sq 8 bit histogram images. Eigenanalysis on these images only takes a few seconds When k = 2 the tooth is separated from the volume box k=10 shows segments of the tooth crown and root k = 20 shows different material boundaries. These segments tightly cover the whole histogram. So how many segments should we choose to compose our visualization?

Which k should we pick ? Iteratively picking k is tedious Increasing k may not subdivide region of user interest Try k = 2, 3, 4, … Iteratively test and pick k is time consuming and yet increasing k may not subdivide regions of interest. For example when we increase k from 4 to 5 , the box is subdivided. We believe users would prefer subdividing the tooth.

Replace k with User-driven Exploration Multilevel Segmentation Hierarchy: Apply normalized-cut recursively we need to allow users to select the appropriate segments instead of choosing a specific k, In order to accomplish this, we build a hierarchy by recursively applying the normalized cut.

Multilevel Segmentation Hierarchy Selectively inspect segments of choice This hierarchy leads the users to traverse the dataset, selectively subdivide and inspect segments of their choice

Multilevel Segmentation Hierarchy Any cut guarantees complete coverage View-dependent LoD hierarchies [Xia & Varshney Vis 96, Hoppe SIGGRAPH 97, Luebke & Erikson SIGGRAPH 97] Any cut along this hierarchy guarantees complete coverage. This is similar to view dependent level of details hierarchy

Multilevel Segmentation Hierarchy Having this flexibility, the next question is which segment should we subdivide?

Information Guided Traversal Segment entropy High entropy ⇒ Complex segment We evaluate information content of the segments to provide some guidance We use red to show which segment has a higher entropy for example the entropy of the tooth is higher than the volume box, suggesting the tooth is more complex, we subdivide the tooth. Entropy 6.8 2.8

What if the Entropies are Similar… The segment entropies can be similar Which segment should we divide next? Use Information Gain Entropy But, It is possible that the segment entropies are similar. For example there is no difference for the 2 segments in the right arc. Which segment should we divide next ? We use info gain to answer this question

Information-Gain Guided Traversal Information Gain = Entropy reduction after a subdivision High Information Gain ⇒ Structural separation we alternatively evaluate the information gain of a subdivision The information gain is defined as the entropy reduction of a subdivision Subdivisions with high information gain suggest separations of structures. We use blue to represent the information gain, for example splitting the lower segment with high information gain separates the surfaces of the tooth crown and dentine 0.01 Information Gain 0.11

Interactive Exploration Explore the segmentation hierarchy Selective expansion Interactive visualizations Exhaustively explore the tooth in 1 minute Here we show how a user can interactively explore the segmentation hierarchy, the user selectively click on the histogram to expand the segments and interactively compose visualizations Notice that the tooth crown in the upper arc can be identified in only 3 expansions. Users iteratively expand the regions of interest and hide uninteresting regions. A visualization of the volume dataset is composed along this hierarchy traversal In addition to identifying different meaningful segments, users can also segment and hide the undesired noise In contrast with extracting a single region. We can now exhaustively explore the whole tooth in 1 min.

Examples Engine block Visible Human Male Head Let’s take a look at some results Internal components Engine block and surface Skull, flesh, skin, highlight sinus and teeth

Examples Tomato Hurricane Isabel Tomato Skin, Chamber, Center Column Isabel pressure, with no distinct material boundaries Eye, Eye Wall, Rain band, and further segment the eye

Conclusions Computational segmentation mimics user interactions Intuitive volumetric classification Exhaustive multilevel hierarchy Information guided traversal Interactive exploration In conclusion, we have shown the idea of using computational visual segmentation to effectively mimic user interaction. It classifies intuitive volumetric regions. We have built an exhaustive multilevel hierarchy from these segments and We have provided information content to guide the hierarchy traversal Users can interactively explore and visualize with these ingredients.

Future Work Improve the information content measures Automatic color assignment for segments Segment histograms with different attributes Time varying datasets

Acknowledgements Thank you! National Science Foundation: CCF 05-41120, CMMI 08-35572, CNS 09-59979 NVIDIA CUDA Center of Excellence Program Derek Juba, Sujal Bista, Yang Yang, M. Adil Yalcin, and the reviewers for improving this paper and presentation The SciVis Best Paper Award committee Thank you!

Questions ? Source code for building the hierarchy: www.cs.umd.edu/~ipcy/software/volsegtree/ Papers and videos: Cheuk Yiu Ip www.cs.umd.edu/~ipcy/ GVIL Research Highlights www.cs.umd.edu/gvil/