The importance of visualization 12° 0° V1 V1 Calcarine Calcarine V1 Calcarine Upper lip Upper lip Lower lip Lower lip Left hemisphere gyri sulci 1cm

The importance of visualization Precision Understanding, understanding, understanding

Visualization should offer precise communication Why precision? (a) (b)

Visualization should offer precise communication Why precision? Neighborhoods – topology Measurement of distances, surface areas, and volumes

Cortical Topology: distances on the volume and cortical surface are not the same 15mm 5mm Figure 5. A problem with volume-based data visualizations is reconciled using cortical surface visualizations in single subjects. Left: Example axial slice from a single subject. Middle: Zoomed portion surrounding the posterior inferotemporal sulcus indicated by the dotted red outline. Two regions of interest (green, red) are illustrated in different anatomical locations that would appear to be one contiguous region using large functional voxels (3-5mm on a side) and spatial smoothing (e.g. Figures 3-4). Notably, neurons close to one another in volume space due to the sulcal and gyral folding patterns may perform different functions (e.g. Figure 9). Right: Inflated cortical surface illustrating the precise anatomical locations of these ROIs. The distance on the gray matter between these two ROIs is 15mm rather than 5mm in volume space. Weiner & Grill-Spector, Psychological Research 2013.

Understanding maps on multi-slice format Visual field map Color indicates stimulus eccentricity most effective at driving a response

Maps are easier to understand on the cortical surface

Surface based visualization provides a better understanding of anatomical features than the volume OTS CoS FG lateral lateral medialFG posterior Weiner et al. NeuroImage, 2014

Surface based visualization provides a better understanding of anatomical features than the volume OTS CoS FG lateral lateral medialFG posterior Weiner et al. NeuroImage, 2014

Surface based visualization provides a better understanding of anatomical features than the volume OTS CoS FG lateral lateral medialFG posterior Weiner et al. NeuroImage, 2014

Surface based visualization provides a better understanding of anatomical features than the volume MFS OTS CoS (a) (b) medialFG lateral FG lateral posterior FG Figure 1. The mid-fusiform sulcus (MFS). Example right hemisphere from a ten-year old male. (a) Inflated cortical surface with sulci illustrated in dark gray. The MFS (outlined in red) is a longitudinal sulcus dividing the fusiform gyrus (FG) into lateral and medial partitions, flanked by the occipito-temporal sulcus (OTS) laterally and the collateral sulcus (CoS) medially (inset for location of zoomed portion). (b) The MFS, OTS, and CoS have a distinctive ω pattern on single coronal slices where the MFS is the shallower sulcus flanked by the much deeper CoS and OTS. Top: Example coronal slice from the position of the dotted line in (a). Bottom: Schematic of the ω pattern of the MFS, OTS, and CoS. MFS= Mid fusiform sulcus Weiner et al. NeuroImage, 2014

Understanding anatomical features may reveal functional-structural relationships MFS OTS CoS (a) (b) medialFG lateral FG lateral posterior FG 1cm Functional ROI CoS-places/PPA mFus-faces.FFA2 Weiner et al. NeuroImage, 2014

Surface based visualization enables comparing different maps on a common surface revealing relationships among maps and the cortical folding Multiple representations superimposed on VTC are aligned to the mid fusiform sulcus (MFS) Eccentricity bias Domain specificity Animacy Real world object size MFS MFS MFS MFS OTS CoS MFS anterior foveal peripheral faces places animate inanimate small big Weiner et al., 2014 Nasr et al., 2011 Haxby et al., 2011 Konkle et al., 2012 Grill-Spector & Weiner, Nature reviews neuroscience, 2014

Visualization techniques How to manual Iso-voxel Gray/white segmentation Surface boundaries (Marching cubes) Surface smoothing

If the original scan did not obtain isotropic voxels (0.9375,0.9375,1.2) mm If the original scan did not obtain isotropic voxels

Reslice data to obtain isotropic voxels (0.9,0.9,0.9) mm Reslice data to obtain isotropic voxels Note noise in the image

(0.9,0.9,0.9) mm Co-register and average multiple anatomical scans to reduce noise in anatomical image

Averaging makes segmentation of gray and white matter easier Better separation Gray White N voxels N voxels Intensity level Intensity level

Visualization techniques How to manual Iso-voxel Segment brain into gray and white matter Surface boundaries (Marching cubes) Surface smoothing

Gray-white matter segmentation is essential (b)

White and gray matter topology: Solids, handles WM CSF

White Matter Topology: Avoiding Cavities and Handles Euler number is a means of determining the presence of such imperfections Algorithms for locating and replacing exist (Kriegeskorte and Goebel) Cavity (hole) Handle http://www.scipress.org/journals/forma/pdf/1703/17030183.pdf Euler Number and Connectivity Indexes of a Three Dimensional Digital Picture Junichiro TORIWAKI1* and Tatsuhiro YONEKURA2 1Department of Information Engineering, Graduate School of Engineering, Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464-8603, Japan 2Department of Computer and Information Sciences, School of Engineering, Ibaraki University, 4-12-1, Nakanarusawa-cho, Hitachi, Ibaraki 316-8511, Japan *E-mail address: toriwaki@nuie.nagoya-u.ac.jp

Finding the First Gray Matter layer and determine the connectivity based on white matter CSF WM GM

Thickening the Gray Matter and Retaining Connectivity CSF WM GM

White, Gray, Connectivity Classification Gray matter (a) (c) Connectivity (b) (d) Gray nodes are connected if (a) they share a common parent, or (b) their parents are connected

The Importance of Being a Mesh From the white matter classification we create A surface mesh for visualization A gray node connection graph Gray nodes are properly connected No connections across sulci Visualization simplified Measurement of area and distance

Visualization techniques How to manual Iso-voxel Segment brain into gray and white matter Determine the surface boundaries (Marching cubes algorithm) Surface smoothing

Creating the surface mesh from the classified voxels Start by building a mesh at the boundary of the white matter;

Surface boundary identification “Marching squares” Black: Shape Green: Inside vertices Red: New vertices. Draw boundary through these. There are basically only a few lines we can draw for the boundaries. http://www.exaflop.org/docs/marchcubes/ind.html http://www.cs.unc.edu/~marc/tutorial/node130.html The first step is to calculate the corners that are inside the shape (represented by the green dots). We can now insert some vertices, since we know which points are inside and which are outside we can guess that a vertex should be positioned approximately halfway between an inside corner and any outside corners that are connected by the edge of a cell. The diagram shows the discussed vertices as small red dots

Surface boundary identification “Marching squares” Black: Shape Green: Inside vertices Red: New vertices. Draw boundary through these. There are basically only a few lines we can draw for the boundaries. http://www.exaflop.org/docs/marchcubes/ind.html http://www.cs.unc.edu/~marc/tutorial/node130.html The diagram shows the matching surface formed by joining the vertices with lines

Surface boundary identification Each line is like one of the above, except for a possible rotation. Very simple. The surface is an OK representation of the circle, but it is more jagged than we would like. Smoothing is required.

For surfaces, there are 15 basic shapes that replace each cube pattern: Marching cubes Unimportant differences Rotation by any degree over any of the 3 primary axis Mirroring the shape across any of the 3 primary axis Inverting the state of all corners and flipping the normals of the relating polygons. We are now of course dealing with cubes that have 8 corners and therefore a potential 256 possible combinations of corner status. However to simplify the algorithm we can reduce the complexity by taking into account cell combinations that duplicate under rotation, mirroring & inversion. Taking this into account we can resolve the original 256 combinations down to a total of 15 combinations (see image). The blue spheres denote corners that have tested as inside the shape and the green arrows denote the surface normals of the relevant triangles.

Visualization techniques How to manual Iso-voxel Segment brain into gray and white matter Determine the surface boundaries (Marching cubes algorithm) Smooth the resultant cortical surface

Surface boundary identification Each line is like one of the above, except for a possible rotation. Very simple. The surface is an OK representation of the circle, but it is more jagged than we would like. Smoothing is required.

Boundary smoothing Many ideas about smoothing. Illustrated here: Find the normals, smooth their orientation changes, and reconstruct the curve from normals

Smoothing Many ideas about smoothing. Illustrated here: Find the normals, smooth their orientation changes, and reconstruct the curve from normals

Viewing the Gray Matter Surface

Smoothing the mesh 3D Graphical Tools for further smoothing, without destroying the good topological properties of the surface, are available in ITK gray (visualization toolkit) and other open graphics toolkits

Inflating the cortical mesh to better visualize activations buried in sulci

Inflating the cortical mesh to better visualize activations buried in sulci Places > Faces, bodies, words, objects 3 T-value 8

Shading the surface: gyri and sulci Local curvature is inverse to the radius of the circle that best fits the local curve High curvature (+) Low curvature (+)

Despite extreme smoothing topology is unchanged