Mesh Segmentation Zhenyu Shu

References Gelfand N, Guibas L J. Shape segmentation using local slippage analysis [C]. Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing, Nice, France, Nice, France: ACM, 2004: Katz S, Leifman G, Tal A. Mesh segmentation using feature point and core extraction [J]. The Visual Computer. 2005, 21(8): Podolak J, Shilane P, Golovinskiy A, et al. A planar-reflective symmetry transform for 3D shapes [C]. ACM SIGGRAPH 2006 Papers, Boston, Massachusetts, Boston, Massachusetts: ACM, 2006: Reniers D, Telea A. Hierarchical part-type segmentation using voxel-based curve skeletons [J]. The Visual Computer. 2008, 24(6):

Mesh Segmentation Become a key ingredient in many geometric modeling and computer graphics tasks and applications –Parameterization –Texture mapping –Shape matching –Morphing –Multiresolution modeling –Mesh editing –Compression –Animation –And more

Mesh Segmentation Base Definition: Mesh segmentation : Let M be a 3D boundary-mesh, and S the set of mesh elements which is either V, E or F. A segmentation of M is the set of sub- meshes = {M 0, M 1...,M k−1 } induced by a partition of S into k disjoint sub-sets.

Mesh Segmentation

The key question in all mesh segmentation problem is how to partition the set S. And this relies heavily on the application. Mesh segmentation as an optimization problem: –Given a mesh M and the set of elements S ∈ {V, E, F}, find a disjoint partitioning of S into S 0,..., S k−1 such that the criterion function J = J(S 0,..., S k−1 ) be minimized (or maximized) under a set of constraints C.

Constraints Commonly used constraints: –Cardinality Bound on the maximum and/or minimum number of elements in each part S i to eliminate too small or too large partitions. Bound on the ratio between the maximum and minimum number of elements in all parts to create more balanced partition. Bound on the maximum or minimum number of segments used to balance the partition.

Constraints –Geometric Maximum/minimum area of sub-mesh. Maximum/minimum length of diameter or perimeter of sub-mesh. More complex constraints such as convexity of either 2D patch or volumetric 3D part. Soft constraints in the form of a bias towards specific shapes.

Constraints –Topological constraints Restriction of each S i to be topologically equivalent to a disk. Restriction of each S i to be a single connected component.

Mesh Attributes Commonly used attributes –Planarity of various forms. –Higher degree geometric proxies (spheres, cylinders, cones, quadrics developable surfaces). –Difference in normals of vertices or dihedral angles between faces. –Curvature. –Geodesic distances on the mesh. –Slippage. –Symmetry. –Convexity. –Medial axis and shape diameter. –Motion characteristics.

Shape segmentation using local slippage analysis Gelfand, Natasha Guibas, Leonidas J Computer Graphics Laboratory, Stanford University Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing

Main idea Decompose the mesh into some simple surfaces, such as spheres, planes, cylinders and surfaces of revolution, so- called kinematic surfaces.

Shape classification through slippable motions Slippable motion: Given a surface S, we call a rigid motion M a slippable motion of S if the velocity vector of each point x S is tangent to S at x. The surface under slippable motions can be thought of as sliding against itself, without forming any gaps between the moving surface and the original copy. That is, the surface S is invariant under its slippable motions.

Kinematic surfaces Surfaces which are invariant under at least one of the types of rigid motions are known as kinematic surfaces. –Rotational –Translational –Helical motions POTTMANN H., WALLNER J.: Computational Line Geometry. Springer Verlag, 2001.

Computing slippable motions x is a point belong to the surface S, n is the normal at x, r is a rotation vector around x,y,z axis, t is translation vector.

Computing slippable motions That is

Computing slippable motions Eigenvectors of C whose corresponding eigenvalues are 0 correspond to the slippable motions. In practice, due to noise C is likely to be full rank. In this case, the slippable motions are those eigenvectors of C whose eigenvalues are sufficiently small.

Kinematic surfaces

Segmentation into slippable components Goal: Discover a decomposition of P into P 1, P 2, …,P k such that each P i is large, connected and slippable. Algorithm: –Initialization: compute similarity score between each pair of adjacent patches. –Patch growing: at each step, select the most similar adjacent patches and collapse them into a single patch. –Termination: stop when the similarity score of all the pair of patches drop below a threshold.

Similarity score Two patches P i and P j belong to the same component if –Their corresponding covariance matrixes C i and C j have the same number of small eigenvalues –The corresponding slippage signatures are the same, that is we can express the slippable motions of P i as a combination of slippable motions of P j and vice versa. Let X 1... k and Y 1... k be the small eigenvalues of P i and P j respectively, we just need to test if each column X 1... k can be expressed as a linear combination of columns of Y 1... k.

Similarity Score is the (k+1)st singular value of the combined matrix [X 1…k Y 1…k ] F is a Gaussian centered around 0 and map small singular values into high similarity scores.

Result

Mesh segmentation using feature point and core extraction Katz, Sagi Leifman, George Tal, Ayellet Israel Institute of Technology The Visual Computer. 2005, 21(8):

Main idea Produce hierarchical segmentations into meaningful components and –Be invariant both to the pose of the model and to different proportions between the model’s components. –Produces correct hierarchical segmentations of meshes, both in the coarse levels of the hierarchy and in the fine levels. –The boundaries between the segments go along the natural seams of the models.

Algorithm overview Mesh coarsening: to accelerate the algorithm when executed on the large meshes and decrease the sensitivity of noise. Pose-invariant representation: Multi-dimensional scaling is used to transform the mesh S into a pose-invariant representation S MDS.Pose-invariant representation: Multi-dimensional scaling is used to transform the mesh S into a pose-invariant representation S MDS. Feature point detection: feature points are computed on S MDS, and mapped back to S.Feature point detection: feature points are computed on S MDS, and mapped back to S. Core component extraction: the core component is extracted. Mesh segmentation: compute the other segments( exclude the core component), each segment contain at least one feature point.Mesh segmentation: compute the other segments( exclude the core component), each segment contain at least one feature point. Mesh refinement: map the segmentation back to original, fine-resolution mesh.

Pose-invariant representation Multi-dimensional scaling: representing dissimilarities as distances in an m- dimensional Euclidean space. The more dissimilar two items are, the larger the distance between them in this space. Here, we define the dissimilarity between points on the mesh as the geodesic distance between them.

MDS Two major types –Metric MDS: preserve the intervals and the ratio between the dissimilarities. –Non-metric MDS: preserve only the order of the dissimilarities, rather than the exact intervals and ratios. Empirical studies shows that non-metric MDS have better results here.

MDS MDS attempt to minimize

MDS iteration Initialization: –Find initial configuration of the points in m- dimensional MDS space (random) (m=3). Iteration: –Compute Euclidean distances d ij between the points in the MDS space. –Using pool adjacent violators algorithm to find the optimal monotonic function f. –Each vertex is re-mapped to a point in the MDS space by minimizing F s. –These points are the input of next iteration. Barlow, R., Bartholomew, D., Bremner, J., Brunk, H.: Statistical Inference Under Order Restrictions. Wiley, New York (1972)

MDS example Return

Feature point detection Feature points: –Should reside on tips of prominent components of a given model Intuitively. –Should be invariant to the pose of the model.

Feature point detection Definition: vertex v is called feature point if –It is a local maximum of the sum of the geodesic distance functional, that is, –And it resides on the convex-hull of S MDS. Detection: Compute the convex hull of S MDS and then find vertices of convex hull satisfy equation. Return

Core extraction and segmentation The segmentation algorithm: –Spherical mirroring of S MDS. –Extraction of the core component of S. –Extraction of the other components of S.

Spherical mirroring Find a bounding sphere and mirror the vertices.

Core component extraction Compute the convex hull of the mirrored vertices. The vertices reside on the convex hull, along with the faces they define on S, are considered the initial core component.

Core component extraction If initial core does not separate all the feature points, the core component is extended. Core extension: Iteratively add the neighboring faces of the current core until –the current core separate all the feature points or –The distance from the core to the closest feature point is reduced by more than a constant factor (0.5)

Extraction of other segments Extract other segments from mesh by subtracting the core component. A connected component that contains at least one feature point is a segment of the mesh. Component does not contain any feature point joins the core component. Return

Results

A planar-reflective symmetry transform for 3D shapes Podolak, Joshua; Shilane, Philip; Golovinskiy, Aleksey; Rusinkiewicz, Szymon; Funkhouser, Thomas Princeton University ACM SIGGRAPH 2006

Main idea Decompose a mesh such that the faces with each segment have the same distinct symmetries.

Symmetry Symmetry is an important feature of almost all shapes. Perfect symmetries –Be unstable with added noise or missing data Imperfect symmetries –Define symmetry distance to measure imperfect symmetries

Symmetry distance Symmetry distance : –Distance between f and the nearest function that is invariant to that reflection f is a scalar-valued function defined over a d- dimensional space of points. is a plane reflection.

Planar Reflective Symmetry Transform is a mapping from to a scalar- value, which measures f ’s symmetry with respect to the plane Define if f is perfectly symmetric with respect to if f is perfectly anti-symmetric with respect to

Visualization of the PRST The darkness of each point represents the maximum of PRST values over all planes passing through.

Properties of PRST Dominant points and planes of symmetry match our human intuition of the “center” and “major axes” PRST is not sensitive to noise and varies continuously with deformations.

Computation of PRST KATZ, S., AND TAL, A Hierarchical mesh decomposition using fuzzy clustering and cuts. Proceedings of ACM SIGGRAPH 22, 3, 954–961. has observed that the nearest symmetric function to f is simply the average of f and.

Computation of PRST To apply PRST definition, convert the surfaces into a binary occupancy grid. Use Gaussian Euclidean Distance Transform to smooth for the existence of small noise and small feature

Discrete computation for surfaces Adapt Monte Carlo method to efficiently compute for sparsity. Original method: Monte Carlo method:

Weighting Consists of two terms: –Importance sampling perform, be the reciprocal of the probability of having selected x and x’ –Change-of-variables weight

Monte Carlo estimator Monte Carlo estimator is

Segmentation Find the significant local maxima of the PRST Compute, for each face and every symmetry plane, the degree to which the face contributes to the symmetry with respect to that plane If there are m local maxima in the PRST, then every point has m values representing its support for symmetry with respect to each of the m planes.

Segmentation Treat these m values as a feature vector and cluster faces according to their proximity in the m-dimensional feature space. For each split, perform k-means( k=2) clustering and take 2 largest connected components and find extra boundary between them using min-cut.

Segmentation Segmentation terminated condition –at a user-supplied depth –or when the only planes of local maxima reflect either more than 90% or less than 10% of the surface onto itself.

Result

Hierarchical part-type segmentation using voxel-based curve skeletons Reniers, Dennie; Telea, Alexandru Eindhoven University of Technology, The Netherlands The Visual Computer. 2008, 24(6):

Main ideas Use curve skeleton to segment a voxelized shape

Surface skeleton Given a 3D object, the surface skeleton is defined as: –a, b are called feature points of p –the surface skeleton consists of 2D manifolds is called sheets.

Curve skeleton When curve skeleton is not incident with a sheet- intersection curve, define curve skeleton C as: – is two shortest geodesics between two feature points of p. When curve skeleton is incident with a sheet-intersection curve, the definition above can not detect correct point.

Curve skeleton Define the combination of the shortest geodesics for a surface skeleton point p the shortest-geodesic set : Definition of curve skeleton: Junction point:

Computation of skeleton curve

Component sets divide the object surface into multiple components, called component sets

Flat segmentation

Hierarchical Segmentation Let F be the set of foreground components of all critical points combined. Consider all components of F in ascending order of area. For, –if f does not overlap any existing segments in S, add f directly. –Else, the potential component s is computed as the set difference between f and the existing segments in S. If the difference add at least 10% of the area it overlaps, add s to S.

Results

Advantages The borders exhibit minimal twist on the surface and look natural Using geodesics for segment borders yields stable and robust segments for very noisy shapes The segmentations respect the object’s circular symmetry and are invariant for different poses of the same object

Limitations Segment borders do not always tightly wrap around attached object parts. The reason is the associated junctions lie deep within the palm, so that the feature points and the ends of the geodesics are far from the attachment.

Thanks