CASA 2006 CASA 2006 A Skinning Approach for Dynamic Mesh Compression Khaled Mamou Titus Zaharia Françoise Prêteux

3D animation industry Context & Objectives  Applications  Virtual and augmented reality  Cartoons  Video games and CGI films

Dynamic 3D content Context & Objectives How to exchange, transmit and visualize such 3D content in a platform-independent manner ?!  Content creation  Motion capture  Skinning models  Physical-based simulation  …

Context & Objectives  Principle  Represent the animation sequence as a set of key-meshes Key-frames Animation Interpolation 3D animation industry: key-frame representations Dynamic 3D content  Apply interpolation procedures to generate the in-between frames at the desired framerate

Context & Objectives Constant topology Time-varying geometry  Sequence of meshes with:  Constant topology  Time-varying geometry Dynamic 3D content Key-frame representations: dynamic 3D meshes  Advantages  Generality  Interoperability  Content protection  Drawbacks  Huge amount of data  Need of compact representations

Objectives Context & Objectives  Compression efficiency Compactness of the coded representation  Progressive transmission Bitstream adaptation to different, fixed or mobile communication networks and terminal devices  Scalable rendering Bitstream adaptation for real-time rendering

Outline Experimental evaluation Conclusion et perspectives Previous work Skinning-based compression

Outline Experimental evaluation Conclusion et perspectives Previous work Skinning-based compression

Wavelets Vertex prediction State of the art MPEG-4/AFX-IC Dynapack AWC GV PCA-based LPCA CPCA PCA Clustering Dynamic 3D mesh compression RT D3DMC  Emerging field of research  Four families of approaches

Wavelets Vertex prediction State of the art MPEG-4/AFX-IC Dynapack  A more elaborated motion model: skinning AWC GV PCA-based LPCA CPCA PCA Clustering Dynamic 3D mesh compression Skinning-based compression RT D3DMC  New motion-based segmentation procedure  Temporal DCT-based compression of the residual errors  Principle: extension of the RT technique

Outline Experimental evaluation Conclusion et perspectives Previous work Skinning-based compression

General view Skinning-basedcompression Temporal DCT Affine motion and weights estimation Motion-based segmentation Static encoder Quantization and arithmetic encoding Prediction residuals Affine transforms Animation weights Partition (Mi)(Mi) M0M0 Compressed DCT coefficients Compressed M 0

Motion-based segmentation Skinning-basedcompression  Objective Partition the mesh vertices into clusters whose motion can be accurately described by a single affine motion

Motion-based segmentation Skinning-basedcompression  Principle  For each vertex v, select a neighborhood v*

Vector of homogeneous coordinates of vertex p at frame i Motion-based segmentation Skinning-basedcompression  For each frame i, compute an affine transform A i v … Frame 0Frame 1Frame (F-1)  Principle  For each vertex v, select a neighborhood v*  Store the ( A i v ) i of each vertex as a single vector α v

Motion-based segmentation Skinning-basedcompression  For each frame i, compute an affine transform A i v  Principle  For each vertex v, select a neighborhood v*  Store the ( A i v ) i of each vertex as a single vector α v Cow  Determine the partition π = ( π k ) k by applying the k-means clustering algorithm to the set ( α v ) v

Motion-based segmentation Skinning-basedcompression Dancer  For each frame i, compute an affine transform A i v  Principle  For each vertex v, select a neighborhood v*  Store the ( A i v ) i of each vertex as a single vector α v  Determine the partition π = ( π k ) k by applying the k-means clustering algorithm to the set ( α v ) v

General view Skinning-basedcompression Temporal DCT Affine motion and weights estimation Motion-based segmentation Static encoder Quantization and arithmetic encoding Prediction residuals Affine transforms Animation weights Partition (Mi)(Mi) M0M0 Compressed DCT coefficients Compressed M 0

Affine motion estimation Skinning-basedcompression  Principle  Model the motion of each cluster k at each frame i by an affine transform H i k  Predict the geometry of frame i from frame 0 by using the affine transforms ( H i k ) k

Affine motion estimation Skinning-basedcompression  Performances  Captures well the object motion Frame 0Frame 36Predicted frame 36 Error distribution 0% 4%  Induces discontinuities at the level of clusters boundaries We need a more elaborated motion model

Skinning model Skinning-basedcompression  Objective Derive a continuous motion field  Principle  Linearly combine the affine motion of adjacent clusters with appropriate weighting coefficients  Compute the animation weights by solving a least squares minimization problem

General view Skinning-basedcompression Temporal DCT Affine motion and weights estimation Motion-based segmentation Static encoder Quantization and arithmetic encoding Prediction residuals Affine transforms Animation weights Partition (Mi)(Mi) M0M0 Compressed DCT coefficients Compressed M 0

DCT-based compression of the residual errors Skinning-basedcompression  Objective  Compress the residual errors by exploiting the temporal correlations Prediction error at frame i and vertex v  Principle  For each vertex v, compute the spectra of its x, y and z errors  Concatenate the spectral coefficients of all vertices into a single vector S  Quantize and arithmetically encode S Well-adapted to progressive transmission

Outline Experimental evaluation Conclusion et perspectives Previous work Skinning-based compression

Evaluation corpus: Snake Experimentalresults 9179 vertices 134 frames

Evaluation corpus: Dancer Experimentalresults 7061 vertices 201 frames

Evaluation corpus: Humanoid Experimentalresults 7646 vertices 154 frames

Evaluation corpus: Chicken Experimentalresults 3030 vertices 400 frames

Objective evaluation: criteria  Compression rates: bits per frame per vertex (bpfv)  Distortion measures: RMSE [MESH tool, Aspert et al, 2002] D : length of the diagonal of the object’s bounding boxExperimentalresults

Compression results: Chicken Experimentalresults  Performances  D3DMC & skinning: best performances  Skinning: up to 47% gain over D3DMC in term of bitrates RMSE bpfv

Compression results: Snake Experimentalresults  Performances  PCA: worst performances (F>>V not verified)  Skinning: up to 45% gain over RT in term of bitrates RMSE bpfv

Compression results: Humanoid Experimentalresults  Performances  AFX-IC: poor performances: elementary predictor  Skinning: up to 67% gain over D3DMC in term of bitrates RMSE bpfv

Compression results: Dancer Experimentalresults RMSE bpfv  Performances  GV: re-meshing related problems  Skinning: up to 65% gain over GV in term of bitrates

Outline Experimental evaluation Conclusion et perspectives Previous work Skinning-based compression

Conclusion & perspectives Summary  A new skinning-based compression techniques for dynamic meshes  Specifically efficient for articulated dynamic meshes  Gains range from 47% to 67% in terms of bitrates over state-of-the-art encoders

Future work Conclusion & perspectives  Optimize the motion-based segmentation stage: How to determine automatically the number of clusters?  Multiple and dynamic skinning models: Temporal segmentation of the sequence  Compression of other attributes: normals, texture coordinates…

Thank you