1. 3DSkeleton-based Human Modeling with Metaballs 18 April 2008 Donghun Kim Robot Vision Lab

2. Contents Motivation Motivation What is the Metaball? What is the Metaball? How can we visualize Metaballs? How can we visualize Metaballs? 3D Skeleton Human Model 3D Skeleton Human Model Human model using Metaballs Human model using Metaballs Examples Examples Future Works Future Works

3. Motivation Motion Analysis of Non-rigid Object Motion Analysis of Non-rigid Object Motion Analysis of Articulated Object Motion Analysis of Articulated Object Model based Pose Estimation and Tracking Model based Pose Estimation and Tracking 3D Mesh Data by Visual Hall 3D Mesh Data by Visual Hall 3D Skeleton( Curve Skeleton by Thinning, DT, Geometric or General Fields) - on going 3D Skeleton( Curve Skeleton by Thinning, DT, Geometric or General Fields) - on going 3D Skeleton-based Motion tracking 3D Skeleton-based Motion tracking Metaball Human Model for shape recovery Metaball Human Model for shape recovery Motion Analysis by Skeleton-based Motion and Shape Info. (i.e. Manifold learning) Motion Analysis by Skeleton-based Motion and Shape Info. (i.e. Manifold learning)

4. Implicit Surfaces Implicit Surfaces: Surfaces which are contours(isosurface) through some scalar field in 3D Implicit Surfaces: Surfaces which are contours(isosurface) through some scalar field in 3D Different Field function: Metaball, Soft Objects, Blobbies Different Field function: Metaball, Soft Objects, Blobbies

5. Consider a function f(x,y,z) which define a scalar field in 3D space. Consider a function f(x,y,z) which define a scalar field in 3D space. Isosurface S is set of points for which Isosurface S is set of points for which f(x,y,z) = const. It can be thought as an Implicit function relating x,y and z -> so called implicit surface sometimes It can be thought as an Implicit function relating x,y and z -> so called implicit surface sometimes What are isosurfaces?

6. A particularly interesting case A particularly interesting case Use implicit equation of the form Use implicit equation of the form Gradient can be computed directly Gradient can be computed directly Soft/blobby objects that blend into each other Soft/blobby objects that blend into each other Metaballs x: a point in 3d space N: num. of metaballs pi : center position of metaballs ri : Metaball’s own density field

7. Field Functions Blobby Molecules Blobby Molecules Metaballs Metaballs Soft Objects Soft Objects r is distance of a point in space to a particular control point

8. Comparison of Field Functions

9. Examples

10. Metaballs are cool! Metaballs are cool!

11. Marching Cubes Well-known Method for scalr field polygonizataion Well-known Method for scalr field polygonizataion Sample f(x,y,z) on a cubic lattice Sample f(x,y,z) on a cubic lattice For each cubic cell For each cubic cell Estimate where isosurface inetersects cell edges by linear interpolation Estimate where isosurface inetersects cell edges by linear interpolation Tessellate depending on values of f() at cell vertices Tessellate depending on values of f() at cell vertices

12. Marching Cubes Algorithm for creating a polygonal surface representation of an isosurface of a 3D scalar field Algorithm for creating a polygonal surface representation of an isosurface of a 3D scalar field Combine simplicity with high speed because of using lookup tables Combine simplicity with high speed because of using lookup tables Application Application Reconstruction of a surface from volumetric datasets Reconstruction of a surface from volumetric datasets Creating a 3D contour of a mathematical scalar field Creating a 3D contour of a mathematical scalar field

13. Marching Cubes Each vertex can be either “inside” or “outside” Each vertex can be either “inside” or “outside” For each cube cell there are 256 ways for isosurface to intersect it can be simplified down to 15 unique For each cube cell there are 256 ways for isosurface to intersect it can be simplified down to 15 unique

14. Marching Cubes Example Example cubeindex = 0; if (grid.val[0] < isolevel) cubeindex |= 1; if (grid.val[1] < isolevel) cubeindex |= 2; if (grid.val[2] < isolevel) cubeindex |= 4; if (grid.val[3] < isolevel) cubeindex |= 8; if (grid.val[4] < isolevel) cubeindex |= 16; if (grid.val[5] < isolevel) cubeindex |= 32; if (grid.val[6] < isolevel) cubeindex |= 64; if (grid.val[7] < isolevel) cubeindex |= 128; P = P1 + (isovalue - V1) (P2 - P1) / (V2 - V1)

15. Marching Cubes Sampling Grid Resolution Sampling Grid Resolution Smoothness and Processing time to display Smoothness and Processing time to display

16. Examples of Marching Cubes More Information about Marching Cubes is in [3].

17. 3D Skeleton Human Model Structure( 15 nodes, 14 metaballs) Structure( 15 nodes, 14 metaballs) 0 1 23 45 7 6 8 910 11 12 1314 15

18. Human Model using Metaballs Head(1-2-7) Head(1-2-7)

19. Human Model using Metaballs Chest(0-1-2-3-6) Chest(0-1-2-3-6)

20. Human Model using Metaballs Left Arm(1-2-3-8-12) Left Arm(1-2-3-8-12)

21. Human Model using Metaballs Right Arm(1-2-5-11-15) Right Arm(1-2-5-11-15)

22. Human Model using Metaballs Left Hip & Thigh(0-4-9) Left Hip & Thigh(0-4-9)

23. Human Model using Metaballs Right Hip & Thigh(0-5-10) Right Hip & Thigh(0-5-10)

24. Human Model using Metaballs Left Leg(4-9-13) Left Leg(4-9-13)

25. Human Model using Metaballs Right leg(5-10-14) Right leg(5-10-14)

26. Human Model using Metaballs Result (Polygon Surface) Result (Polygon Surface)

27. Human Model using Metaballs Result (Filled surface) Result (Filled surface)

28. Programming 3D cloud data and mesh data from Visual Hall(using EPVH lib) as the input of 3D skeleton 3D cloud data and mesh data from Visual Hall(using EPVH lib) as the input of 3D skeleton 3D Human Model Tool based on Jonney’s 3D Human Model Frame (using Ellipsoid, Cylinder, Sphere) 3D Human Model Tool based on Jonney’s 3D Human Model Frame (using Ellipsoid, Cylinder, Sphere) OpenGL and FLTK GUI based tool OpenGL and FLTK GUI based tool Functions Functions 3D reconstruction by Visual hall 3D reconstruction by Visual hall Obtain Multiple Camera-view point images in OpenGL Environment Obtain Multiple Camera-view point images in OpenGL Environment Calculate Camera matrix by transforming the OpenGL representation to Physical camera representation Calculate Camera matrix by transforming the OpenGL representation to Physical camera representation Parameterized 3D Skeleton Human Model Parameterized 3D Skeleton Human Model 3D Skeleton-based Metaball Model 3D Skeleton-based Metaball Model Simple model size option (Height considering body ratio, fat-thin option) Simple model size option (Height considering body ratio, fat-thin option) Image Saving with OpenCV in OpenGL Image Saving with OpenCV in OpenGL

29. Modeling Tool Made by Dave Kim and Thanks to Jonney

30. Modeling Tool Made by Dave Kim and Thanks to Jonney

31. Data Comparison Multi-View Silhouette Images of Different Human Model Multi-View Silhouette Images of Different Human Model Ellipsoid, Cylinder and Sphere vs. Metaballs Ellipsoid, Cylinder and Sphere vs. Metaballs 3D cloud points and triangle meshes by Visual hall 3D cloud points and triangle meshes by Visual hall

32. Example (I)

34. Example (II) Metaball Human Model Metaball Human Model

35. Example (II) Result by Visual Hall Result by Visual Hall

36. Future Work 3D Curve skeleton Algorithm[2] 3D Curve skeleton Algorithm[2] DT (fast) + General Field (robust) DT (fast) + General Field (robust) 3D skeleton human model fitting to Curve Skeleton -> Pose Estimation 3D skeleton human model fitting to Curve Skeleton -> Pose Estimation Shape recovery using the information from the skeletonization Shape recovery using the information from the skeletonization Motion Analysis Motion Analysis

37. Reference  Clement Menier, Bruno Raffin, “3D Skeleton-based Pose Recovery,” The 3 rd international Symposimum on 3D Data Processing, Visualization and Transmission, 2006  Nicu D. Cormea etc at al., “Curve-Skeleton Application,” IEEE Visualization 2005  T. S. Newman, H. Yi, “A survey of the marching cubes algorithm,” Computers and Graphics, 2006  Implicit Surfaces written by Paul Bourke, June 1997, http://local.wasp.uwa.edu.au/~pbourke/modelling_rendering/implic itsurf/  Polygonising A Scalar Field written by Paul Bourke, May 1994, http://local.wasp.uwa.edu.au/~pbourke/geometry/polygonise