2. rod-tv : surface reconstruction on demand by tensor voting page 2 Outline 1. Introduction and motivation 2. Rod-tv framework overview 3. Rod-tv modules 3a.Multi-scale data hierarchy 3b.Levels-of-detail control 3c.Visibility culling 3d.Tensor voting 4. Results 5. Conclusions and future work

3. Outline 1. Introduction and motivation 2. Rod-tv framework overview 3. Rod-tv modules 3a.Multi-scale data hierarchy 3b.Levels-of-detail 3c.Visibility culling 3d.Tensor voting 4. Results 5. Conclusions and future work

4. rod-tv : surface reconstruction on demand by tensor voting page 4 1. Introduction and motivation Problem statement Ờ Digital models are demanding nowadays in some industries, for instance, 3D movie industry, 3D game industry, etc. Ờ Models can be acquired by using 3D scanning devices > Bunny dataset – 35,948 points (scanned by Stanford University)

5. rod-tv : surface reconstruction on demand by tensor voting page 5 1. Introduction and motivation Problem statement Ờ Usually, digital models are represented as a set of 3D isolated points, i.e. (x i, y i, z i ) Ờ 3D scanned models are huge and noisy - Huge : real, detailed, accurate 3D models - Noisy : measurement errors in the scanning process Ờ Our concern: How to render these 3D models efficiently? - [Approach 1] point-rendering in computer graphics - [Approach 2] surface reconstruction in computer vision

6. rod-tv : surface reconstruction on demand by tensor voting page 6 1. Introduction and motivation Approach 1: Point-rendering Ờ A point-rendering is a graphical technique which uses simple geometrical shapes as rendering primitives Ờ Example - QSplat [Rusinklewicz and Levoy in 2000]

7. rod-tv : surface reconstruction on demand by tensor voting page 7 1. Introduction and motivation Point-rendering in QSplat bunny dataset - 35,948 points point-rendering – using sphere as a rendering primitive [high-resolution] [low-resolution] [close-up view]

8. rod-tv : surface reconstruction on demand by tensor voting page 8 1. Introduction and motivation Point-rendering in QSplat Ờ In QSplat, the input 3D points are organized as a hierarchy Ờ QSplat uses above hierarchy structure for view- dependent, levels-of-detail rendering > A spectrum of data resolution decreasing data resolution

9. rod-tv : surface reconstruction on demand by tensor voting page 9 1. Introduction and motivation Problems in point-rendering bunny [zoom-in] - 8,365 pts point-rendering [zoom-in] Ờ Problems - upper-bound by the scanning resolution - can handle noise-free 3D point input only

10. rod-tv : surface reconstruction on demand by tensor voting page 10 1. Introduction and motivation Summary of point-rendering Ờ Advantages - Fast and costless in rendering [Fast : hierarchy for view-dependent and levels-of-detail control] [Costless : simple geometrical shapes] Ờ Limitations - Upper-bound by scanning resolution - Noise-sensitive application system Ờ Comments - We need: - A flat and smooth surface in any data resolution - A noise-robust application system

11. rod-tv : surface reconstruction on demand by tensor voting page 11 1. Introduction and motivation Approach 2: Surface reconstruction Ờ A surface reconstruction is a vision technique which uses polygonal patches to represent a surface Ờ Example - Tensor voting [Medioni, Mi-Suen Li and Chi-Keung Tang in 2000]

12. rod-tv : surface reconstruction on demand by tensor voting page 12 1. introduction and motivation Tensor voting ~ Flat and smooth surface Ờ Tensor voting can generate a smooth and flat surface at any data scanning resolution

13. rod-tv : surface reconstruction on demand by tensor voting page 13 1. Introduction and motivation Tensor voting ~ Noise-robustness Ờ Tensor voting can handle noisy 3D point as an input

14. rod-tv : surface reconstruction on demand by tensor voting page 14 1. Introduction and motivation Problems in tensor voting Ờ Problems - surface reconstructing process is usually costly and can be regarded as two reasons: 1. Lack of levels-of-detail control - Surface is reconstructed in single data resolution only without consider the viewer position 2. Lack of visibility control - Input 3D Points are fully processed whether they are visible or not given a particular viewing volume

15. rod-tv : surface reconstruction on demand by tensor voting page 15 1. Introduction and motivation Summary of tensor voting Ờ Advantages - Flat and smooth surface is generated in any data resolution - Noise-insensitive reconstruction system Ờ Limitations - Lack of levels-of-detail control - Lack of visibility control Ờ Comments - We need : - View-dependent and levels-of-detail surface reconstruction system

16. rod-tv : surface reconstruction on demand by tensor voting page 16 1. Introduction and motivation Our contributions: Rod-tv Ờ Our approach surface reconstruction on demand by tensor voting - Abbreviate to rod-tv - A “graphics for visions” approach Ờ General idea - Defer vision surface reconstruction [tensor voting] - Use graphical techniques to query a better visible subset to meet the “on demand” requirement [levels-of-detail control and visibility culling]

17. Outline 1. Introduction and motivation 2. Rod-tv framework overview 3. Rod-tv modules 3aMulti-scale dataset 3bLevels-of-detail 3cVisibility culling 3dTensor voting 4. Results 5. Conclusions and future work

18. rod-tv : surface reconstruction on demand by tensor voting page 18 2. Rod-tv framework overview input: - 3D isolated points 1. generate a spectrum of data resolution 2. desire an appropriate data resolution at given particular viewing position 3. cull all the invisible parts 4. reconstruct surface on visible subset output: on demand surface

19. rod-tv : surface reconstruction on demand by tensor voting page 19 Outline 1.Introduction and motivation 2.Rod-tv framework overview 3. Rod-tv modules - 3a. Multi-scale data hierarchy - 3b. Levels-of-detail control - 3c. Visibility culling - 3d. Tensor voting 4.Results 5.Conclusions and future work

20. rod-tv : surface reconstruction on demand by tensor voting page 20 3. Rod-tv modules Module 1: Multi-scale data hierarchy Ờ Module overview - Input: a 3D dataset in single-resolution - Ambition : use a single data structure to generate a spectrum of dataset resolution

21. rod-tv : surface reconstruction on demand by tensor voting page 21 3. Rod-tv modules Module 1: Multi-scale data hierarchy Ờ Module description - Make use of hierarchical spatial data structure ~ Grid Pyramid ~ Octree - Usually it is organized as tree - Resolutions are encoded inside tree internal nodes - Different height generated different resolutions Ờ Module implementation - Component 1: Grid Pyramid - Component 2: Octree [both of them are spatial volume hierarchies]

22. rod-tv : surface reconstruction on demand by tensor voting page 22 3. Rod-tv module 1 – multi-scale data hierarchy Component 1: Grid Pyramid Ờ A Grid Pyramid is: - A hierarchy of volume [3D array] - Volume resolution decrease from the bottom of the pyramid to the top A. lo-resolution – the entire scene is represented by 1 grid cell B. mi-resolution – the entire scene is represented by 8 grid cells C. hi-resolution – the entire scene is represented by 64 grid cells D. the corresponding tree representation of the Grid Pyramid

23. rod-tv : surface reconstruction on demand by tensor voting page 23 3. rod-tv modules – multi-scale dataset component 1: grid pyramid Top-down approach the construction proceeds downwards by iteratively subdividing current cells into 8 equal cells in 3-D case and 4 equal cells in 2-D case Bottom-up approach the construction proceeds upwards by iteratively consolidating a group of 8 cells in 3-D case and a group of 4 cells in 2-D case A. input 3D points (12 pts) B. Grid Pyramid – level 0 cell size = 1 number of points = 11 C. Grid Pyramid – level 1 cell size = 2 number of points = 8 D. Grid Pyramid – level 2 cell size = 4 number of points = 3 E. Grid Pyramid – level 3 cell size = 8 number of points = 1 F. the corresponding tree representation of the Grid Pyramid

24. rod-tv : surface reconstruction on demand by tensor voting page 24 3. Rod-tv module 1 – multi-scale data hierarchy Component 1: Grid Pyramid Ờ Advantages - Multi-scale representation capability - Grid cell elements can be accessed in constant time Ờ Limitations - Memory is wasted on tessellating a large group of empty grid cells - Total number of resolutions are limited

25. rod-tv : surface reconstruction on demand by tensor voting page 25 Ờ An Octree is: - Subset of Grid Pyramid - Adaptive cell tessellation ~ large octant is used to represent low density region ~ small octant is used to represent high density region 3. Rod-tv module 1 – multi-scale data hierarchy Component 2: Octree A. Octree example in 3-D space B. the corresponding tree representation of the Grid Pyramid

26. rod-tv : surface reconstruction on demand by tensor voting page 26 Octree building rules Subdivision stops if 1) number of points in each cell less than certain threshold 2) tree height is higher than certain threshold A. input 3D points (12 pts) B. Octree – level 3 number of cells = 1 number of points = 1 C. Octree – level 2 number of cells = 4 number of points = 3 D. Octree – level 1 number of cells = 13 number of points = 8 E. Octree – level 0 Number of cells = 19 number of points = 11 F. the corresponding tree representation of the Grid Pyramid

27. rod-tv : surface reconstruction on demand by tensor voting page 27 3. Rod-tv module 1 – multi-scale data hierarchy Component 2: Octree Ờ Advantages - Memory is saved using adaptive cell tessellation Ờ Limitations - Points are too close: unlimited subdivision - Octree is not balanced

28. rod-tv : surface reconstruction on demand by tensor voting page 28 3. Rod-tv modules Summary of multi-scale data hierarchy Ờ Comments - Data structure alternative tips: ~ Large memory: Grid Pyramid ~ Limited memory : Octree - Multi-scale dataset to: ~ Facilitate levels-of-detail [module 2] ~ Facilitate visibility culling [module 3]

29. rod-tv : surface reconstruction on demand by tensor voting page 29 Outline 1.Introduction and motivation 2.Rod-tv framework overview 3. Rod-tv modules - 3a. Multi-scale dataset - 3b. Levels-of-detail control - 3c. Visibility culling - 3d. Tensor voting 4.Results 5.Conclusions and future work

30. rod-tv : surface reconstruction on demand by tensor voting page 30 3. Rod-tv modules Module 2: Levels-of-detail (on demand) Ờ Module overview - Input : a multi-scale dataset and a viewing position - Ambition: select an appropriate resolution dataset line-of-sight distance Closer to the viewer [need more points to represent] Farther to the viewer [need less points to represent]

31. rod-tv : surface reconstruction on demand by tensor voting page 31 3. Rod-tv modules description Module 2: Levels-of-detail (on demand) Ờ Module description - Make use of the hierarchy we build in module 1 - Traverse the hierarchy from coarse-to-fine resolution - In other words, we traverse the tree from root-to-leaf order - Then, selection criteria is made on each internal node and finally appropriate data resolution is returned ~ selection criteria can be made either in object-space or image-space Ờ Module implementation - Component 1: Range based [object-space] - Component 2: Screen based [image-space]

32. rod-tv : surface reconstruction on demand by tensor voting page 32 3. Rod-tv module 2 – levels-of-detail (on demand) Component 1: range-based method Ờ Levels-of-detail selection criteria: - Separate the 3-D space into different ranges {d 1,d 2,d 3 } - Selection is made based on the viewer-object distance d2d2 d3d3 line-of-sight distance d1d1 Decision rule If distance < d1 then LOD 0 is returned elseif d1 < distance < d2 then LOD 1 is returned elseif d2 < distance < d3 then LOD 2 is returned else LOD 3 is returned endif [LOD 0][LOD 1][LOD 2][LOD 3]

33. rod-tv : surface reconstruction on demand by tensor voting page 33 step 1. define ranges d1, d2, d3 and d4 step 2. compute the viewer-object distance step 3. start traverse from the root level step 4. check whether the criteria is met or not If yes, then stop Else, then continue traversing next lower level and repeat step 4 d1d1 d2d2 d3d3 d4d4 viewer-object distance line-of-sight distance viewer object d1d1 d3d3 d4d4 LOD 3 LOD 2 LOD 1 LOD 0 LOD 1LOD 2LOD 3 d2d2

34. rod-tv : surface reconstruction on demand by tensor voting page 34 step 1. define ranges d1, d2, d3 and d4 step 2. compute the viewer-object distance step 3. start traverse from the root level step 4. check whether the criteria is met or not If yes, then stop Else, then continue traversing next lower level and repeat step 4 d1d1 d2d2 d3d3 d4d4 viewer-object distance line-of-sight distance viewer object d1d1 d3d3 d4d4 LOD 3 LOD 2 LOD 1 LOD 0 LOD 1LOD 2LOD 3 d2d2

35. rod-tv : surface reconstruction on demand by tensor voting page 35 3. Rod-tv module 2 – levels-of-detail (on demand) Component 2: screen-based method Ờ Levels-of-detail selection criteria: - Project the cell on the image plane - Use sphere instead of octant/cube to avoid alignment - Selection is made based on the number of pixel covered on the image plane

36. rod-tv : surface reconstruction on demand by tensor voting page 36 step 1. Define a threshold t for pixel coverage step 2. Start traversing from the root of the tree step 3. Computer the cell projection s step 4. check whether the criteria is met or not If s < t then stop Else, then continue traversing next lower level and repeat step 3 line-of-sight distance viewer object LOD 3 LOD 2 LOD 1 LOD 0 LOD 1LOD 2LOD 3 image-plane 1 4 12 24

37. rod-tv : surface reconstruction on demand by tensor voting page 37 3. Rod-tv modules Summary of levels-of-detail (on demand) Ờ Advantages - Range-based method : easy to implement - Screen-based method: a single threshold is used Ờ Limitations - Range-based method: hard to set a range-thresholds Ờ Comments - Decision is made on-the-fly (demand) - An appropriate resolution dataset is generated after running this module, therefore “on demand” processing is achieved

38. rod-tv : surface reconstruction on demand by tensor voting page 38 Outline 1.Introduction and motivation 2.Rod-tv framework overview 3. Rod-tv modules - 3a. Multi-scale dataset - 3b. Levels-of-detail control - 3c. Visibility culling - 3d. Tensor voting 4.Results 5.Conclusions and future work

39. rod-tv : surface reconstruction on demand by tensor voting page 39 3. Rod-tv modules Module 3: Visibility culling Ờ Module overview - Input: an appropriate resolution dataset - Ambition: cull all the invisible parts Ờ Module description - Cull all the tokens which are outside viewing frustum - Cull all the tokens which are occluded by the frontiers Ờ Module implementation - Component 1: View frustum culling - Component 2: Occlusion culling

40. rod-tv : surface reconstruction on demand by tensor voting page 40 3. Rod-tv module 3 – visibility culling Component 1: View frustum culling - Find out a visible subset that is inside the frustum - A token-frustum intersection test - Usually, a frustum consists of six clipping planes, namely, left, right, top, bottom, near and far.

41. rod-tv : surface reconstruction on demand by tensor voting page 41 3. Rod-tv module 3 – visibility culling Component 1: View frustum culling - A token is inside the view frustum if d >= 0 in all clipping planes - Plane equation: - Token position: - Token-plane intersection test : D > 0 – in front of the plane D = 0 – on the plane D < 0 – behind the plane

42. rod-tv : surface reconstruction on demand by tensor voting page 42 - Find out a better visible subset that is inside the viewing frustum but not visible in the final result - In other words, we cull all the tokens which are occluded by the frontier 3. Rod-tv module 3 – visibility culling Component 2: Occlusion culling

43. rod-tv : surface reconstruction on demand by tensor voting page 43 3. Rod-tv module 3 – visibility culling Component 2: Occlusion culling - Project token on the image plane and decision is made based on the projection and the depth information - Make use of occlusion map (o-map) and depth map (d-map) - o-map: potential occluder are recorded and corresponding covering pixels are marked - d-map: record projected token depth information image plane occlusion map (o-map) depth map (d-map)

44. rod-tv : surface reconstruction on demand by tensor voting page 44 Depth test If the token projection overlaps in the o-map, then we compare the token depth in the d-map. We keep the one which is closer to the viewer and cull the other one.

45. rod-tv : surface reconstruction on demand by tensor voting page 45 3. Rod-tv module 3 Summary of visibility culling Ờ Advantages - No CPU time is wasted on invisible parts: ~ Outside the viewing volume ~ Occluded by the frontier ~ Less than certain pixel on the image plane Ờ Limitations - Hard to produce extract occluder set Ờ Comments - A better visible subset dataset is produced after running this module, therefore “on demand” processing, once again, is achieved

46. rod-tv : surface reconstruction on demand by tensor voting page 46 Outline 1.Introduction and motivation 2.Rod-tv framework overview 3. Rod-tv modules - 3a. Multi-scale dataset - 3b. Levels-of-detail control - 3c. Visibility culling - 3d. Tensor voting 4.Results 5.Conclusions and future work

47. rod-tv : surface reconstruction on demand by tensor voting page 47 3. Rod-tv modules Module 4: Tensor voting Ờ Module overview ~ [module 2] range/screen levels-of-detail selection ~ [module 3] view frustum and occlusion culling - Input: a “on demand” visible point set - Ambition: reconstruct “on demand” surfaces

48. rod-tv : surface reconstruction on demand by tensor voting page 48 3. Rod-tv modules Module 4: Tensor voting Ờ Module description: - Make use of the original tensor voting algorithm - use tensor as data representation - use voting as data communication - consists of four stages: ~ stage 1: information encoding ~ stage 2: sparse tensor voting ~ stage 3: dense tensor voting ~ stage 4: feature extraction

49. rod-tv : surface reconstruction on demand by tensor voting page 49 3. Rod-tv module 4 – tensor voting Data representation - Tensor voting uses a 2 nd order symmetric tensor as a data representation - Symmetric tensor can be visualized as an ellipsoid - Mathematically, we can use eigensystem to represent the ellipsoid

50. rod-tv : surface reconstruction on demand by tensor voting page 50 3. Rod-tv module 4 – tensor voting Data representation - After reorganizing the eigensystem, the ellipsoid can be decomposed into stick, plate and ball tensor [stick tensor] [plate tensor] [ball tensor]

51. rod-tv : surface reconstruction on demand by tensor voting page 51 3. Rod-tv module 4 – tensor voting Data communication - Tensor voting uses voting algorithm as a data communication - Voter site propagates data to its neighborhood by using predefined voting kernel - Votes then are then collected and accumulated in the votee site voter site p voter site r votee site q

52. rod-tv : surface reconstruction on demand by tensor voting page 52 3. rod-tv module 4 – tensor voting tensor voting system

53. rod-tv : surface reconstruction on demand by tensor voting page 53 3. Rod-tv module 4 – tensor voting Stage 1: information encoding

54. rod-tv : surface reconstruction on demand by tensor voting page 54 3. Rod-tv module 4 – tensor voting Stage 2: sparse tensor voting

55. rod-tv : surface reconstruction on demand by tensor voting page 55 3. Rod-tv module 4 – tensor voting Stage 3: dense tensor voting

56. rod-tv : surface reconstruction on demand by tensor voting page 56 3. Rod-tv module 4 – tensor voting Stage 4: feature extraction

57. rod-tv : surface reconstruction on demand by tensor voting page 57 Outline 1.Introduction and motivation 2.Rod-tv framework overview 3.Rod-tv modules - 3a. Multi-scale dataset - 3b. Levels-of-detail control - 3c. Visibility culling - 3d. Tensor voting 4. Results 5.Conclusions and future work

58. rod-tv : surface reconstruction on demand by tensor voting page 58 4. Results Surface reconstruction (on demand) by tensor voting

59. rod-tv : surface reconstruction on demand by tensor voting page 59 4. Results Surface reconstruction (on demand) by tensor voting

60. rod-tv : surface reconstruction on demand by tensor voting page 60 4. Results Surface reconstruction (on demand) by tensor voting

61. rod-tv : surface reconstruction on demand by tensor voting page 61 4. Results Surface reconstruction (on demand) by tensor voting

62. rod-tv : surface reconstruction on demand by tensor voting page 62 4. Results Surface reconstruction (on demand) by tensor voting

63. rod-tv : surface reconstruction on demand by tensor voting page 63 4. Results Surface reconstruction (on demand) by tensor voting

64. rod-tv : surface reconstruction on demand by tensor voting page 64 4. Results Surface reconstruction (on demand) by tensor voting

65. rod-tv : surface reconstruction on demand by tensor voting page 65 5. Conclusions and future work Conclusions Ờ Introduce levels-of-detail surface reconstruction for dense and noise 3D points - [module 1] exploit the Grid Pyramid or Octree data structure for multi-scale data representation - [module 2] use range-based or screen-based method for levels-of- detail “on demand” selection - [module 3] apply culling algorithm for finding better “on demand” visible subset - [module 4] reconstruct surface by means of tensor voting Ờ Implementation Issue - nearly real time and “on demand” surface can be generated on- the-fly (Tensor voting is the bottleneck)

66. rod-tv : surface reconstruction on demand by tensor voting page 66 5. Conclusions and future work Future work Ờ System optimization - Optimized neighborhood searching for tensorial support - Octree parallel data structure - Compact hierarchical data structure, such as wavelet - Sub-pixel technique in both occlusion map and depth map - Viewer caching for dynamic update surface

67. rod-tv : surface reconstruction on demand by tensor voting page 67 Question and Answer