1. Utilization of 3D Scene Data for Improving Segmentation and Tracking
Yingdong Ma
Supervisors: Dr. Stewart Worrall and Prof. Ahmet Kondoz

2. Outline
Introduction
Surface Reconstruction from Unorganised Sample Points
3D Object Segmentation
Depth Assisted Video Object Segmentation
Depth Assisted Video Object Tracking
Conclusion and Future Works
List of Publications

3. Introduction Motivation:
Automatic video object detection and tracking is a hard work for current mono-view video processing systems
The ability of handling new input, such as the 3D scene data

4. Introduction
Reconstruction of 3D scene further benefits video content detection
Possible application: NASA’s Mars pathfinder mission
Decomposition of 3D object model reveals structual informaiton
(Image from NASA Mars pathfinder web page) 4

5. Introduction
Aim: development of techniques for better utilization of 3D scene data in semantic object segmentation and tracking
Objectives:
3D object surface reconstruction
3D object segmentation
Depth assisted video object segmentation
Depth assisted multiple objects tracking

6. Outline
Introduction
Surface Reconstruction from Unorganised Sample Points
3D Object Segmentation
Depth Assisted Video Object Segmentation
Depth Assisted Video Object Tracking
Conclusion and Future Works
List of Publications

7. 3D Object Surface Reconstruction
Given a point cloud on or near the object surface, recovers the geometric shape from the point cloud and represent it by means of a polygon mesh
Challenges:
The sample point might be very large, noisy, arbitrary shape
Local under sampling, where point samples are not dense enough to capture local features

8. 3D Object Surface Reconstruction
Typical reconstruction approaches overview
Functional approaches: a implicit surface is defined as the zero-set of a scalar function, such as weighted quadrics.
Voronoi/Delaunay filtering: the surface mesh is obtained by extracting triangles from a Delaunay triangulation/Voronoi diagram
Region growing: starts from an initial triangle and iterates to attach new triangles to the region’s boundaries.

9. 3D Object Surface Reconstruction
Delaunay/Voronoi-based methods
Provide geometric structure information for unorganized sample points
Multiple Delaunay/Voronoi computation
Extra points introduced (poles)
Region growing methods
Computationally efficient
Reconstruction quality depends on user defined parameters

10. 3D Object Surface Reconstruction
Delaunay-based region growing method
Build Delaunay triangulation
Smallest top triangle
Select candidate triangle for each boundary edge
2D example

11. 3D Object Surface Reconstruction
Selection of candidate triangle
The confident order of a candidate triangle is defined based on the angular distance and the geometric distance

12. 3D Object Surface Reconstruction
Experimental results (number of triangles selected so far)
10040f
200f
1000f
5000f
200f
1000f
5000f
14785f

13. 3D Object Surface Reconstruction
Performance analysis
Efficient (compute Delaunay triangulation once)
The code is written in C on a laptop computer (1400 MHz, 512Mb memory)
Small holes filling by greedy triangulation
Robust in the case of sharp boundary if the point cloud is dense enough
Model
Sample Points
Triangles
Running time (s)
Powercrust (s)
CAD model
3799
7396
6.05
7.18
Hand
5029
10040
6.63
8.04
Head
7399
14785
9.26
11.63
Duck
7862
19647
17.97
18.21
Rock-arm
10044
20102
35.82
25.39

14. Outline
Introduction
Surface Reconstruction from Unorganised Sample Points
3D Object Segmentation
Depth Assisted Video Object Segmentation
Depth Assisted Video Object Tracking
Conclusion and Future Works
List of Publications

15. 3D Object Segmentation
Given a 3D object, which is represented by a set of sample points or a surface mesh, the 3D object segmentation problem refers to partition the object into meaningful parts based on geometric properties of the sample points or faces that comprise the object
Properly decomposing objects into meaningful components recovers useful structural properties of a 3D object
3D object segmentation is an ill-posed problem: segmentation results highly depends on the application context

16. 3D Object Segmentation Approaches overview
The fuzzy k-means clustering method computes the probability that a point/face belongs to a patch based on a distance measurement
Number of components
Hierarchical binary decomposition method
Skeleton-based method represents objects by a lower dimensional construction
Skeleton generation: principal axis, edge contraction
Skeleton segmentation or sweep skeleton branches

17. 3D Object Segmentation
Aim: given a 3D object (point cloud), automatically divide the object into several meaningful components
System overview
Using a set of local maxima and a fuzzy cluster validity index to find the number of components
Decompose object by means of fuzzy k-means
Find the cut line between components by Max flow/Min cut

18. 3D Object Segmentation Find local maxima
The distance measurement: the shortest path between two vertices
The root vertex is the one which has the longest geodesic distance to other vertices
A vertex is labelled as local maximum if its neighbours closer to the root vertex
Fuzzy cluster validity index (the separation index)
Order local maximum based on their geodesic distance
The smallest S(f) indicates the optimal number of components
Separation index S(f)
Local maxima

19. 3D Object Segmentation Object segmentation by the fuzzy k-means method
Minimization of the object function
Find the cut line between components
Give large weight to an edge if the angle between its two end points normal is negative (concave edge)

20. 3D Object Segmentation Experimental results Separation index S(f)
Local maxima

21. 3D Object Segmentation Performance evaluation
automatic calculation of the optimal number of meaningful components
The MDS (multi-dimensional scaling) transform method
Unfold object model and construct the convex hull of transformed model
Critical points: Local maxima on the convex hull

22. Outline
Introduction
Surface Reconstruction from Unorganised Sample Points
3D Object Segmentation
Depth Assisted Video Object Segmentation
Depth Assisted Video Object Tracking
Conclusion and Future Works
List of Publications

23. Depth Assisted Object Segmentation
A process that involves partitioning a video scene into semantic meaningful components in a generic video sequence
Most vision-based systems involving video object tracking and moving objects recognition require fast and reliable detection of foreground regions
Performance of the video object segmentation techniques can be effected by reasons such as shadows, illumination change, cluttered background, and background movement

24. Depth Assisted Object Segmentation
Object segmentation approaches overview
Spatial segmentation
Efficient in the case of simple scene but tends to get over-segmentation under cluttered background
Post-processing is required such as small regions merging
Motion-based segmentation
Frame difference method is fast but sensitive to shadows and background movement

25. Depth Assisted Object Segmentation
Joint spatial-temporal segmentation takes into account motion information and various spatial features including colour, texture and edge
Depth-based segmentation
Robust in complex backgrounds but sensitive to object texture and the distance between objects and the camera

26. Depth Assisted Object Segmentation
Mono-view videos lost some important information of the scene, such as the depth information
Develop an automatic segmentation method to extract meaningful video objects by combining depth and other spatial-temporal features
System overview

27. Depth Assisted Object Segmentation
Object mask generation
Depth map segmentation
u/v-projection images
Motion segmentation
Frame difference images
Object boundary refinement by active contour model
Object masks provide the initial contour
The intensity and edge are the external energies to be minimized in active contour model

28. Depth Assisted Object Segmentation
Experimental results
Depth map, motion-based segmentation, without boundary refinement
Depth map, motion, active contour model based segmentation

29. Depth Assisted Object Segmentation
Performance evaluation
Object segmentation by the proposed method and the background subtraction method
How much relevant object pixels the proposed method has extracted

30. Outline
Introduction
Surface Reconstruction from Unorganised Sample Points
3D Object Segmentation
Depth Assisted Video Object Segmentation
Depth Assisted Video Object Tracking
Conclusion and Future Works
List of Publications

31. Depth Assisted Video Object Tracking
Locating and assigning consistent labels to the tracked objects in a generic video sequence
Target locating: predict the location of interesting objects being tracked in the next frame
Object matching: establish correspondence of detected objects across frames
Accurate and robust multiple object tracking is a challenging problem
complex backgrounds, arbitrary object motion, changing appearance patterns of non-rigid objects, and partial or full object overlaps

32. Depth Assisted Video Object Tracking
Region-based tracking
Video objects are segmented into a set of small regions
Object identification is established based on these regions
Kernel-based tracking
The kernel is an object region, either a simple shape with associated colour histogram or a small area roughly represents object shape
Feature-based tracking
Images elements, including edges, colour, texture, motion vector, object contour, etc.
Object identification is established in the feature space, such as the colour histogram

33. Depth Assisted Video Object Tracking
Develop a depth assisted solution of multiple objects tracking under various type of overlaps
A stereo-vision system has the ability to separate objects at different depth layers under partial overlap
System overview:
Stable object segmentation from cluttered backgrounds
Depth assisted overlap detection
Depth-based partial overlap handling
Severe/full overlap handling

34. Depth Assisted Video Object Tracking
Depth assisted overlap detection
Detect the occurrence and the end of partial overlap based on the depth map segmentation and object’s overlap situation in the previous frame
Overlap handling
Different object tracking techniques are employed according to various overlap situations

35. Depth Assisted Video Object Tracking
Tracking non-overlaid objects: the shortest three-dimensional Euclidean distance
Tracking partial overlaid objects in different disparity layers: colour-based silhouette matching
Tracking partial overlaid objects in one disparity layer: iterative silhouette matching algorithm
Overlaid objects are separated based on their average disparity range in the previous frame

36. Depth Assisted Video Object Tracking
Severe/full overlap handling:
Unmatched foreground region can be a new object or a splitting object
Local best matching
Splitting object: foreground region matches one of the overlaid object

37. Depth Assisted Video Object Tracking
Experimental results
Object tracking under partial overlap in different disparity layers
Object tracking under partial and severe overlap
Tracking system can rematch objects after full overlap

38. Depth Assisted Video Object Tracking
Performance evaluation
Object tracking by template matching
works well in the case of non-overlap and partial overlap due to the updating of template but failed under severe overlap
Object tracking by mean-shift
Lost of severe overlaid object and mismatches the splitting object

39. Depth Assisted Video Object Tracking
Performance evaluation

40. Outline
Introduction
Surface Reconstruction from Unorganised Sample Points
3D Object Segmentation
Depth Assisted Video Object Segmentation
Depth Assisted Video Object Tracking
Conclusion and Future Works
List of Publications

41. Conclusions
A Delaunay-based region growing method is developed to reconstruct 3D object surface from a set of sample points
The new candidate triangle selection criterion ensures the region growing process smooth and robust at sharp boundaries
The 3D object segmentation algorithm divides object into several meaningful components by means of the fuzzy k-means method
A fuzzy cluster validity index is used to find the optimal number of components from a set of local maxima
Depth assisted video object segmentation
A depth-based segmentation framework is introduced, which consists of a depth and motion based object mask generation step and an object boundarie refinement step to extract semantic object regions
Depth assisted video object tracking
The overlap detection method is based on the depth map segmentation and the overlap situation of each track in the previous frame
Different object tracking strategies are employed according to the various overlap situations

42. Future Works
Non-uniform point cloud simplification: remove redundant sample points adaptively according to the surface curvature
Combination of depth map segmentation and motion segmentation
A quality measurement is needed to evaluate the quality of depth map and motion segmentation
Assign a probability measurement to each associated object according to the matching result so that the tracking system can recover from failed object association in the previous frame

43. List of Publications
Y. Ma, S. Worrall, and A. M. Kondoz, “Depth Assisted Occlusion Handling in Video Object Tracking,” Signal Processing: Image Communication, Elsevier Science (under review)
Y. Ma S. Worrall, and A. M. Kondoz, “3D Point Segmentation Using Critical Points and Fuzzy Clustering,” in Proc. the 4th IET Conference on Visual Information Engineering, London, 2007
Y. Ma S. Worrall, and A. M. Kondoz, “Automatic Video Object Segmentation Using Depth Information and an Active Contour Model,” in Proc. IEEE International Workshop on Multimedia Signal Processing, Cairns, Queensland, Australia, 2008
Y. Ma S. Worrall, and A. M. Kondoz, “Video Object Segmentation in Cluttered Background Using Depth and Spatial-temporal information,” in Proc. 3rd International Workshop on Hybrid Artificial Intelligence Systems, Burgos, Spain, 2008
Y. Ma S. Worrall, and A. M. Kondoz, “Depth Assisted Visual Tracking,” in Proc. 10th IEEE International Workshop on Image Analysis for Multimedia Interactive Services, London, 2009

44. Thank You!.. Any questions? Contacts: Yingdong Ma