The Implementation of Markerless Image-based 3D Features Tracking System Lu Zhang Feb. 15, 2005

Motivations Objective Find more efficient algorithms to implement 3D volume tracking based on 2D Image sequences. Problems in this topic 1. Huge datasets For only one data file: 128*128* Computation time Applications 1. On sensors 2. On robotics

Outline Previous work  Flowchart of the system  Algorithms Current work  Improved algorithms  Comparisons Future work  Problems unsolved  How to enhance computation speed

Previous Work The original image- based 2D dataset : Size: 512*512*40*(R, G, B) Flowchart and Modulus Input images Segmentation Feature extraction Classification Graph building Basic features classes Directed acyclic graph

Previous Work - Algorithms Modulus1: Segmentations  Global Thresholding: Problems: One threshold to all image sequences.  Iterative region growing method [1] After applying this method to segmented image sequences: VS

Previous Work-Feature Extraction Output from Feature extraction module viewID mx my areas labeling timeID label. Modulus2: Feature Extraction After gaining region information from segmentation stage, we can browse each region to find basic features: Areas – The count of all pixels in the region. Center of Gravity –The center of all points in one region.

Classification /Feature tracking Modulus3: Classification  One Assumption Time between successive data sets is small: we can assume the difference between a pair of views should not vary too much.  Euclidean Distance Classifier

Current Work-Improvement Modulus 1: Segmentation: Optimal Thresholding: Isodata algorithm  Segment images into two parts using a starting threshold value.  Calculate the mean (mf,0) of the foreground pixels and the mean (mb,0) of background pixels.  A new threshold value is now computed as the average of these two sample means.  The process is repeated, based upon the new threshold, until the threshold value does not change any more.

Previous Work - Algorithms Modulus1: Segmentations  Region growing: Purpose: Trying to separate overlapped objects Algorithms: Region growing based on Marr-Hildreth and sobel edge detectors

Current Work-Features extraction Feature Extraction  Diameter - Diameter is the distance between two points on the boundary of the region whose mutual distance is the maximum.  Major Axis of The Region – the major axis of the region is the line which minimizes: These two features are relatively robust, and the second feature: major axis can help detect the reflection part on objects.

Current Work-Features extraction Feature Extraction  Compute major axis PCA  Diameter 1. Rotate the X-Y coordinate to let the new X-coordinate is the major axis 2. Divide the 2D plane into four regions, find the furthest points on each region 3. Calculate the Euclidean distance

Current Work-Features extraction Experiment results from diameter detector

Current Work-Features extraction Experiment results of from feature extraction modulus: TimeID ViewID Mx My R G B areas diameter angle label

Current Work-Feature Extraction Modulus2: Feature Extraction Problems solving: Reflections: According to the experiment result on the right: to some big objects, their reflections which come from the distance transformation when we pre-projected 3D objects onto 2D image plane are distracted as different objects. Algorithms: Using the property of major axis: because they belong to the same object, their major axis should parallel or at least have similar angles to each other

Current work-Classification /Feature tracking Classification methods Euclidean Distance Classifier Evolution in time-varying images There are five different changes of regions between a pair of views.  Continuation: one feature continues from dataset at t1 to the next dataset at t2  Creation: new feature appear in t2  Dissipation: one feature weakens and becomes part of the background  Bifurcation: one feature in t1 separates into two or more features in t2.  Amalgamation: two or more features merge from one time step to the next.

Current Work Output from Classification module New class to preserve the output dataset from Classification module: class LabelTrack(). It preserve the information: 1. ViewID: camera positions, we will move camera around the object in order to restore 3D object. 2. timeID: time order, for each camera position, we will take several time- varying images 3. classID: class number after correspondence computation between a pair of images in time order 4. Label: the original region numbers before correspondence computaton 5. R, G, B: the color information for each pixel 6. Coordinate x, y: the 2D coordinate of the projection of 3D object. 7. Forward pointer: preserve the labeling information of the previous dataset 8. Backward pointer: preserve the labeling information of the next dataset

Future Work-Speed Enhancement The importance of computation time Size of mine dataset: 512*512*24*40(time orders)*N(camera positions) In [5], the computation time for 128^3*10 is 7 minutes. In the previous work, I use 4 minutes for 512*512*24*40. In the current work, most I/O operations have been removed, although the computation time is around 5 minutes. Most of the time is consumed on Marr-Hildreth edge detector.

REFERENCES [1] Snyder and Cowart, “An Iterative Approach to Region Growing”, IEEE transaction on PAMI, 1983 [2] Wesley E.Snyder and Hairong Qi, “Machine Vision”, Cambridge [3] Richard O.Duda, Peter Hart, David Stork, “Pattern Classification”, Prentice Hall [4] Rafael Gonzalez, Richard Woods,”Digital Image Processing”, 2 nd, Prentice Hall [5] D.Silver, Xin Wang, ”volume tracking”, Visualization '96. Proceedings.27 Oct.-1 Nov. 1996

Thanks Any questions?