Shape Recognition and Pose Estimation for Mobile Augmented Reality Author ： N. Hagbi, J. El-Sana, O. Bergig, and M. Billinghurst Date ： Speaker ： Sian-Lin Hong IEEE Transactions On Visualization And Computer Graphics, Vol. 17, No. 10, pp , October 2011.

Outline 1. Introduction 2. Related Work 3. Nestor 4. Contextual Shape Learning 5. Experimental Results 2

1. Introduction (1/5) 1. Model-Based visual tracking has become increasingly attractive in recent years in many domains 2. Visual tracking is often combined with object recognition tasks 3. In AR applications, model-based recognition and 3D pose estimation are often used for superposing computer-generated images over views of the real world in real time 3

1. Introduction (2/5) 1. Fiducial-based computer vision registration is popular in AR applications due to the simplicity and robustness it offers 2. Fiducials are of predefined shape, and commonly include a unique pattern for identification 4

1. Introduction (3/5) 1. Natural Feature Tracking (NFT) methods are becoming more common, as they are less obtrusive and do not require to modify the scene 2. This paper describe a recognition and pose estimation approach that is unobtrusive for various applications, and still maintains the high levels of accuracy and robustness offered by fiducial markers 3. We recognize and track shape contours by analyzing their structure 5

1. Introduction (4/5) 1. When a learned shape is recognized at runtime, its pose is estimated in each frame and augmentation can take 6

1. Introduction (5/5) 1. Virtual content can be automatically assigned to new shapes according to a shape class library 2. When learning a new shape, the system can classify it to one of the predefined shape classes 3. Which define the default virtual content that should be automatically assigned to it 7

2. Related work (1/2) 1. Object recognition and pose estimation are two central tasks in computer vision and Augmented Reality 2. Object recognition methods aim to identify objects in images according to their known description 3. The cores of AR applications are based on recognition and pose estimation to allow the appropriate virtual content to be registered and augmented onto the real world 8

1. Fiducial-based registration methods have been used from the early days of AR 2. The frame is first used for rectification of the pattern inside of it 3. ARToolKit locates a square frame in the image and calculates its pose 2. Related work (2/2) 9

3. Nestor (1/9) 1. Nestor is a recognition and 3D pose tracking system for planar shapes 2. The main goal of Nestor is To serve as a registration solution for AR applications, which allows augmenting shapes with 3D virtual content 3. Nestor can be used to augment shapes that have visual meanings to humans with 3D models having contextual correspondence to them 10

3. Nestor (2/9) 1. Features extracted from each concavity are then used to generate a first estimate for the homography between each hypothesized library shape and the image shape 2. We calculate an estimate of the homography between the image and library shapes using features from all concavities 11

3. Nestor (3/9) 1. Begin the processing of each frame by extracting the contours of visible shapes 2. We generally assume the shapes are highly contrasted from their background and take a thresholding-based approach 3. Apply adaptive thresholding to the image using integral images That a window of size 8*8 usually gives the most pleasing results 12

3. Nestor (4/9) 1. The contour of each image shape is then extracted by straightforward sequential edge linking as an ordered list of points 2. Check for the convexity of contours and drop ones that are convex or close to convex 3. Finally apply median filtering to each contour and get smooth contours 13

3. Nestor (5/9) 1. We use a construction which is based on the bitangent lines to the contour, illustrated in Fig. 2a 2. Each bitangent line l gives two tangency points, Pa and Pb, which segment a concavity from the rest of the curve, known as the M-curve 14

3. Nestor (6/9) 1. The occluded shape may thus contain concavities that point to different library shapes 2. Since we are tracking recursively on a frame-to-frame basis, a shape can be tracked from previous frames 15

3. Nestor (7/9) 1. The system maintains a shape library that contains the shapes learned so far 2. The system can load a directory of shape files and learn them 3. User can also teach the system new shapes at runtime 16

3. Nestor (8/9) 1. When teaching the system a new shape, the image goes through the same recognition step described in the Shape Recognition Section, and its signatures are hashed 2. The curve, its signatures, and additional required information are stored in the shape library 3. Once the shape is found, it is moved into the visible shape list 17

3. Nestor (9/9) 1. The shape list is searched for each shape once per execution, when the shape first appears 2. This strategy can be useful when Only a few shapes are visible in a single frame Only a small number of shapes are used through a single execution 18

4. Contextual shape learning (1/4) 1. Previously, to teach the system a new shape, the user had to Show it frontally to the camera explicitly assign a model to it 2. To learn an unknown shape appearing in the image, upon user request, we automatically perform rectification according to the rectifying transformation recovered from a tracked shape that lies in the same plane 19

4. Contextual shape learning (2/4) 1. The nearest tracked shape NC to the new shape C is found according to the shapes’ centroids 2. This projects C to the image plane outside of the image bounds and to a scale that depends on its location relative to C in the real world 3. We finally centralize the rectified contour of C 20

4. Contextual shape learning (3/4) 21

4. Contextual shape learning (4/4) 22

5. Experimental results (1/6) 1. We benchmarked and tested Nestor on a Nokia N95 mobile phone and a Dell Latitude D630 notebook computer 2. The Nokia N MHz processor camera that captures 320 ＊ 240 pixel images 3. The Dell notebook 2.19 GHz processor webcam that provides 640 ＊ 480 pixel images 23

5. Experimental results (2/6) 1. We measured the relation between the number of tracked shapes in each frame and per-frame tracking time 24

5. Experimental results (3/6) 1. To assess this relation, we measured the recognition rate of the system with different shape library sizes and slants 25

5. Experimental results (4/6) 1. The experiment was performed using the notebook configuration 2. The camera was fixed approximately 40 cm from the shapes 3. For each library size, the recognition rate was tested on all of the shapes in the library 26

5. Experimental results (5/6) 1. We also measured the reprojection error for different distances of the camera from imaged shapes 2. For each library shape and ARToolkit fiducial, 50 randomly sampled points in the area of the shape/fiducial were checked using a random transformation synthesizer 27

5. Experimental results (6/6) 28