Tracking Features with Large Motion

Abstract Problem: When frame-to-frame motion is too large, KLT feature tracker does not work. Solution: Estimate the motion at the deepest pyramid level by matching the characteristic curves of the consecutive images.

Introduction Feature tracking is an important issue in many computer vision applications. In order to allow the tracker to handle large motion, people usually use a pyramidal implementation of the KLT feature tracker.

We propose a method to extend the pyramidal KLT feature tracker to deal with image I and J are taken from widely different viewpoint. I and J are two consecutive images in an image sequence.

Sum of square differences (SSD) Given a feature point on I, the goal is to find the correspondence on J. Where I(x) and J(x) are intensity with image point. d: displacement vector W: a small integration window centered at the feature point.

KLT feature tracker Let, we can use the linear system to find d:

Automatically selecting good features is also an important issue. Let if,then the image point is a good feature.

Pyramidal implementation of KLT feature tracker Let be original image of I and J. : downsampling of : downsampling of… : height of the image pyramid. Similarly, we can obtain images for image J. After constructing the image pyramid of image I and J, we apply the pyramidal KLT feature tracker.

First step: : the displacement vector at the deepest level. Second step: s: downsampling factor. = + repeat second step until estimate.

Accommodating very large motion There are two problems: In practice, is a small number; otherwise the image size of will be too small to carry enough details for each feature. where : d* : true displacement vector for a feature point

The feature point dissolve when the height of the image pyramid is too large.

For those cases, our method provides a solution by computing the motion estimates and at the deepest pyramid level. The effect of computing and is to provide a coarse motion at deepest level which makes the residual motion small enough to satisfy the assumption.

Characteristic curves Define as the characteristic curve for x-axis computed from image I. for y-axis from image I.

After the four curveshave been computed, we compute by matching the two characteristic curves. can be computed in the same manner.

Motion estimation at the deepest pyramid level The goal of the motion estimator is to find the best labeling that assigns a label to each element.

: domain of the curve ( ) : displacement to element ( ) : when element is considered to be occluded. : the ordered set of element where The penalty is use to penalize the situations where discontinuity is occurred. The penalty serves as a threshold that affects whether the motion estimator should assign a label to the element.

After finding the optimal labeling, the motion estimator computes by: For those element in, we compute their motion estimates by interpolating the displacements of the elements in the neighborhood.

Feature tracking with pre-checking Consider a feature point. x is a lost feature when one of the following conditions is satisfied: Therefore, no computational power is wasted.

Results and comparisons Two image sequences are tested here. Show : 71-frames 320 x 240 pixels Building : 73-frames 480 x 320 pixels

Birchfield’s implementation

Comparison between the number of the tracked features

Comparison between the running time