Motion from image and inertial measurements

Motion from image and inertial measurements
Dennis Strelow Carnegie Mellon University

On the web Related materials: these slides related papers movies
VRML models at: Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Introduction (1) From an image sequence, we can recover:
6 degree of freedom (DOF) camera motion without knowledge of the camera’s surroundings without GPS Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Introduction (2) Fitzgibbon
Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Introduction (3) Potential applications include: modeling from video
Yuji Uchida Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Introduction (4) micro air vehicles (MAVs) AeroVironment Black Widow
AeroVironment Microbat Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Introduction (5) rover navigation Hyperion Nister, et al.

Introduction (6) search and rescue robots
Rhex (movies: Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Introduction (7) NASA Personal Satellite Assistant (PSA)

Introduction (8) For these problems, we want: 6 DOF motion
in unknown environments without GPS or other absolute positioning Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

in unknown environments without GPS or other absolute positioning using small, light, and cheap sensors Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

in unknown environments without GPS or other absolute positioning using small, light, and cheap sensors over the long term Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Introduction (9) Long-term motion estimation:
absolute distance or time is long only a small fraction of the scene is visible at any one time Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Introduction (10) given these requirements, cameras are promising sensors… …and many algorithms for motion from images already exist Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Introduction (11) But, where are the systems for estimating the motion of: over the long term? Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Introduction (12) …and for automatically modeling rooms buildings
cities from a handheld camera? Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Introduction (13) Motion from images suffers from some long-standing difficulties This work attacks these problems by… exploiting image and inertial measurements robust image feature tracking recognizing previously mapped locations exploiting omnidirectional images Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Outline Motion from images refresher bundle adjustment difficulties
Motion from image and inertial measurements Robust image feature tracking Long-term motion estimation Conclusion Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Motion from images: refresher (1)
A two-step process is common: sparse feature tracking estimation Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

A two-step process is common: sparse feature tracking estimation Sparse feature tracking: inputs: raw images outputs: projections Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Template matching: correlation tracking Lucas-Kanade (Lucas and Kanade, 1981) Extraction and matching: Harris features (Harris, 1992) Scale Invariant Feature Transform (SIFT) keypoints (Lowe, 2004) Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

The second step is estimation: inputs: projections outputs: 6 DOF camera position at the time of each image 3D position of each tracked point Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

bundle adjustment (various, 1950’s) Kalman filtering (Broida, Chandrashekhar, and Chellappa, 1990) variable state dimension filter (VSDF) (McLauchlan, 1996) two- and three-frame methods (Hartley and Zisserman, 2000; Nister, et al. 2004) Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Motion from images: bundle adjustment (1)
From tracking, we have the image locations xij for each point j and each image i Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Suppose we also have estimates of: the camera rotation ρi and translation ti at time of each image 3D point positions Xj of each tracked point Then, we can compute reprojections: Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

So, minimize: with respect to all the ρi, ti, Xj Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Motion from images: difficulties (1)
Estimation step can be very sensitive to… incorrect or insufficient image feature tracking camera modeling and calibration errors outlier detection thresholds sequences with degenerate camera motions Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Iterative batch methods have poor convergence or may fail to converge if: observations are missing the initial estimate is poor Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Recursive methods suffer from: poor prior assumptions on the motion poor approximations in state error modeling Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Resulting errors are: gross local errors long term drift Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

151 images, 23 points manually corrected Lucas-Kanade Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

dash-dotted line: accurate estimate solid line: image-only, bundle adjustment estimate squares: ground truth points Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Outline Motion from images Motion from image and inertial measurements
inertial sensors algorithms and results related work Robust image feature tracking Long-term motion estimation Conclusion Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Motion from image and inertial measurements: inertial sensors (1)
inertial sensors can be integrated to estimate six degree of freedom motion Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

But many applications require small, light, and cheap sensors Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Integrating the outputs of these low grade sensors will produce drifting motion because of: noise unmodeled nonlinearities Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

And, we can’t even integrate until we can separate the effects of… rotation ρ gravity g acceleration a slowly changing bias ba noise n …in the accelerometer measurements Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Image and inertial measurements are highly complementary With inertial measurements we can: decrease sensitivity in image-only estimates establish two rotation angles without drift establish the global scale Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Image and inertial measurements are highly complementary With inertial measurements we can: decrease sensitivity in image-only estimates establish two rotation angles without drift establish the global scale …even with our low-grade sensors Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

With image measurements, we can: reduce the drift in integrating inertial data distinguish between… rotation ρ gravity g acceleration a bias ba noise n …in accelerometer measurements Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Motion from image and inertial measurements: algorithms and results (1)
This work has developed both: batch recursive algorithms for motion from image and inertial measurements Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Gyro measurements: ω’, ω: measured and actual angular velocity bω: gyro bias n: gaussian noise Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Accelerometer measurements: ρ: rotation a’, a: measured and actual acceleration g: gravity vector ba: accelerometer bias n: gaussian noise Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

batch algorithm minimizes a combined error: Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

image term Eimage is the same as before Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

inertial error term Einertial is: Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

( : translation estimate for image i – 1) ti-1 ( : translation estimate for image i) translation ti I(ti-1, …) ( : translation integrated from previous estimate) τi-1 (time of image i - 1) τi (time of image i) time Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

translation time τ0 τ1 τ2 τ3 τ4 τ5 τf-3 τf-2 τf-1 Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

It(τi-1, τi ,…, ti-1) depends on: τi-1, τi (known) all inertial measurements for times τi-1< τ < τi (known) ρi-1, ti-1 g bω, ba camera linear velocities: vi Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

dash-dotted line: batch estimate from image and inertial solid line: image-only, bundle adjustment estimate squares: ground truth points Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

IEKF for the same sensors, unknowns dash-dotted line: batch estimate solid line: IEKF estimate Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Difficulties with IEKF for this application: prior assumptions about motion smoothness cannot model relative error between adjacent camera positions So, converting the batch algorithm into a variable state dimension filter (VSDF) is a promising future direction Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

IEKF assumptions on motion smoothness dash-dotted line: batch estimate solid line: IEKF estimate left: IEKF propagation variances just right right: IEKF propagation variances too strict Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Motion from image and inertial measurements
Recap: image, gyro, and accelerometer measurements batch algorithm recursive algorithm experiments evaluate batch and recursive algorithms establish basic facts about motion from image and inertial measurements Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Robust image feature tracking Lucas-Kanade and real sequences The “smalls” tracker Long-term motion estimation Conclusion Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Robust image feature tracking: Lucas-Kanade and real sequences (1)
Combining image and inertial measurements improves our situation, but… we still need accurate feature tracking tracking some sequences do not come with inertial measurements Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

better feature tracking for improved 6 DOF motion estimation remaining results will be image-only Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Lucas-Kanade has been the go-to feature tracker for shape-from-motion minimizes a correlation-like matching error using general minimization evaluates the matching error at only a few locations subpixel resolution Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Additional heuristics used to apply Lucas-Kanade to shape-from-motion: task: heuristic: choose features to track high image texture identify mistracked, occluded, no-longer-visible convergence, matching error handle large motions image pyramid Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

But Lucas-Kanade performs poorly on many real sequences… Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Robust image feature tracking: the “smalls” tracker (1)
smalls is a new feature tracker targeted at 6 DOF motion estimation exploits the rigid scene assumption eliminates the heuristics normally used with Lucas-Kanade SIFT is an enabling technology here Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

First step: epipolar geometry estimation use SIFT to establish matches between the two images get the 6 DOF camera motion between the two images get the epipolar geometry relating the two images Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Second step: track along epipolar lines use nearby SIFT matches to get initial position on epipolar line exploits the rigid scene assumption eliminates heuristic: pyramid Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Third step: prune features geometrically inconsistent features are marked as mistracked and removed clumped features are pruned eliminates heuristic: detecting mistracked features based on convergence, error Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Fourth step: extract new features spatial image coverage is the main criterion required texture is minimal when tracking is restricted to the epipolar lines eliminates heuristic: extracting only textured features Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

left: odometry only right: images only average error: 1.74 m maximum error: 5.14 m total distance: 230 m Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Recap: exploits the rigid scene and eliminates heuristics allows hands-free tracking for real sequences can still be defeated by textureless areas or repetitive texture Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Robust image feature tracking Long-term motion estimation proof of concept system experiment Conclusion Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Long-term motion estimation: proof of concept system (1)
Image-based motion estimates from any system will drift: if the features we see are always changing given sufficient time if we don’t recognize when we’ve revisited a location Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

To limit drift: recognize when we’ve returned to a previous location exploit the return A proof of concept system demonstrates these capabilities Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

system state S image indices: I = {i1, …, in} “smalls” tracker state: 2D feature history for images in I variable state dimension filter (VSDF) state for images in I: 6 DOF camera positions, covariances for images in I 3D positions for features visible in I SIFT keypoints for image in Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

{0} {0, 1, 2} {0, 1, …, 8} 1 2 3 4 5 6 7 8 {0, 1} Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

{0} {0, 1, 2} {0, 1, …, 8} 1 2 3 4 5 6 7 8 {0, 1} States: rollback non-rollback Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

{0} {0, 1, 2} {0, 1, …, 8} 1 2 3 4 5 6 7 8 {0, 1} 8 States: rollback non-rollback Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

{0} {0, 1, 2} 1 2 3 4 5 6 7 8 {0, 1} 8 {0, 1, 2, 3, 8} States: rollback non-rollback pruned Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

{0, …, 6, 11, 12, 17, …, 20} 17 18 19 20 11 12 13 14 1 2 3 4 5 6 7 8 8 9 10 11 14 15 16 17 States: rollback non-rollback pruned Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

When to “roll back”? examine the camera covariances for the current state and the candidate rollback state check the number of SIFT matches extend from the candidate state examine the camera covariances for the current state and the resulting extended state Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Long-term motion estimation: experiment (1)
CMU FRC highbay views; 945 images total Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

CMU FRC highbay Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

CMU FRC highbay (first forward pass: images 0-213) Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

CMU FRC highbay (first backward pass: images ) Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

CMU FRC highbay (second forward pass: images ) Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

CMU FRC highbay (second backward pass: images ) Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

CMU FRC highbay (third forward pass: images ) Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

CMU FRC highbay (third backward pass: images ) Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

normally, the system produces a general tree of states 17 18 19 20 11 12 13 14 1 2 3 4 5 6 7 8 8 9 10 11 14 15 16 17 States: rollback non-rollback pruned Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

for this example, the “rollback” states are restricted to the first forward pass 10 11 12 14 13 14 15 14 … 1 2 3 4 5 6 7 213 8 9 14 16 17 18 17 States: rollback non-rollback pruned Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

movie…bottom half is smalls output: Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

movie…top half is motion estimates: Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Robust image feature tracking Long-term motion estimation Conclusion remaining issues relationship to recent Sarnoff work Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Conclusion: remaining issues
all: system is experimental, not optimized for speed image and inertial: VSDF “smalls”: integration of gyro, more robustness to poor texture needed long-term: “roll back” space, computation grow with sequence length Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Conclusion: relationship to recent Sarnoff work (1)
This work starts from general estimation frameworks optimal estimates error estimates good platform for evaluating sensor models general Sarnoff work emphasizes two- and three-frame methods speed Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

This work is more and less general than recent Sarnoff work This work: image and inertial measurements omnidirectional images long-term estimation Sarnoff: can exploit stereo images Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

This system should exploit Sarnoff’s work accurate motion estimation always requires accurate tracking the “smalls” tracker accuracy depends on robust two- and three-frame estimation so, “smalls” should adopt Sarnoff’s fast two- and three-frame estimation to increase trials Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Thanks! Related materials: these slides related papers movies
VRML models at: Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Motion from omnidirectional images (1)

left: non-rigid camera right: rigid camera squares: ground truth points solid: image-only estimates dash-dotted: image-and-inertial estimates Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

In this experiment: omni images conventional images + inertial have roughly the same advantages But in general: inertial has some advantages that omni images alone can’t produce omni images can be harder to use Dennis Strelow -- Motion estimation from image and inertial measurements – December 10,

Motion from image and inertial measurements

Similar presentations

Presentation on theme: "Motion from image and inertial measurements"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Motion from image and inertial measurements

Similar presentations

Presentation on theme: "Motion from image and inertial measurements"— Presentation transcript:

Similar presentations

About project

Feedback