Download presentation
Presentation is loading. Please wait.
1
Towards kHz 6-DoF Visual Tracking Using an Egocentric Cluster of Rolling Shutter Cameras
Akash Bapat1, Enrique Dunn1,2 & Jan-Michael Frahm1, UNC Chapel Hill, USA1, Stevens Institute of Technology, USA2
2
AR/VR System Components
Tracking Rendering Display Image source :
3
AR/VR System Components
Tracking Rendering Display Tz Tx Ty Find 6-DoF head pose1 LaValle etal, Head Tracking for Oculus Rift
4
Positional tracking 2,3 Camera fps = 60 fps
Tracking Systems Our NDI Optotrak Certus1 Tracking frequency ≈ 5kHz State of the art Positional tracking 2,3 Camera fps = 60 fps Orientation tracking IMU at 1000 Hz 4,5 Oculus Rift DK2 NDI Frequency HiBall Self-tracker DK2 Complexity or cost LaValle etal, Head Tracking for Oculus Rift
5
Rolling shutter capture1
Row-by-row acquisition of linescan snapshots at slightly different times Stream of row-images
6
Stream of Rows Frequency of row-samples F = FPS * Height
Rolling shutter1 Frequency of row-samples F = FPS * Height = 120 * 720 > 80kHz
7
Our Tracker Enabling component for rendering Use commodity cameras
>30 kHz tracker 30 kHz Renderer Enabling component for rendering Zheng et al., 2014 Lincoln et al., 2016 Use commodity cameras Tracking frequency Up to 80 kHz Break frame-rate barrier Process each row of image GoPro1
8
Artifact visualisation1
Related Work Removing rolling shutter Forssen et al., CVPR 2010 Track points using KLT tracker, estimate rotation Parametrize intra-frame rotation as spline Geometric problems Multi-view stereo: Saurer et al., ICCV 2013 Adapt plane sweep stereo for rolling shutter Velocity estimation Ait-Aider et al., ICVS 2006 Solve for pose and velocity using bundle adjustment Use 2D-3D correspondence, similar to camera calibration
9
High frequency 6DoF tracking
Approach Overview Stream of rows Cluster Estimation Framework Camera Cluster High frequency 6DoF tracking Output Spatial Stereo Temporal Stereo Input Our approach
10
Dynamic sensor with periodic movement2
Cluster Spatial Stereo Temporal Estimation Framework Our Cluster GoPro1 Dynamic sensor with periodic movement2 Small vertical FoV
11
Our Cluster N stereo-pairs of rolling shutter camera, N=5
Spatial Stereo Temporal Estimation Framework Our Cluster 1 2 3 4 5 N stereo-pairs of rolling shutter camera, N=5 Precalibrated intrinsic and extrinsics Temporal sync Known history of motion Hi-Ball for ground truth H Geometry of cluster H
12
Motion Model 3D point 𝑿 𝑐𝑎𝑚𝑖 𝑡+1 = 𝑴 𝑐𝑎𝑚𝑖 𝑉1 𝑿 𝑐𝑎𝑚𝑖 𝑡 3D point X
Cluster Spatial Stereo Temporal Estimation Framework Motion Model 3D point 𝑿 𝑐𝑎𝑚𝑖 𝑡+1 = 𝑴 𝑐𝑎𝑚𝑖 𝑉 𝑿 𝑐𝑎𝑚𝑖 𝑡 3D point X
13
Motion Model Moved 3D point 3D point Motion of camera
Cluster Spatial Stereo Temporal Estimation Framework Motion Model Moved 3D point 3D point Motion of camera 𝑿 𝑐𝑎𝑚𝑖 𝑡+1 = 𝑴 𝑐𝑎𝑚𝑖 𝑉 𝑿 𝑐𝑎𝑚𝑖 𝑡 3D point X
14
Motion Model Moved 3D point 3D point 𝑿 𝑐𝑎𝑚𝑖 𝑡+1 = 𝑴 𝑐𝑎𝑚𝑖 𝑉1 𝑿 𝑐𝑎𝑚𝑖 𝑡
Cluster Spatial Stereo Temporal Estimation Framework Motion Model Moved 3D point 3D point 𝑿 𝑐𝑎𝑚𝑖 𝑡+1 = 𝑴 𝑐𝑎𝑚𝑖 𝑉 𝑿 𝑐𝑎𝑚𝑖 𝑡 𝑿 𝑐𝑎𝑚𝑖 𝑡+1 = 𝑽𝒊 𝑴𝑐𝑙𝑢𝑠𝑡𝑒𝑟 𝑽 𝒊 −𝟏 𝑿 𝑐𝑎𝑚𝑖 𝑡 Vi cami RS cluster
15
Linearized Motion Model
Cluster Spatial Stereo Temporal Estimation Framework Linearized Motion Model 𝑿 𝑐𝑎𝑚𝑖 𝑡+1 = 𝑴 𝑐𝑎𝑚𝑖 𝑉 𝑿 𝑐𝑎𝑚𝑖 𝑡 𝑿 𝑐𝑎𝑚𝑖 𝑡+1 = 𝑽𝒊 𝑴𝑐𝑙𝑢𝑠𝑡𝑒𝑟 𝑽 𝒊 −𝟏 𝑿 𝑐𝑎𝑚𝑖 𝑡 𝒅𝑴= 1 −θ 𝑧 θ 𝑦 𝛿𝑇 𝑥 θ 𝑧 1 −θ 𝑥 𝛿𝑇 𝑦 −θ 𝑦 θ 𝑥 1 𝛿𝑇 𝑧 𝑴 𝒄𝒍𝒖𝒔𝒕𝒆𝒓 = 𝑅( 𝜃 𝑥 , 𝜃 𝑦 , 𝜃 𝑧 ) 𝑇 0 1
16
Motion Estimation Stereo in space
Cluster Spatial Stereo Temporal Estimation Framework Motion Estimation 𝑿 𝑐𝑎𝑚𝑖 𝑡+1 = 𝑽𝒊 𝑴𝑐𝑙𝑢𝑠𝑡𝑒𝑟 𝑉 𝑖 −1 𝑿 𝑐𝑎𝑚𝑖 𝑡 𝑿 𝑐𝑎𝑚𝑖 𝑡 = 0 y d 1 𝑇 Stereo in space
17
Motion Estimation 𝑿 𝑐𝑎𝑚𝑖 𝑡+1 = ∆x y+∆𝑦 d+∆𝑑 1 𝑇
Cluster Spatial Stereo Temporal Estimation Framework Motion Estimation 𝑿 𝑐𝑎𝑚𝑖 𝑡+1 = ∆x y+∆𝑦 d+∆𝑑 1 𝑇 𝑿 𝑐𝑎𝑚𝑖 𝑡+1 = 𝑽𝒊 𝑴𝑐𝑙𝑢𝑠𝑡𝑒𝑟 𝑉 𝑖 −1 𝑿 𝑐𝑎𝑚𝑖 𝑡
18
Stereo Estimation Spatial stereo sd Temporal stereo st
Cluster Spatial Stereo Temporal Estimation Framework Stereo Estimation Spatial stereo sd Use stereo-pair of cameras Measure pixel disparity Temporal stereo st Measure small shifts in pixel Compare row at time t and t-k t1 t2
19
Visualization of row-image
Cluster Spatial Stereo Temporal Estimation Framework Binary Row Descriptor Visualization of row-image (296th) … 𝑑 2 𝐺(𝑥,0,σ) 𝑑 𝑥 2 ∗𝑟𝑜𝑤 2nd derivative Sign
20
Spatial Stereo : Measure Disparity
Cluster Spatial Stereo Temporal Estimation Framework Spatial Stereo : Measure Disparity Compare binary descriptor of rows of stereo cameras Fast hamming cost matching 𝑑𝑒𝑝𝑡ℎ= 𝑓𝑜𝑐𝑎𝑙 ∗𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒 𝑑𝑖𝑠𝑝𝑎𝑟𝑖𝑡𝑦 Row samples of stereo pair Reconstructed row using homography Stereo Pair Binary descriptor Previous frame Current frames st sd baseline focal length P (x,y,d) xL xR Stereo in space
21
Temporal Stereo: Measure Shift st
Cluster Spatial Stereo Temporal Estimation Framework Temporal Stereo: Measure Shift st Different regions in space are captured at different timestamps Stereo in time Need snaps of same space but at different timestamps Row-wise homography Known [R |t] per row Reference time Current time Compare Warped to match reference time pose Previous Frame
22
Temporal Stereo: Measure Shift st
Cluster Spatial Stereo Temporal Estimation Framework Temporal Stereo: Measure Shift st Reconstruct row using per- row homography Use binary descriptors and hamming distance Row sample of stereo pair Reconstructed row using homography One Camera of Stereo Pair Binary descriptor Sampled from previous frame sx Current frame
23
Adaptive Reference Leave sufficient motion
Cluster Spatial Stereo Temporal Estimation Framework Adaptive Reference Leave sufficient motion Row-to-row motion Interpolation & pixel measurement noise Satisfy small-motion assumption Reference should not be far away Current frame Current Row (t) Reference row: t-k Noisy measurements Small motion assumption invalid Estimation reliability k
24
Confidence Scores Quality of minimum: cPKR Temporal consistency : ct
Cluster Spatial Stereo Temporal Estimation Framework Confidence Scores High confidence Low Confidence Shift Hamming distance Quality of minimum: cPKR Is the valley unique? Use Peak ratio (PKR)[1] Temporal consistency : ct Consistent shifts in time Penalize sudden changes 𝑐 𝑖,𝑡 = 𝑐 𝑃𝐾𝑅 ∗ 𝑐 𝑡 Time Shift C(t) = 𝑐 1,𝑡 ⋯ 0 ⋮ ⋱ ⋮ 0 ⋯ 𝑐 𝑛,𝑡 X. Hu and P. Mordohai. A quantitative evaluation of confidence measures for stereo vision, PAMI 2012
25
Motion Estimation Stereo in time Stereo in space
Cluster Spatial Stereo Temporal Estimation Framework Motion Estimation 𝑿 𝑐𝑎𝑚𝑖 𝑡+1 = ∆x y+∆𝑦 d+∆𝑑 1 𝑇 Stereo in time 𝑿 𝑐𝑎𝑚𝑖 𝑡+1 = 𝑽𝒊 𝑴𝑐𝑙𝑢𝑠𝑡𝑒𝑟 𝑉 𝑖 − 𝑿 𝑐𝑎𝑚𝑖 𝑡 𝑿 𝑐𝑎𝑚𝑖 𝑡 = 0 y d 1 𝑇 Stereo in space
26
Weighted Linear System
Cluster Spatial Stereo Temporal Estimation Framework Weighted Linear System 𝑿 𝑐𝑎𝑚𝑖 𝑡+1 =𝑽𝒊 𝑴𝒄𝒍𝒖𝒔𝒕𝒆𝒓 𝑽𝒊 −1 𝑿 𝑐𝑎𝑚𝑖 𝑡 One equation from each camera Use more cameras for robustness 𝑪 𝑨 𝜹 𝑻 𝒙 𝜹 𝑻 𝒚 𝜹 𝑻 𝒛 𝜽 𝒙 𝜽 𝒚 𝜽 𝒛 = 𝑪 𝑩
27
Approach Summary Our approach Spatial Stereo Weighted Least Squares
High frequency 6DoF tracking Stream of rows Temporal Stereo Confidence Scores Input Output
28
Simulator Developed in OpenGL+Qt and Unity3D
Support for Hi-Ball tracker data Motion Blur
29
Experiments Camera specs:
640 × 480 pixels FoV = 60° Stereo pairs = 10 120 fps Quantify errors in terms of display pixel errors Display specs : HTC Vive 120° FoV 1080 × 1200 pixels per eye Point 1m in front 1.Mingsong Duo et al, Exploring high-level plane primitives for indoor 3d reconstruction with a hand-held rgb-d camera, ACCV 2012
30
Display Pixel Error Pixels shown in HMD
31
Experiments : Real motion1 in Synthetic Room
f= 56.7kHz Frame i i+1 i+2 i+3 i+4 … 1. Hi-Ball tracker data
32
Experiments: Synthetic Large Motion
f= 56.7kHz v= 1.4 m/s ω= 120 deg/s
33
Experiments : Synthetic Extreme Motion
f= 56.7kHz v= 1.4 m/s ω= 500 deg/s Sampling not possible
34
Results for Real Imagery
f= 80.4kHz
35
Direct pixel comparison
Conclusion High frequency visual tracker Up to 80 kHz Off-the-shelf cameras Rolling Shutter High Frequency Direct pixel comparison Redundant cameras Motion linearization
36
Contact: akash@cs.unc.edu
Thank you! Contact:
37
Camera tracking : The spectrum
Global Shutter Rolling Shutter as 1-D sensor 1-D sensor 1-D camera1 No distortion Distortion Drift correction No drift correction High cost Cheap High Cost Low fps ≈ 200Hz High fps, 56 kHz High fps, till 87 kHz Higher noise Low noise
Similar presentations
