3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D Workshop (CVPR 2008)

AutoCalibration Algorithms Camera Calibration Moving Camera? Tracking Local Features Boujou, 3D-Equalizer, Matchmover, Voodoo, … yes no Known Structure? Exploit Known Objects no Common in Surveillance

Main Idea 1 Correct Camera Matrix Pedestrian observations are consistent (no height change). 3

Main Idea 1 Incorrect Camera Matrix Pedestrian grows or shrinks. Pedestrians can be used as a measuring stick to calibrate the camera. 4

Main Idea 2 Image Plane Camera Center 3D World P1P1 Camera Matrix (P F ): P F = P 1 Light Source P1P1 P2P2 Shadow Camera Matrix (P S ): P S = P 1 o P 2 5

Main Idea 2 Correct light source position Pedestrian shadow observations are consistent. 6 Analogously, a reflection camera can be defined.

In summary Simultaneously Estimate: 1- Ground Positions (in 3D) 2- Horizon height (in 2D) 3- Light source position (in 3D) 4- Pedestrian height (in world units) 5- Axes scaling (to define the unit of length) X Z Y 7 That are Mutually Consistent and Explain the observations.

Object 3D Bounding Box Single Frame Consistency? Camera Consistency Test HeightGround Position Observation Consistency (Likelihood) Camera matrix (or Shadow Camera) Model (3D prior) 8

3D Priors Voxel V4V4 V6V6 V1V1 V2V2 V3V3 V7V7 V8V8 V9V9 V5V5 Pixel Camera Q2Q2 Q3Q3 Q1Q1 Q4Q4 Voxel V i : - Occupied (v i = 1) with probability p i. - Blocks light if it is occupied. - Independent of other voxels. Problems: - Discretization matters. - Equal contributions voxels ray. Solution: Beer-Lambert law correction (predicts light attenuation in solutions), R 1,1 R 2,1 R 6,1 R 3,1 - measured in [blocking probability / meter]. - Same to traverse 1 big voxel or 2 of half the size. 9 2D Prior3D Prior

3D Priors 10 Whole walking cyclePart of the walking cycle

Graphical Model Observed Pixel Colors in Frame t C1C1 CMCM Voxels (3D Prior) P V1 P V2 P VN Pixel Class (2D Prior) Foreground Q F1 Q FM Q S1 Q SM Shadow Geometry (Projection) Camera Matrix Light Position Ground Position Background Shadow Color Models 11 F1 Likelihood

Trajectory unregularized Scene Parameters F1 Likelihood G1G1 12 F1 Likelihood G2G2 F1 Likelihood G3G3

Trajectory Regularized F1 Likelihood (F2) G1G1 F1 G2G2 G3G3 Prior Acceleration Optimum trajectory and F2 computed in O(N F. N G 3 ) using Dynamic Programming. 13 F2

Search Solution Space Search the solution in the whole 4D parameter space: 1.Horizon Height 2.Y-Axis Scale. 3.Light Theta 4.Light Phi 14 Likelihood Camera Matrix Light Position F2 Optimum trajectory Camera Matrix Light Direction

Results To speed up computation, search first in the lowest resolution. 15 Half Resolution Original Resolution Then, refine in the next higher, and so on. Fast, so the whole space can be searched.

Results 16 Half Resolution Original Resolution Solution superimposed. Shape of the peak defines the types of errors. Estimated Horizon

Results Metrology comparison Measure Ground Truth (m) Estimated (m) P P P P Localization error No Shadows (cm) Shadows (cm) Mean error lower than 2% (relative to the people average height). 17 Shadows are not disturbances, their use improve localization. Estimated Horizon

Conclusions Presented: Novel object model (not limited to people) and probabilistic framework For camera calibration and simple lighting estimation. Using the Foreground and the Shadows. That works in situations where other methods fail. 18

Learning 3D Priors V4V4 V6V6 V1V1 V2V2 V3V3 V7V7 V8V8 V9V9 V5V5 C1C1 C2C2 Method of Moments, yields one Equation per ray: This is the Fan Beam Radon transform. Just solve linear system. Silhouette in frame tAverage 19

3D Priors 20

Search Solution Space x y z 21