1. Visual 3D Modeling using Cameras and Camera Networks
Marc Pollefeys
University of North Carolina at Chapel Hill

2. Visual 3D Modeling using Cameras and Camera Networks
Talk outline
Introduction
Visual 3D modeling with a hand-held camera
Acquisition of camera motion
Acquisition of scene structure
Constructing visual models
Camera Networks
Camera Network Calibration
Camera Network Synchronization
Towards Active Camera Networks…
Conclusion
Visual 3D Modeling using Cameras and Camera Networks

3. Visual 3D Modeling using Cameras and Camera Networks
What can be achieved?
Can we get 3D models from images?
How much do we need to know about the camera?
Can we freely move around? Hand-held?
Do we need to keep parameters fixed? Zoom?
What about auto-exposure?
What about camera networks?
Can we provide more flexible systems? Avoid calibration?
What about using IP-based PTZ cameras? Hand-held camcorders?
Unsynchronized or even asynchronous?
Visual 3D Modeling using Cameras and Camera Networks

4. Visual 3D Modeling using Cameras and Camera Networks
Talk outline
Introduction
Visual 3D modeling with a hand-held camera
Acquisition of camera motion
Acquisition of scene structure
Constructing visual models
Camera Networks
Camera Network Calibration
Camera Network Synchronization
Towards Active Camera Networks…
Conclusion
Visual 3D Modeling using Cameras and Camera Networks

5. Visual 3D Modeling using Cameras and Camera Networks
(Pollefeys et al. ’98)
Visual 3D Modeling using Cameras and Camera Networks

6. Visual 3D Modeling using Cameras and Camera Networks
(Pollefeys et al. ’04)
Video Key-frame selection
More efficient RANSAC
Fully projective
Improved self-calibration
Deal with dominant planes
Bundle adjustment
Polar stereo rectification
Deal with radial distortion
Faster stereo algorithm
Deal with specularities
Volumetric 3D integration
Deal with Auto-Exposure
Image-based rendering
Visual 3D Modeling using Cameras and Camera Networks

7. Feature tracking/matching
Shape-from-Photographs: match Harris corners
Shape-from-Video: track KLT features
Problem: insufficient motion between consecutive video-frames to compute epipolar geometry accurately and use it effectively as an outlier filter
Visual 3D Modeling using Cameras and Camera Networks

8. Visual 3D Modeling using Cameras and Camera Networks
Key-frame selection
Select key-frame when F yields a better model than H
Use Robust Geometric Information Criterion
Given view i as a key-frame, pick view j as next key-frame for first view where GRIC(Fij)>GRIC(Hij) (or a few views later)
(Torr ’98)
bad fit penalty
model complexity
H-GRIC
F-GRIC
(Pollefeys et al.’02)
Visual 3D Modeling using Cameras and Camera Networks

9. Epipolar geometry
Underlying structure in set of matches for rigid scenes
Computable from corresponding points
Simplifies matching
Allows to detect wrong matches
Related to calibration C1 C2 l2 P l1 e1 e2
Fundamental matrix (3x3 rank 2 matrix)
Visual 3D Modeling using Cameras and Camera Networks

10. Epipolar geometry computation: robust estimation (RANSAC)
Step 1. Extract features
Step 2. Compute a set of potential matches
Step 3. do
Step 3.1 select minimal sample (i.e. 7 matches)
Step 3.2 compute solution(s) for F
Step 3.3 count inliers, if not promising stop
until (#inliers,#samples)<95%
(generate hypothesis)
(verify hypothesis)
Step 4. Compute F based on all inliers
Step 5. Look for additional matches
Step 6. Refine F based on all correct matches
#inliers
90%
80%
70%
60%
50%
#samples 5 13 35
106
382
Visual 3D Modeling using Cameras and Camera Networks

11. Epipolar geometry computation
geometric relations between two views is fully described by recovered 3x3 matrix F
Visual 3D Modeling using Cameras and Camera Networks

12. Sequential Structure and Motion Computation
Initialize Motion
(P1,P2 compatibel with F)
Initialize Structure
(minimize reprojection error)
Extend motion
(compute pose through matches
seen in 2 or more previous views)
Extend structure
(Initialize new structure,
refine existing structure)
Visual 3D Modeling using Cameras and Camera Networks

13. Dealing with dominant planar scenes
(Pollefeys et al., ECCV‘02)
USaM fails when common features are all in a plane
Solution: part 1 Model selection to detect problem
Visual 3D Modeling using Cameras and Camera Networks

14. Dealing with dominant planar scenes
(Pollefeys et al., ECCV‘02)
USaM fails when common features are all in a plane
Solution: part 2 Delay ambiguous computations
until after self-calibration
(couple self-calibration over all 3D parts)
Visual 3D Modeling using Cameras and Camera Networks

15. Refine Structure and Motion
Use projective bundle adjustment
Sparse bundle allows very efficient computation (2 levels)
Take radial distortion into account (1 or 2 parameters)
Visual 3D Modeling using Cameras and Camera Networks

16. Self-calibration using absolute conic
(Faugeras ECCV’92; Triggs CVPR’97;
Pollefeys et al. ICCV’98; etc.)
Euclidean projection matrix:
some constraints, e.g. constant, no skew,... * *
projection
constraints
Absolute conic projection:
Translate constraints on K
through projection equation
to constraints on *
Upgrade from projective to metric
Transform structure and motion
so that *  diag(1,1,1,0)
Visual 3D Modeling using Cameras and Camera Networks

17. Practical linear self-calibration
(Pollefeys et al., ECCV‘02)
Don’t treat all constraints equal
after normalization!
(relatively accurate for most cameras)
(only rough aproximation,
but still usefull to avoid
degenerate configurations)
when fixating point at image-center not only
absolute quadric diag(1,1,1,0) satisfies ICCV98 eqs.,
but also diag(1,1,1,a), i.e. real or imaginary spheres!
Visual 3D Modeling using Cameras and Camera Networks

18. Refine Metric Structure and Motion
Use metric bundle adjustment
Use Euclidean parameterization for projection matrices
Same sparseness advantages, also use radial distortion
Visual 3D Modeling using Cameras and Camera Networks

19. Mixing real and virtual elements in video
Virtual reconstruction of ancient fountain
Preview fragment of sagalassos TV documentary
Similar to 2D3‘s Boujou and RealViz‘ MatchMover
Visual 3D Modeling using Cameras and Camera Networks

20. Intermezzo: Auto-calibration of Multi-Projector System
(Raij and Pollefeys, submitted)
hard because screens are planar, but still possible
Visual 3D Modeling using Cameras and Camera Networks

21. Visual 3D Modeling using Cameras and Camera Networks

22. Visual 3D Modeling using Cameras and Camera Networks
Stereo rectification
Resample image to simplify matching process
Visual 3D Modeling using Cameras and Camera Networks

23. Stereo rectification Resample image to simplify matching process
Also take into account radial distortion!
Visual 3D Modeling using Cameras and Camera Networks

24. Polar stereo rectification
(Pollefeys et al. ICCV’99)
Polar reparametrization of images around epipoles
Does not work with standard
Homography-based approaches
Visual 3D Modeling using Cameras and Camera Networks

25. General iso-disparity surfaces
(Pollefeys and Sinha, ECCV’04)
Example: polar rectification preserves disp.
Application: Active vision
Also interesting relation to human horopter
Visual 3D Modeling using Cameras and Camera Networks

26. Stereo matching Constraints Similarity measure epipolar (SSD or NCC)
ordering
uniqueness
disparity limit
disparity gradient limit
Trade-off
Matching cost
Discontinuities
Similarity measure
(SSD or NCC)
Optimal path
(dynamic programming )
(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97;
Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
Visual 3D Modeling using Cameras and Camera Networks

27. Hierarchical stereo matching
Allows faster computation
Deals with large disparity ranges
Downsampling
(Gaussian pyramid)
Disparity propagation
Visual 3D Modeling using Cameras and Camera Networks

28. Visual 3D Modeling using Cameras and Camera Networks
Disparity map
image I(x,y)
Disparity map D(x,y)
image I´(x´,y´)
(x´,y´)=(x+D(x,y),y)
Visual 3D Modeling using Cameras and Camera Networks

29. Example: reconstruct image from neighbors
Visual 3D Modeling using Cameras and Camera Networks

30. Multi-view depth fusion
(Koch, Pollefeys and Van Gool. ECCV‘98)
Compute depth for every pixel of reference image
Triangulation
Use multiple views
Up- and down sequence
Use Kalman filter
Also allows to compute robust texture
Visual 3D Modeling using Cameras and Camera Networks

31. Real-time stereo on GPU
(Yang and Pollefeys, CVPR2003)
Plane-sweep stereo
Computes Sum-of-Square-Differences (use pixelshader)
Hardware mip-map generation for aggregation over window
Trade-off between small and large support window
150M disparity hypothesis/sec (Radeon9700pro)
e.g. 512x512x20disparities at 30Hz
(Demo GeForce4)
GPU is great for vision too!
Visual 3D Modeling using Cameras and Camera Networks

32. Dealing with specular highlights
(Yang, Pollefeys and Welch, ICCV’03)
Extend photo-consistency model to include highlights
Visual 3D Modeling using Cameras and Camera Networks

33. Visual 3D Modeling using Cameras and Camera Networks

34. Visual 3D Modeling using Cameras and Camera Networks
3D surface model
Depth image
Triangle mesh
Texture image
Textured 3D
Wireframe model
Visual 3D Modeling using Cameras and Camera Networks

35. Volumetric 3D integration
(Curless and Levoy, Siggraph´96)
Multiple depth images
Volumetric integration
Texture integration
patchwork texture map
Visual 3D Modeling using Cameras and Camera Networks

36. Dealing with auto-exposure
(Kim and Pollefeys, submitted)
Estimate cameras radiometric response curve,
exposure and white balance changes
Extends prior HDR work at Columbia, CMU, etc.
to moving camera
brightness
transfer curve
robust estimate using DP
auto-exposure
fixed-exposure
response curve model
Visual 3D Modeling using Cameras and Camera Networks

37. Dealing with auto-exposure
(Kim and Pollefeys, submitted)
Applications:
Photometric alignment of textures (or HDR textures)
HDR video
Visual 3D Modeling using Cameras and Camera Networks

38. Part of Jain temple Recorded during post-ICCV tourist trip in India
(Nikon F50; Scanned)
Visual 3D Modeling using Cameras and Camera Networks

39. Example: DV video  3D model
accuracy ~1/500 from DV video (i.e. 140kb jpegs 576x720)
Visual 3D Modeling using Cameras and Camera Networks

40. Unstructured lightfield rendering
(Heigl et al.’99)
demo
Visual 3D Modeling using Cameras and Camera Networks

41. Visual 3D Modeling using Cameras and Camera Networks
Talk outline
Introduction
Visual 3D modeling with a hand-held camera
Acquisition of camera motion
Acquisition of scene structure
Constructing visual models
Camera Networks
Camera Network Calibration
Camera Network Synchronization
towards active camera networks…
Conclusion
Visual 3D Modeling using Cameras and Camera Networks

42. Visual 3D Modeling using Cameras and Camera Networks
CMU’s Dome, 3D Room, etc.
MIT’s Visual Hull
Maryland’s Keck lab, ETHZ’s BLUE-C and more
Recently, Shape-from-Silhouette/Visual-Hull systems have been very popular
Visual 3D Modeling using Cameras and Camera Networks

43. Visual 3D Modeling using Cameras and Camera Networks
Offline Calibration Procedure
Special Calibration Data
Planar Pattern
moving LED
Requires physical access to environment
Active Camera Networks
How do we maintain calibration ?
Visual 3D Modeling using Cameras and Camera Networks

44. Visual 3D Modeling using Cameras and Camera Networks
An example
P. Sand, L. McMillan, and J. Popovic.
Continuous Capture of Skin Deformation.
ACM Transactions on Graphics 22, 3, , 2003.
4 NTSC videos recorded by 4 computers for 4 minutes
Manually synchronized and calibrated using MoCap system
Visual 3D Modeling using Cameras and Camera Networks

45. Can we do without explicit calibration?
Feature-based?
Hard to match features between very different views
Not many features on foreground
Background often doesn’t overlap much between views
Silhouette-based?
Necessary for visual-hull anyway
But approach is not obvious
Visual 3D Modeling using Cameras and Camera Networks

46. Multiple View Geometry of Silhouettesx1 x2
x’1
x’2
Frontier Points
Epipolar Tangents
Points on Silhouettes in 2 views do not correspond in general except for projected Frontier Points
Always at least 2 extremal frontier points per silhouette
In general, correspondence only over two views
Visual 3D Modeling using Cameras and Camera Networks

47. Calibration from Silhouettes: prior work
Epipolar Geometry from Silhouettes
Porril and Pollard, ’91
Astrom, Cipolla and Giblin, ’96
Structure-and-motion from Silhouettes
Joshi, Ahuja and Ponce’95 (trinocular rig/rigid object)
Vijayakumar, Kriegman and Ponce’96 (orthographic)
Wong and Cipolla’01 (circular motion, at least to start)
Yezzi and Soatto’03 (only refinement)
None really applicable to calibrate visual hull system
Visual 3D Modeling using Cameras and Camera Networks

48. Camera Network Calibration from Silhouettes
(Sinha, Pollefeys and McMillan, submitted)
7 or more corresponding frontier points needed to compute epipolar geometry for general motion
Hard to find on single silhouette and possibly occluded
However, Visual Hull systems record many silhouettes!
Visual 3D Modeling using Cameras and Camera Networks

49. Camera Network Calibration from Silhouettes
If we know the epipoles, it is simple
Draw 3 outer epipolar tangents (from two silhouettes)
Compute corresponding line homography H-T (not unique)
Epipolar Geometry F=[e]xH
Visual 3D Modeling using Cameras and Camera Networks

50. Let’s just sample: RANSAC
Repeat
Generate random hypothesis for epipoles
Compute epipolar geometry
Verify hypothesis and count inliers
until satisfying hypothesis
Refine hypothesis
minimize symmetric transfer error of frontier points
include more inliers
Until error and inliers stable
(use conservative threshold, e.g. 5 pixels,
but abort early if not promising)
(use strict threshold, e.g. 1 pixels)
We’ll need an efficient representation
as we are likely to have to do many trials!
Visual 3D Modeling using Cameras and Camera Networks

51. A Compact Representation for Silhouettes Tangent Envelopes
Convex Hull of Silhouette.
Tangency Points
for a discrete set of angles.
Approx. 500 bytes/frame. Hence a whole video sequences easily fits in memory.
Tangency Computations are efficient.
Visual 3D Modeling using Cameras and Camera Networks

52. Epipole Hypothesis and Computing H
Visual 3D Modeling using Cameras and Camera Networks

53. Visual 3D Modeling using Cameras and Camera Networks
Model Verification
Visual 3D Modeling using Cameras and Camera Networks

54. Visual 3D Modeling using Cameras and Camera Networks
Remarks
RANSAC allows efficient exploration of 4D parameter space (i.e. epipole pair) while being robust to imperfect silhouettes
Select key-frames to avoid having too many identical constraints (when silhouette is static)
Visual 3D Modeling using Cameras and Camera Networks

55. Reprojection Error and Epipole Hypothesis Distribution
40 best hypothesis out of 30000
Residual Distribution
Hypotheses along y-axis
Sorted Residuals along x-axis.
Pixel Error along z-axis.
Typically, 1/5000 samples converges to global minima after
non-linear refinement (corresponds to 15 sec. computation time)
Visual 3D Modeling using Cameras and Camera Networks

56. Computed Fundamental Matrices
Visual 3D Modeling using Cameras and Camera Networks

57. Computed Fundamental Matrices
F computed directly (black epipolar lines)
F after consistent 3D reconstruction (color)
Visual 3D Modeling using Cameras and Camera Networks

58. Computed Fundamental Matrices
F computed directly (black epipolar lines)
F after consistent 3D reconstruction (color)
Visual 3D Modeling using Cameras and Camera Networks

59. From epipolar geometry to full calibration
Not trivial because only matches between two views
Approach similar to Levi et al. CVPR’03, but practical
Key step is to solve for camera triplet
Assemble complete camera network
projective bundle, self-calibration, metric bundle
(v is 4-vector )
(also linear in v)
Choose P3 corresponding to closest
Visual 3D Modeling using Cameras and Camera Networks

60. Visual 3D Modeling using Cameras and Camera Networks
Experiment
4 video sequences
at 30 fps.
All F Matrices
computed
from silhouettes
Full calibration
Visual 3D Modeling using Cameras and Camera Networks

61. Metric Cameras and Visual-Hull Reconstruction from 4 views
Final calibration quality comparable to explicit calibration procedure
Visual 3D Modeling using Cameras and Camera Networks

62. What if the videos are unsynchronized?
For videos recorded at a constant framerate, same contraints are valid, up to some extra unknown temporal offsets
Visual 3D Modeling using Cameras and Camera Networks

63. Visual 3D Modeling using Cameras and Camera Networks
Synchronization and calibration from silhouettes (Sinha and Pollefeys, submitted)
Add a random temporal offset to RANSAC hypothesis generation, sample more
Use multi-resolution approach:
Key-frames with slow motion, rough synchronization
Add key-frames with faster motion, refine synchronization
Visual 3D Modeling using Cameras and Camera Networks

64. Synchronization experiment
Total temporal offset search range [-500,+500] (i.e. ±15s)
Unique peaks for correct offsets
Possibility for sub-frame synchronization
Visual 3D Modeling using Cameras and Camera Networks

65. Synchronize camera network
Consider oriented graph with offsets as branch value
For consistency loops should add up to zero
MLE by minimizing
in frames (=1/30s) +3 -5 +8 +6 +2
ground truth
Visual 3D Modeling using Cameras and Camera Networks

66. Towards active camera networks
Provide much more flexibility by making use of pan-tilt-zoom range, networked cameras
(maintaining) calibration is a challenge
up to 3Gpix!
Visual 3D Modeling using Cameras and Camera Networks

67. Calibration of PTZ cameras
similar to
Collins and Tsin ’99,
but with varying
radial distortion
Visual 3D Modeling using Cameras and Camera Networks

68. Visual 3D Modeling using Cameras and Camera Networks

69. Visual 3D Modeling using Cameras and Camera Networks
Conclusion
3D models from video, more flexibility, more general
Camera networks synchronization and calibration, just from silhouettes, great for visual-hull systems
Future plans
Deal with sub-frame offset for VH reconstruction
Extend to active camera network (PTZ cameras)
Extend to asynchronous video streams (IP cameras)
view01
Visual 3D Modeling using Cameras and Camera Networks

70. Visual 3D Modeling using Cameras and Camera Networks
Acknowledgment
NSF Career, NSF ITR on 3D-TV, DARPA seedling, Link foundation
EU ACTS VANGUARD, ITEA BEYOND, EU IST MURALE, FWO-Vlaanderen
Sudipta Sinha, Ruigang Yang, Seon Joo Kim, Andrew Raij,
Greg Welch, Leonard McMillan (UNC)
Maarten Vergauwen, Frank Verbiest, Kurt Cornelis, Jan Tops,
Luc Van Gool (KULeuven), Reinhard Koch (UKiel), Benno Heigl
Visual 3D Modeling using Cameras and Camera Networks