1. Cameras and Projectors
Hao Jiang
Computer Science Department
Boston College
Oct. 2, 2007
CS335 Principles of Multimedia Systems

2. Cameras and Projectors
Cameras and projectors have been intensively used in many different multimedia applications.
It is important to understand techniques to manipulate these devices.
We will study basic methods about
calibration,
image warping and blending,
and other issues in building a camera/projector system.
CS335 Principles of Multimedia Systems

3. Example Application: Projector Geometry Distortion Compensation
Automatically correcting projector geometry distortion using
a camera and projector system.
CS335 Principles of Multimedia Systems

4. Camera / Projector Geometry Model
Camera and projector can be modeled as a pinhole imaging system.
Object Point
Optical Center
Focal length
Optical
axis
Image
point
Image Plane
CS335 Principles of Multimedia Systems

5. CS335 Principles of Multimedia Systems
Camera Geometry Model
A camera projects a 3D point onto a 2D point in a image. y x
3D point (X, Y, Z)
in camera’s frame is
(X’,Y’,Z’)
(Ox, Oy) f
x = (f X’/Z’)/dx + Ox = fx X’/Z’ + Ox
y = (f Y’/Z’)/dy + Oy = fy Y’/Z’ + Oy
dx and dy are width and height of an image
pixel
CS335 Principles of Multimedia Systems

6. Homogenous Coordinate
The homogenous coordinate of a 3D point (x,y,z) is
(X,Y,Z,W) where X/W = X, Y/W=Y and Z/W=Z.
The homogenous coordinate of a 2D point (x,y) is (X,Y,W) where X/W = x, Y/W=Y.
We can convert a 3D point from one frame to another, by simply T*P, where T is a 4x4 matrix and P is the homogeneous coordinate of the 3D point.
CS335 Principles of Multimedia Systems

7. CS335 Principles of Multimedia Systems
Camera Matrix
Using homogenous coordinate, the camera projection can be represented as X
a11 a12 a13 a14
a21 a22 a23 a24
a31 a32 a33 a34 x Y
P =
A3x4
= A P
p = ~ y Z w W
Where
A3x4
is a 3 rows and 4 columns matrix, called
camera matrix.
CS335 Principles of Multimedia Systems

8. Projection from a Plane to Another PlaneZ X Y P p
Since
p = A P
We have O x y w
a11 a12 a14
a21 a22 a24
a31 a32 a34 X Y 1
p = ~ =
H p’
H is defines a Homography.
CS335 Principles of Multimedia Systems

9. CS335 Principles of Multimedia Systems
The Projector Model
Projector shares similar model with a camera.
The only difference is that a projector projects a flat object that emits lights based on a computer image onto an image plane, the screen.
Based on the previous analysis about planar object projection, the image from the computer and the one projected on the screen are related by a homography.
CS335 Principles of Multimedia Systems

10. The Camera-Projector System
H2*r
H1*p
Image on the screen q p
H2*H1*p
Camera image
Image sent to projector
CS335 Principles of Multimedia Systems

11. CS335 Principles of Multimedia Systems
Calibration
We would like to compute H1 and H2.
We project marker points on the screen and form equations
a11Xn + a12Yn + a14
a31Xn + a32Yn + a34
a21Xn + a22Yn + a24
= xn
n = 1 .. N
= yn
a34=1
CS335 Principles of Multimedia Systems

12. CS335 Principles of Multimedia Systems
Image Adjustment (I)
Image on the screen
Assume that
the viewer is
close to the camera.
H2*H1*p
H1*p q p
The desired
Image.
Pre-warping
Camera image
Projector image
Each p is projected to the camera image and the color is determined
by color interpolation in the desired image.
CS335 Principles of Multimedia Systems

13. CS335 Principles of Multimedia Systems
Image Adjustment (II)
Image on the screen
The desired image
Estimate H2 using
screen corners projections.
H2*H1*p
H1*p
Pre-warping
Camera image
Project image
Pre-warp image based on the desired image on the screen.
CS335 Principles of Multimedia Systems

14. Projection onto Arbitrary Surfaces
We have learned how to deal with projector distortion using a planar screen.
We can extend the method into other types of surfaces, such as cylinder or spheres.
We need a relative dense mesh grid to capture the local deformation model.
CS335 Principles of Multimedia Systems

15. Titled Large Screen Display
CS335 Principles of Multimedia Systems

16. Titled Large Screen Display
LCD or CRT are still not easy to be made as large as a wall.
Titling multiple projector images into a large screen display is flexible and relatively cheap.
The shortcoming is we need to align the images from different projectors in both geometry and color.
Manually adjusting the projectors is a tedious task.
Camera project system can be used to solve the problem.
CS335 Principles of Multimedia Systems

17. CS335 Principles of Multimedia Systems
Titled Images 2 3 4
Calibration
Patterns 1
(Projector to
Camera
Homography
can be computed
based on these
patterns) 6 7 8 5 9 11 10 12
Projector images
The camera view
CS335 Principles of Multimedia Systems

18. Large Screen Projection
projector image 3 p
pcamera = Hqprojector 2 3 4
The
big
image q 1 6 7 8 5 9 11 10 12
Projector images
The camera view
CS335 Principles of Multimedia Systems

19. Edge and Color Alignment
Colors of different projectors are usually different. A color calibration and adjustment procedure is needed.
Blending
Color(p) = a Color(2,p) + (1-a)Color(3,p) 2 3 p
a is determined by the dominance of 2 or 3.
CS335 Principles of Multimedia Systems

20. CS335 Principles of Multimedia Systems
Blending
The size of the blending region has to be carefully selected
If the region is too small, large scale structures will show abrupt changes. If it is too big, small structures (edges, dots) will overlap in a big region and therefore will result in blur (ghost) images. a
1-a
Blending region
CS335 Principles of Multimedia Systems

21. CS335 Principles of Multimedia Systems
Multi-band Blending
Multi-band blending can be used to address the problem.
In multi-band blending,
images are filtered into different bands.
The mask is also low-pass filtered to generate mask for each band images.
Images are blending in each channel.
The blended images are summed up to get the final result.
CS335 Principles of Multimedia Systems

22. CS335 Principles of Multimedia Systems
Image Pyramids
The Gaussian Pyramid
CS335 Principles of Multimedia Systems

23. CS335 Principles of Multimedia Systems
Image Pyramids
The Laplacian Pyramid
CS335 Principles of Multimedia Systems

24. CS335 Principles of Multimedia Systems
Image Pyramids
The Gaussian pyramid of the mask
CS335 Principles of Multimedia Systems

25. CS335 Principles of Multimedia Systems
Multi-band Blending
Multi-band blending
CS335 Principles of Multimedia Systems

26. CS335 Principles of Multimedia Systems
Multi-band Blending
CS335 Principles of Multimedia Systems

27. System Issues of Large Screen Display
Large screen projection needs multiple projectors working simultaneously.
The first structure uses share memory system such as SGI Oynx2, that employs a shared-memory model. A single program can have different threads writing OpenGL primitives into different pipes while reading from a single shared database and synchronizing display update over shared flags.
The second structure is PC cluster, in which each PC handles one projector. This framework is much cheaper but the synchronization is a challenge problem.
CS335 Principles of Multimedia Systems

28. CS335 Principles of Multimedia Systems
Applications
Visualization and Collaboration
IEEE Computer Graphics and Applications, 2000
CS335 Principles of Multimedia Systems

29. CS335 Principles of Multimedia Systems
Immersive Reality
The CAVE (University of Illinois at Chicago)
IEEE Computer Graphics and Applications, 2000
CS335 Principles of Multimedia Systems

30. Projection for Augmented Environment
Anton Treskunov and Jarrell Pair, PROJECTOR-CAMERA SYSTEMS FOR
IMMERSIVE TRAINING, ASC06
CS335 Principles of Multimedia Systems

31. Input Methods for Large Screens
It presents challenges for traditional input methods.
Possible HCI methods include:
Pointing devices, including 3D tracking, passive optical (video) tracking, ultrasonic tracking, mice, and tablet interfaces;
User tracking, for point-of-view rendering or for gaze directed interaction, via optical tracking or electromagnetic tracking;
Handheld devices, providing control interfaces that can be out of band from the display;
Voice commands with audio feedback;
And haptics interfaces.
CS335 Principles of Multimedia Systems

32. Multiple Camera System
Multiple camera system can capture video from different locations simultaneously.
Stereo system has been widely used for inferring the “depth” of objects.
CS335 Principles of Multimedia Systems

33. CS335 Principles of Multimedia Systems
3D reconstruction from Multiple Views
CS335 Principles of Multimedia Systems

34. Application of Multi-camera Imaging
Image based rendering
The Matrix
CS335 Principles of Multimedia Systems

35. CS335 Principles of Multimedia Systems
Cameras in the Matrix
CS335 Principles of Multimedia Systems

36. Camera Calibration Toolboxes
OpenCV
Matlab Calibration Toolbox
Multiple Camera Calibration
CS335 Principles of Multimedia Systems