Omnidirectional Vision Spring 2006 This course is a basic introduction to parts of the field of computer vision. This version of the course covers topics in 'early' or 'low' level vision and parts of 'intermediate' level vision. It assumes no background in computer vision, a minimal background in Artificial Intelligence and only basic concepts in calculus, linear algebra, and probability theory. Lecture 3 - Part 2 Omnidirectional Cameras Zhang Aiwu

Lecture Outline Applications Robot navigation, Surveillance, Smart rooms Video-conferencing/ Tele-presence Multimedia/Visualization GIS Page of Omnidirectional Vision (Many universities and companies….) http://www.cis.upenn.edu/~kostas/omni.html Design Requirements 360 degree FOV, or semi-sphere or full sphere in one snapshot Single effective viewpoint Image Resolutions – one or more cameras? Image Sharpness – optics as well as geometry Several Important Designs Catadioptric imaging : mirror (reflection) + lens ( refraction) Mirrors: Planar, Conic, Spherical, Hyperboloidal, Ellipsoidal, Paraboloidal Systematic design ( S. Nayar’s group) Calibrations Harder or simpler?

Which one? Which one? From the Page of Omnidirectional Vision http://www.cis.upenn.edu/~kostas/omni.html

(Poly-)Dioptric solutions One to two fish-eye cameras or many synchornized cameras Homebrewed polydioptric cameras are cheaper, but require calibrating and synchronizing; commercial designs tend to be expensive

Catadioptric solutions Usually single camera combined with convex mirror Pros: - Single image Cons: - Blindspot - Low resolution

Sensor Design Catadioptric imaging : mirror (reflection) + lens ( refraction) Theory of Catadioptric Image Formation ( S. Nayar’s group) "A Theory of Single-Viewpoint Catadioptric Image Formation" , Simon Baker and Shree K. Nayar ,International Journal of Computer Vision, 1999. Mirrors Planar Conic, Spherical Hyperboloidal, Ellipsoidal Paraboloidal Cameras (Lens) Perspective (pinhole) or orthogonal (tele-centric lens) projection One or more? Implementations Compactness - size, support, and installation Optics – Image sharpness, reflection, etc.

Planar Mirror Panoramic camera system using a pyramid with four (or more) planar mirrors and four (or more) cameras (Nalwa96) has a single effective viewpoint A very important concept in perspective geometry for computer vision and computer graphics Mirror pyramid 6 cameras 4 camera design and 6 camera prototype: FullView - Lucent Technology http://www.fullview.com/

Planar Mirror Panoramic camera system using a pyramid with four (or more) planar mirrors and four (or more) cameras (Nalwa96) has a single effective viewpoint A very important concept in perspective geometry for computer vision and computer graphics Geometry of 4 camera approach: four separate cameras in 4 viewpoints can generate images with a single effective viewpoint

Planar Mirror Approach A single effective viewpoint More than one cameras High image resolution A very important concept in perspective geometry for computer vision and computer graphics

Planar Mirror Approach A single effective viewpoint More than one cameras High image resolution A very important concept in perspective geometry for computer vision and computer graphics

Conic Mirror Viewpoints on a circle semispherical view except occlusion Perspective projection in each direction Robot Navigation (Yagi90, Zhu96/98) viewpoint pinhole A very important concept in perspective geometry for computer vision and computer graphics

Viewpoints on a spherical-like surface Spherical Mirror Viewpoints on a spherical-like surface Easy to construct (Hong91 -UMass ) Intersection of incoming rays are along this line Locus of viewpoints A very important concept in perspective geometry for computer vision and computer graphics

Hyperboloidal Mirror Single Viewpoint if the pinhole of the real camera and the virtual viewpoint are located at the two loci of the hyperboloid Semi-spherical view except the self occlusion pinhole P1 viewpoint P2 Rotation of the hyperbolic curve generates a hyperboloid A very important concept in perspective geometry for computer vision and computer graphics

Hyperboloidal Mirror http://www.accowle.com/english/ ACCOWLE Co., LTD, A Spin-off at Kyoto University http://www.accowle.com/english/  Spherical Mirror Hyperbolic Mirror A very important concept in perspective geometry for computer vision and computer graphics Image: High res. in the top

Ellipsoidal Mirror Single Viewpoint if the pinhole of the real camera and the virtual viewpoint are located at the two loci of the ellipsoid Semi-spherical view except the self occlusion pinhole viewpoint P1 P2 A very important concept in perspective geometry for computer vision and computer graphics

Panoramic Annular Lens - geometric mathematical model for image transform & calibration p p1 pinhole P1 P B O C Ellipsoidal mirror Hyperboloidal mirror panoramic annular lens (PAL) - invented by Pal Greguss * 40 mm in diameter, C-mount * view: H: 360, V: -15 ~ +20 * single view point (O)

Panoramic Annular Lens panoramic annular lens (PAL) - invented by P. Greguss * 40 mm in diameter, C-mount * view: H: 360, V: -15 ~ +20 single view point (O) C-Mount to CCD Cameras Image: High res. In the bottom

Cylindrical panoramic un-warping Two Steps: (1). Center determination (2) Distortion rectification 2-order polynomial approximation

Tele-lens - orthographic projection is used Paraboloidal Mirror Semi-spherical view except the self occlusion Single Viewpoint at the locus of the paraboloid, if Tele-lens - orthographic projection is used Mapping between image, mirror and the world invariant to translation of the mirror. This greatly simplifies calibration and the computation of perspective images from paraboloidal images A very important concept in perspective geometry for computer vision and computer graphics P1 viewpoint tele-lens P2

Paraboloidal Mirror Remote Reality – A Spin-off at Columbia University http://www.remotereality.com/ A very important concept in perspective geometry for computer vision and computer graphics Camcorder Web Camera Back to Back : Full Spherical View

Paraboloidal Mirror Remote Reality – A Spin-off at Columbia University http://www.remotereality.com/ A very important concept in perspective geometry for computer vision and computer graphics

Confused? Confused? Confused? Confused? Confused? Q: What kind of sensor should one use? A: Depends on your application. 1. If you are primarily concerned with: – resolution – surveillance (coverage) and can afford the bandwidth & expense, you might stick with polydioptric solutions 2. If you are concerned with – bandwidth –servoing, SFM investigate catadioptric or single wide FOV dioptric solutions

Application

Application

Application

Application

Application

用GPS和全景相机 日本开发出绘城市地图新技术

Image Properties of Paraboloid System (Assuming aspect ratio = 1) The Image of a Line is a circular arc if the line is not parallel to the optical axis Is projected on a (radial) line otherwise Dual Vanishing Points There are two VPs for each set of parallel lines, which are the intersections of the corresponding circles Collinear Centers The center of the circles for a set of parallel lines are collinear Vanishing Circle The vanishing points of lines with coplanar directions* lie on a circle ( all the lines parallel to a common plane)

Image Properties of Paraboloid System (with aspect ratio) The Image Center Is on the (“vanishing”) line connecting the dual vanishing points of each set of parallel lines Can be determined by two sets of parallel lines Projection of a Line with unknown aspect ratio Is an elliptical arc in the general case The Aspect Ratio Is determined by the ratio of the lone-short axes of the ellipse corresponding to a line Intrinsic Calibration Estimate aspect ratio by the ratio of ellipse Estimate the image center by the intersection of vanishing lines of two sets of parallel lines in 3-D space

Calibration of Paraboloid System The Image Center Is on the (“vanishing”) line connecting the dual vanishing points of each set of parallel lines Can be determined by two sets of parallel lines

Calibration of Paraboloid System The Image Center Yellow “vanishing” line of horizontal set of parallel lines Pink “vanishing” line of vertical set of parallel lines The Vanishing Circle (Red dotted) The vanishing points of lines with coplanar directions ( on a plane in this example) Projected to the plane of the calibration pattern

Next Turn in your projects and schedule meetings with me END