Surveillance and Security

Project Industrial site Monitored by two camera’s

Camera Calibration Thomas Flamant

Goal Camera Calibration Reconstruction Camera matrix get_mapped.m (WP3) Calibration

Camera Calibration (Camera resectioning) = the process of finding the true parameters of the camera that produced a given photograph or video Pinhole camera model Camera matrix Represents the camera parameters Calibration

Pinhole camera model Describes the mathematical relationship between the coordinates of a 3D point and its projection onto the image plane of an ideal pinhole camera Calibration

Camera matrix Describes the mapping of a pinhole camera from 3D points in the world to 2D points in an image. 𝑥 𝑝𝑖𝑥𝑒𝑙 𝑦 𝑝𝑖𝑥𝑒𝑙 1 = ??? ∗ ??? ∗ 𝑥 𝑤𝑜𝑟𝑙𝑑 𝑦 𝑤𝑜𝑟𝑙𝑑 𝑧 𝑤𝑜𝑟𝑙𝑑 1 Intrinsic parameters Extrinsic parameters Calibration

Intrinsic Parameters (intrinsic calibration) Used to convert between the coordinates of the image plane and the actual pixel-coordinates Depends on camera hardware Consists of: Focal length 𝑓 𝑥 , 𝑓 𝑦 Camera centre 𝑐 𝑥 , 𝑐 𝑦 Lens distortion ? Barrel Pincushion mustache 𝐴= 𝑓 𝑥 0 𝑐 𝑥 0 𝑓 𝑦 𝑐 𝑦 0 0 1 Calibration

Extrinsic Parameters (extrinsic calibration) Used to describe the relative position and orientation of the camera with regards to a world coordinate system Does not depend on camera hardware Consists of: 3x3 rotation matrix 𝑅 3x1 translation vector 𝑡 𝑅= 𝑟 11 𝑟 12 𝑟 13 𝑟 21 𝑟 22 𝑟 23 𝑟 31 𝑟 32 𝑟 33 𝑡= 𝑡 𝑥 𝑡 𝑦 𝑡 𝑧 Calibration

Camera matrix Describes the mapping of a pinhole camera from 3D points in the world to 2D points in an image. 𝑥 𝑝𝑖𝑥𝑒𝑙 𝑦 𝑝𝑖𝑥𝑒𝑙 1 = ??? ∗ ??? ∗ 𝑥 𝑤𝑜𝑟𝑙𝑑 𝑦 𝑤𝑜𝑟𝑙𝑑 𝑧 𝑤𝑜𝑟𝑙𝑑 1 Intrinsic parameters Extrinsic parameters Calibration

Camera matrix Describes the mapping of a pinhole camera from 3D points in the world to 2D points in an image. 𝑥 𝑝𝑖𝑥𝑒𝑙 𝑦 𝑝𝑖𝑥𝑒𝑙 1 = 𝐴 ∗ 𝑅 | 𝑡 ∗ 𝑥 𝑤𝑜𝑟𝑙𝑑 𝑦 𝑤𝑜𝑟𝑙𝑑 𝑧 𝑤𝑜𝑟𝑙𝑑 1 Intrinsic parameters Extrinsic parameters Calibration

Camera matrix Describes the mapping of a pinhole camera from 3D points in the world to 2D points in an image. 𝑥 𝑝𝑖𝑥𝑒𝑙 𝑦 𝑝𝑖𝑥𝑒𝑙 1 = 𝑓 𝑥 0 𝑐 𝑥 0 𝑓 𝑦 𝑐 𝑦 0 0 1 ∗ 𝑟 11 𝑟 12 𝑟 13 𝑡 1 𝑟 21 𝑟 22 𝑟 23 𝑡 2 𝑟 31 𝑟 32 𝑟 33 𝑡 3 ∗ 𝑥 𝑤𝑜𝑟𝑙𝑑 𝑦 𝑤𝑜𝑟𝑙𝑑 𝑧 𝑤𝑜𝑟𝑙𝑑 1 Intrinsic parameters Extrinsic parameters Calibration

Camera matrix Describes the mapping of a pinhole camera from 3D points in the world to 2D points in an image. 𝑥 𝑝𝑖𝑥𝑒𝑙 𝑦 𝑝𝑖𝑥𝑒𝑙 1 = 𝑓 𝑥 0 𝑐 𝑥 0 𝑓 𝑦 𝑐 𝑦 0 0 1 ∗ 𝑟 11 𝑟 12 𝑟 13 𝑡 1 𝑟 21 𝑟 22 𝑟 23 𝑡 2 𝑟 31 𝑟 32 𝑟 33 𝑡 3 ∗ 𝑥 𝑤𝑜𝑟𝑙𝑑 𝑦 𝑤𝑜𝑟𝑙𝑑 𝑧 𝑤𝑜𝑟𝑙𝑑 1 Intrinsic parameters Extrinsic parameters (Bouguet) (POSIT) Calibration

Camera Calibration Toolbox for Matlab (Bouguet) Input: 20+ images of a planar checkerboard Output: Intrinsic parameters Corner extraction Automatic: RADOCC Camera Calibration Toolbox (CornerFinder.m) Calibration

Camera Calibration Toolbox for Matlab (Bouguet) Input: 30 images of each camera (squares = 28mm) Camera 1: Camera 2: Calibration

Camera Calibration Toolbox for Matlab (Bouguet) Input: 30 images of each camera (squares = 28mm) Camera 1: Camera 2: Calibration

Camera Calibration Toolbox for Matlab (Bouguet) Output: intrinsic parameters of each camera Camera 1: Camera 2: 𝐴 𝑐𝑎𝑚1 = 1458.4 0 686.5 0 1456.9 350.1 0 0 1 𝐴 𝑐𝑎𝑚2 = 2067.7 0 678.8 0 2068.2 340.5 0 0 1 Calibration

POSIT Description: POSIT is a fast iterative algorithm for finding the pose (rotation and translation) of an object or scene with respect to a camera when points of the object are given in some object coordinate system and these points are visible in the camera image and recognizable, so that corresponding image points and object points can be listed in the same order. [rotation, translation] = Posit(imagePoints, objectPoints, focalLength, center) Calibration

POSIT Input: [rotation, translation] = Posit(imagePoints, objectPoints, focalLength, center) Input: nbPts: 4+ noncoplanar feature points of the object imagePoints: matrix of size nbPts x 2 objectPoints: matrix of size nbPts x 3 focalLength: focal length of the camera in pixels Center: row vector with the elements of the image center Camera 2 Calibration

POSIT Input: Output: [rotation, translation] = Posit(imagePoints, objectPoints, focalLength, center) Input: nbPts: 4+ noncoplanar feature points of the object imagePoints: matrix of size nbPts x 2 objectPoints: matrix of size nbPts x 3 focalLength: focal length of the camera in pixels Center: row vector with the elements of the image center Output: rotation: 3 x 3 rotation matrix of scene with respect to camera translation: 3 x 1 translation vector from projection center of camera to FIRST POINT in list of object points Calibration

POSIT Output: extrinsic parameters of each camera Camera 1: Camera 2: 𝑅 𝑐𝑎𝑚1 = 1.2454 −0.3342 −0.0195 −0.0904 −0.4501 −0.6249 0.2001 0.7800 −0.5908 𝑡 𝑐𝑎𝑚1 = 1051 2830 32268 𝑅 𝑐𝑎𝑚2 = 1.3088 0.0228 −0.0423 −0.0242 −0.3105 −0.6971 −0.0290 0.9135 −0.4058 𝑡 𝑐𝑎𝑚2 = 357 3823 26989 Calibration

Camera matrix Describes the mapping of a pinhole camera from 3D points in the world to 2D points in an image. 𝑥 𝑝𝑖𝑥𝑒𝑙 𝑦 𝑝𝑖𝑥𝑒𝑙 1 = 𝑓 𝑥 0 𝑐 𝑥 0 𝑓 𝑦 𝑐 𝑦 0 0 1 ∗ 𝑟 11 𝑟 12 𝑟 13 𝑡 1 𝑟 21 𝑟 22 𝑟 23 𝑡 2 𝑟 31 𝑟 32 𝑟 33 𝑡 3 ∗ 𝑥 𝑤𝑜𝑟𝑙𝑑 𝑦 𝑤𝑜𝑟𝑙𝑑 𝑧 𝑤𝑜𝑟𝑙𝑑 1 Intrinsic parameters Extrinsic parameters (Bouguet) (POSIT) Calibration

Camera matrix Describes the mapping of a pinhole camera from 3D points in the world to 2D points in an image. Camera 1: Camera 2: 𝑥 𝑝𝑖𝑥𝑒𝑙 𝑦 𝑝𝑖𝑥𝑒𝑙 1 = 1458.4 0 686.5 0 1456.9 350.1 0 0 1 ∗ 1.2454 −0.3342 −0.0195 1051 −0.0904 −0.4501 −0.6249 2830 0.2001 0.7800 −0.5908 32268 ∗ 𝑥 𝑤𝑜𝑟𝑙𝑑 𝑦 𝑤𝑜𝑟𝑙𝑑 𝑧 𝑤𝑜𝑟𝑙𝑑 1 𝑥 𝑝𝑖𝑥𝑒𝑙 𝑦 𝑝𝑖𝑥𝑒𝑙 1 = 2067.7 0 678.8 0 2068.2 340.5 0 0 1 ∗ 1.3088 0.0228 −0.0423 357 −0.0242 −0.3105 −0.6971 3823 −0.0290 0.9135 −0.4058 26989 ∗ 𝑥 𝑤𝑜𝑟𝑙𝑑 𝑦 𝑤𝑜𝑟𝑙𝑑 𝑧 𝑤𝑜𝑟𝑙𝑑 1 Calibration

Goal Camera Calibration Reconstruction Camera matrix get_mapped.m (WP3) Calibration

Reconstruction 3D-reconstruction: Triangulation 2D-reconstruction: Calibration matrix Calibration

3D-reconstruction (Triangulation) Epipolar geometry: When two cameras view a 3D scene from two distinct positions, there are a number of geometric relations between the 3D points and their projections onto the 2D images that lead to constraints between the image points. Calibration

3D-reconstruction (Triangulation) The process of determining a point in 3D space given its projections onto two, or more, images. In order to solve this problem it is necessary to know the parameters of the camera projection function from 3D to 2D for the cameras involved, represented by the camera matrices Calibration

2D-reconstruction (Calibration matrix) 3 dimensions ? 𝑥 𝑝𝑖𝑥𝑒𝑙 𝑦 𝑝𝑖𝑥𝑒𝑙 1 = 𝑓 𝑥 0 𝑐 𝑥 0 𝑓 𝑦 𝑐 𝑦 0 0 1 ∗ 𝑟 11 𝑟 12 𝑟 13 𝑡 1 𝑟 21 𝑟 22 𝑟 23 𝑡 2 𝑟 31 𝑟 32 𝑟 33 𝑡 3 ∗ 𝑥 𝑤𝑜𝑟𝑙𝑑 𝑦 𝑤𝑜𝑟𝑙𝑑 𝑧 𝑤𝑜𝑟𝑙𝑑 1 Camera 2 Calibration

2D-reconstruction (Calibration matrix) 2 dimensions ! 𝑥 𝑝𝑖𝑥𝑒𝑙 𝑦 𝑝𝑖𝑥𝑒𝑙 1 = 𝑓 𝑥 0 𝑐 𝑥 0 𝑓 𝑦 𝑐 𝑦 0 0 1 ∗ 𝑟 11 𝑟 12 𝑟 13 𝑡 1 𝑟 21 𝑟 22 𝑟 23 𝑡 2 𝑟 31 𝑟 32 𝑟 33 𝑡 3 ∗ 𝑥 𝑤𝑜𝑟𝑙𝑑 𝑦 𝑤𝑜𝑟𝑙𝑑 𝑧 𝑤𝑜𝑟𝑙𝑑 1 Camera 2 Calibration

2D-reconstruction (Calibration matrix) 2 dimensions ! 𝑥 𝑝𝑖𝑥𝑒𝑙 𝑦 𝑝𝑖𝑥𝑒𝑙 1 = 𝑓 𝑥 0 𝑐 𝑥 0 𝑓 𝑦 𝑐 𝑦 0 0 1 ∗ 𝑟 11 𝑟 12 𝑡 1 𝑟 21 𝑟 22 𝑡 2 𝑟 31 𝑟 32 𝑡 3 ∗ 𝑥 𝑤𝑜𝑟𝑙𝑑 𝑦 𝑤𝑜𝑟𝑙𝑑 1 Camera 2 Calibration

2D-reconstruction (Calibration matrix) 2 dimensions ! 𝑥 𝑤𝑜𝑟𝑙𝑑 𝑦 𝑤𝑜𝑟𝑙𝑑 1 = 𝑖𝑛𝑣𝑒𝑟𝑠 𝑓 𝑥 0 𝑐 𝑥 0 𝑓 𝑦 𝑐 𝑦 0 0 1 ∗ 𝑟 11 𝑟 12 𝑡 1 𝑟 21 𝑟 22 𝑡 2 𝑟 31 𝑟 32 𝑡 3 ∗ 𝑥 𝑝𝑖𝑥𝑒𝑙 𝑦 𝑝𝑖𝑥𝑒𝑙 1 Camera 2 Calibration

2D-reconstruction (Calibration matrix) 2 dimensions ! Function get_mapped.m  WP3 function [ x_world , y_world ] = get_mapped(camera_nr, x_pixel, y_pixel) 𝑥 𝑤𝑜𝑟𝑙𝑑 𝑦 𝑤𝑜𝑟𝑙𝑑 1 = 𝑖𝑛𝑣𝑒𝑟𝑠 𝑓 𝑥 0 𝑐 𝑥 0 𝑓 𝑦 𝑐 𝑦 0 0 1 ∗ 𝑟 11 𝑟 12 𝑡 1 𝑟 21 𝑟 22 𝑡 2 𝑟 31 𝑟 32 𝑡 3 ∗ 𝑥 𝑝𝑖𝑥𝑒𝑙 𝑦 𝑝𝑖𝑥𝑒𝑙 1 Calibration

Motion detection Hannes Van De Vreken

Person detection Jan Heuninck

Demonstration