Robot Vision SS 2011 Matthias Rüther / Matthias Straka 1 ROBOT VISION Lesson 7: Volumetric Object Reconstruction Matthias Straka

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 2 Motivation  We want to create 3D objects from real-world observations

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 3 Content  Volume Elements (Voxels) –Voxel vs. Pixel  Shape by Space Carving –Excurse: Stone Carving –Projective Camera Revised –Space Carving from Silhouettes –Other Space Carving methods  Applications

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 4 Voxels  An image is made out of a regular grid of discrete pixels  Similarly, a volume can be discribed as a grid of volume elements (voxels)  Voxels can –Be binary (0/1, object/background,...) –Represent a value (object density,...) –Have color information

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 5 Stone Carving Template Object Carve away everything but the object

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 6 Really Impressive Stone Carving There are no columns, just stone that has not been carved away Elephanta Island, Mumbai, India

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 7 Foam Cutting Images: and Use a 2D template, cut away regions that do not lie within the template area using a hot wire „Deluxe“ CNC Foam Cutter

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 8 Space Carving in Computer Vision  Our template is a set of silhouette images from multiple views (including the camera projection matrices P)  Our „Foam“ is a volume made up of discrete voxels  Our „cutting wire“ is the „viewing ray “

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 9 Perspective Camera Model For every 3D point we can calculate the projected 2D image point = For each image point we get a viewing ray  See Lesson 4

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 10 Space Carving Principle: Shape from Silhouette Camera Center Template Image Voxel Volume Viewing Ray Space Carving starts with a full grid of voxels For each pixel outside the object, we delete all voxels along its viewing ray

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 11 Space Carving Principle: Shape from Silhouette  Deleting voxels along the viewing ray is not very efficient –Detect which voxels are penetrated by viewing ray (hit-test) –Repeat for every camera  Better: Project each voxel center into all silhouette images –Only voxels that project to pixels inside the silhouette in all images are object-voxels For Each voxel v = 1..V isobject(v) = true For Each camera c = 1..C p = project v into image I(c) using projection matrix P(c) If p inside image I(c) Then isinside = pixel p is an object pixel in image I(c) isobject(v) = isobject(v) & isinside End If End For

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 12 Shape from Silhouette

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 13 Shape from Silhouette: Visual Hull Images: pages.cs.wisc.edu/~lizhang Space Carving from Silhouettes only reconstructs the Visual Hull The reconstructed object is generally larger than the real object

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 14 Shape from Silhouette: Pros and Cons  Pros –Fast –Easy to implement –Easy to parallelize (GPU computing) –No texture required, just the silhouette  Cons –Result is not photo consistent (only visual hull) –No concavities –False holes in the silhouette lead to holes in the object

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 15 Probabilistic Space Carving  No binary decision whether to carve the voxel or not  Silhouettes are no longer binary but encode probability of objectness  Each voxel represents a „object probability„ which is equal to the probability that all cameras observe this voxel as an object where is the projection of the voxel into image c  Similarly, a background probability can be defined:  Benefit: Increased robustness towards errors in silhouette images

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 16 Other Space Carving Methods  Voxel Coloring –Project each voxel into all camera images –Only voxels that have pixels with similar color information in all images are surface voxels –Allows for concavities –Reconstructs only surface voxels –Needs to handle visibility / occlusion Images: S. Seitz et al.

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 17 Other Space Carving Methods  Photo Consistency –Similar to voxel coloring –Use more advanced methods than color-consistency (i.e. NCC patch matching, stereo matching,...)  Energy Minimization Methods –Minimize a cost that uses object probabilies and photo consistency for each voxel as well ensures that the resulting surface is „smooth“ –Result: optimal object surface Images: G. Vogiatzis et al.

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 18 Application of Space Carving  3D Reconstruction from a Turn-Table

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 19 Application of Space Carving  Using a single camera and mirrors to generate multiple views (catadioptric setup)

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 20 Application of Space Carving  Human Body Reconstruction in Real-Time 10 fixed cameras record a live-stream of a person, green background facilitates generation of silhouette images

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 21 Application of Space Carving

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 22 Application of Space Carving  Human Body Reconstruction and Virtual Mirror Image

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 23 Virtual Mirror System Architecture  Technology Used: –FireWire 800 / 1394-B (up to 800Mbit/s / Bus) –Topology: Star (via Hubs), Daisy-Chain –Allows for camera synchroniztaion and power supply via FireWire cable Computer (FireWire PCIe interface card) Multiple Cameras 5-Port Hub with Power Supply

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 24 Camera Calibration  How to calibrate 10 cameras?  Internal Calibration –Each camera individually using a planar target –Estimate intrinsic parameters (K matrix) –Estimate radial and tangential distortion  External Calibration –Use a 3D target (see image) with multiple markers and known configuration –Each camera detects at least one marker and calculates its position relative to the marker –The center of the 3D target defines the coordinate origin (0, 0, 0) and the orientation –This gives the [R|t] matrix for each camera

Robot Vision SS 2011 Matthias Rüther / Matthias Straka 25 Summary  A voxel is similar to a pixel, but in 3D  Space Carving using silhouette images is fast and easy to implement but only reconstructs the Visual Hull  When using color/texture information, this disadvantage can be reduced  There is a variety of applications where space carving/volumetric object reconstruction can be used –Many projects at the ICG