1. 3D computer vision techniques v.4b21 3D computer vision techniques KH Wong

2. 3D computer vision techniques v.4b22 Seminar Title: 3D computer vision techniques. Abstract In this talk, the ideas of obtaining 3D information of objects (or called 3D reconstruction) using different techniques are discussed. Currently, the most popular one is the image based method that uses 2D cameras for 3D reconstruction; in particular reconstruction based on one-image, two-image and multiple- image are discussed. Moreover, batch and sequential treatments of input data are studied. I will also talk about novel techniques, such as using multiple cameras and laser based methods to obtain 3D information. And I will discuss how 3D computer vision is used in film and game production. Finally naked-eye 3D display technologies will be mentioned.

3. 3D computer vision techniques v.4b23 Overview (part1) Introduction From 2D to 3D Camera systems/calibration Feature extraction/correspondence Reconstruction algorithms 2 views, 3 views, N views Real-time algorithms/Kalman filter Previous projects Virtual viewer/ Projector camera systems Keystone correction Novel setups Multiple cameras/ Camera array Obtain 3D directly Structured light Laser approach Kinect approach Photometric stereo

4. 3D computer vision techniques v.4b24 Overview (part 2) Applications Photos from tourists (photo tourism) http://phototour.cs.washington.edu/ http://phototour.cs.washington.edu/ 3D displays Possible future research Classification based on 3D information Content search 3D based on 3D keys Merging with sound information

5. 3D computer vision techniques v.4b25 Motivation We live in a 3D world We see 2D images but perceive the world in 3D Intelligent robot should have this 3D reconstruction capability

6. 3D computer vision techniques v.4b26 How to obtain 3D information? Cameras-2D Range sensors-3D

7. 3D computer vision techniques v.4b27 Challenges Obtain 3D information for tasks in a 3D world. 2D-to-3D reconstruction from a camera 3D directly— laser range sensor, kinect sensor Novel sensors Camera array/ multiple camera One pixel camera light field camera

8. 3D computer vision techniques v.4b28 2D-to-3D reconstruction (feature based method) Camera (perspective projection) Features-extraction and correspondences Methods One-image method Two-image (Stereo) method Three-image method N-image method Bundle adjustment Kalman filter

9. 3D computer vision techniques v.4b29 Camera: 3D to 2D projection Perspective model u=F*X/Z (nonlinear relation) v=F*Y/Z F Z Y v World center F Thin lens or a pin hole Virtual Screen or CCD sensor Real Screen Or CCD sensor Pinhole Camera http://upload.wikimedia.org/wikipedia/en/8/81/Pinhole-camera.png

10. 3D computer vision techniques v.4b210 Perspective Projective Model M at t=1 c (Image center, o x,o y ) F=focal length image O c = (0,0,0) (Camera center) Xc-axis Zc-axis Yc-axis v-axis u-axis X,Y,Z (u,v) (0,0) of image plane Camera Coordinates. World Coordinates Yw Zw Xw Rc,Tc Principal axis

11. 3D computer vision techniques v.4b211 In paintings Western Fresco by Raphael, 1510 - 1511, Stanza della Signatura, Vatican Palace, Rome. Chinese 《富春山居圖》是 元朝畫家黃公望的 作品，創作於 1347 年至 1350 年 元朝黃公望 Dwelling in the Fuchun Mountains ( 富春 山居圖 ) by Huang Gongwang (1269–1354)Huang Gongwang http://www.es.flinders.edu.au/~mattom/science+society/lectures/illustrations/lecture17/schoolathens.html http://jsl641124.blog.163.com/blog/static/17702514320115219508530/

12. 3D computer vision techniques v.4b212 Feature correspondences --Camera moved, find correspondences for neighboring images --We can use feature to identify the motions of projected 3D features in 2D. Image at t=t 0 (or left image) Image at t=t 0 +dt (or right image) Area a

13. 3D computer vision techniques v.4b213 Demo Youtube Movie http://www.youtube.com/watch?v=azl- DGK6e1U http://www.youtube.com/watch?v=azl- DGK6e1U

14. 3D computer vision techniques v.4b214 One-image 2D-to-3D reconstruction

15. 3D computer vision techniques v.4b215 One image 2D-to-3D reconstruction method Difficult and with ambiguity http://ai.stanford.edu/~asaxena/reconstruction3d/

16. 3D computer vision techniques v.4b216 One image 2D-to-3D Using prior knowledge (e.g. face) http://www.wisdom.weizmann.ac.il/~ronen/papers/ Hassner Basri - Example Based 3D Reconstruction from Single 2D Images.pdf

17. 3D computer vision techniques v.4b217 Two-image 2D-to-3D reconstruction

18. 3D computer vision techniques v.4b218 Two-image 2D-to-3D reconstruction method: stereo vision Objectives: Basic idea of stereo vision Stereo reconstruction by epipolar geometry Stereo camera pair calibration (find Fundamental matrix F) Construct the 3D (graphic) model from 2 images Graphic model Inside a computer

19. 3D computer vision techniques v.4b219 X’ l X’ r Focal Length f Object Px(x,y,z) z Left camera center (reference point) Horizontal Disparity=x L -x R b (Baseline) Left Camera Principle axis Right Camera Principle axis Left Image plane Right Image plane By similar triangle, w.r.t left camera lens center By similar triangle, w.r.t right camera lens center if camera motion is pure translation : Triangular calculation One major problem is to locate x’l and x’r The correspondence problem

20. 3D computer vision techniques v.4b220 If camera motion is NOT pure translation : Use Epipolar Geometry X O2O2 O1O1 right Frame Plane-2  2 left Frame Plane-1  1 e1e1 e2e2 Left epipolar line Right epipolar line R,T (x 1,y 1 ) (x 2,y 2 ) Left side is the reference Focal length=f 1 Focal length=f 2 Base line=||T|| Plane-3  3 Perpendicular to T  X 2 or T  X 1 Right_image_pointT*E*left_image_point=0

21. 3D computer vision techniques v.4b221 Method: 8-point algorithm http://www.cs.manchester.ac.uk/ugt/COMP37111/papers/Hartley.pdf Find 8 point corresponded ( Map 8 Right_image_points to left_image_point Solve the epeiolar formula Right_image_pointT*E*left_image_point=0 Find E. From E we can find camera R (rotation),T (translation) From R,T we can find model (3D positions of the left feature points (using left as reference)

22. 3D computer vision techniques v.4b222 An example of stereo reconstruction An example Short-Baseline Stereo Systems for Mobile Devices http://www.lelaps.de/videos.html#SQx5vU8BA-M http://www.lelaps.de/projects/stmobile.html

23. 3D computer vision techniques v.4b223 Stereo-based Free-space Estimation Another example http://www.lelaps.de/videos.html#VrKBNtAN03o http://www.lelaps.de/projects/freespace.html

24. 3D computer vision techniques v.4b224 Three-image 2D-to-3D reconstruction

25. 3D computer vision techniques v.4b225 Three-image 2D-to-3D reconstruction method More robust using 3 views It contains 3 epipolar relations Stereo1: view1,2, Stereo2: view2,3, Stereo3 :view 3,1. Combine 3 epipolar geometry information together. Similar to the algorithm in epipolar geometry (apply 3 times) http://www.cs.unc.edu/~marc/tutorial/node45.html M=3-D model point M, m’, m” are image points C,C’,C” are camera centers

26. 3D computer vision techniques v.4b226 Example of 3-image reconstruction Example LIBVISO: Feature Matching for Visual Odometry http://www.youtube.com/watch?v=DPLh6MoxPAk

27. 3D computer vision techniques v.4b227 N-image 2D-to-3D reconstruction (batched method: order of images can be random )

28. 3D computer vision techniques v.4b228 N-image 2D-to-3D reconstruction method Bundle adjustment approach Guess iteratively the solution for 3D to explain the measurements of feature points in all images Math: Q(u,v)=g(X), g is nonlinear (projection) because u=fX/Z v=fY/Z, f=focal length Given Q (image measurement), we want to find X=(X,Y,Z) i from image points (u,v) i of all N model points (i=1,,,N), g is the projection formulas A typical non linear optimization problem, Gauss-Newton for non linear optimization method is used.

29. 3D computer vision techniques v.4b229 Batched method: order of images can be random From measurement [u,v]I find X … Camera motion Image t=1 Image t=2 Image t=3 Image t=m … v1 v2 v3 vm X [u,v]2 [u,v]1 [u,v]3 [u,v]m O1 O2 O3 Om R2,T2 R3,T3 Rm,Tm R1,T1

30. 3D computer vision techniques v.4b230 Example Bundle adjustment reconstruction http://www.cse.cuhk.edu.hk/%7Ekhwong/demo/canyon2b2.mpg

31. 3D computer vision techniques v.4b231 N-image 2D-to-3D reconstruction (Sequential method: order of images are used like in a move )

32. 3D computer vision techniques v.4b232 Sequential method: order of images are used like in a move From measurement [u,v]I find X … Camera motion Image t=1 Image t=2 Image t=3 Image t=m … v1 v2 v3 vm X [u,v]2 [u,v]1 [u,v]3 [u,v]m O1 O2 O3 Om R2,T2 R3,T3 Rm,Tm R1,T1

33. 3D computer vision techniques v.4b233 PredictionCorrection Kalman Filter 33 pictures by Ko Hoi Fung

34. 3D computer vision techniques v.4b234 Kalman filter example 34 t = 0: Position = x0 Velocity = v0 t’ = 1: Position = x1’ Velocity = v1’ x1’ = v0 * t + x0 t = 1: Position = x1 PredictionPrediction UpdateUpdate States: Position Velocity Measurements: Position

35. 3D computer vision techniques v.4b235 Example Hernan Badino and Takeo Kanade: "A Head-Wearable Short- Baseline Stereo System for the Simultaneous Estimation of Structure and Motion". IAPR Conference on Machine Vision Applications (MVA), Nara, Japan, June 2011 http://www.youtube.com/watch?v=SQx5vU8BA-M

36. 3D computer vision techniques v.4b236 Novel sensors : Camera array/ Multiple camera systems Camera array/ multiple camera: High Performance Imaging - Using Large Camera Array http://www.youtube.com/watch?v=0W_1Ce2lTBo http://graphics.stanford.edu/papers/CameraArray/

37. 3D computer vision techniques v.4b237 The Self-Reconfigurable Camera Array http://chenlab.ece.cornell.edu/projects/MobileCamArray/ Demo movie http://chenlab.ece.cornell.edu/projects/MobileCamArray/videos/train.mov http://chenlab.ece.cornell.edu/projects/MobileCamArray/videos/self_reconfiguration.mov Each camera

38. 3D computer vision techniques v.4b238 Applications

39. 3D computer vision techniques v.4b239 Photo tourism http://phototour.cs.washington.edu/

40. 3D computer vision techniques v.4b240 Projector-camera system Application of computer vision

41. 3D computer vision techniques v.4b241 A Projector-Camera system

42. 3D computer vision techniques v.4b242 Projector-Camera calibration http://www.youtube.com/watch?v=YHhQSglmuqY&feature=channel_page

43. 3D computer vision techniques v.4b243 Our setup

44. 3D computer vision techniques v.4b244 Calibration procedure

45. 3D computer vision techniques v.4b245 Quadrangle tracking

46. 3D computer vision techniques v.4b246 Experiments

47. 3D computer vision techniques v.4b247 Projection result

48. 3D computer vision techniques v.4b248 Results

49. 3D computer vision techniques v.4b249 Hand held direct manipulation 3D Display http://www.youtube.com/watch?v=vVW9QXuKfoQ&feature=relmfu

50. 3D computer vision techniques v.4b250 Keystone correction Configuration

51. 3D computer vision techniques v.4b251 Aim of this work Desired Results Keystoned projectionCorrected projection

52. 3D computer vision techniques v.4b252 Overview Three major modules Projector-camera pair calibration Projection region detection and tracking Automatic keystone correction Flow chart

53. 3D computer vision techniques v.4b253 Pre-warp projection image Pre-warped projection imageDisplay result http://www.youtube.com/watch?v=y5XYdeh8Bno&list=UUfy2EumiHMeoUorMFR0woZA&index=1&feature=plcp

54. 3D computer vision techniques v.4b254 Keystone correction Some real correction results

55. 3D computer vision techniques v.4b255 Obtain 3D directly Laser range sensor Time of flight Kinect

56. 3D computer vision techniques v.4b256 Photometric stereo http://www.taurusstudio.net/research/photex/ps/equation.htm Lamertian light formula Given 3 or more known light source we can find the normal N From the set of N we can approximate the surface http://www.wisdom.weizmann.ac.il/~vision/photostereo/

57. 3D computer vision techniques v.4b257 Photometric stereo using multiple cameras and multiple light sources Demo Dynamic Shape Capture using Multi-View Photometric Stereo SIGGRAPH 2009 http://www.youtube.com/watch?v=9hgs5zN38lk

58. 3D computer vision techniques v.4b258 Multiple cameras fro human body reconstruction Homepage://media.au.tsinghua.edu.cn

59. 3D computer vision techniques v.4b259 Experimental Results 59 3D Modeling Using MVML Dome2015-8-14

60. 3D computer vision techniques v.4b260 Multiple camera doom http://www.mpi-inf.mpg.de/~yliu/

61. 3D computer vision techniques v.4b261 Structured light method Calculate the shape by how the strip is distorted. http://www.laserfocusworld.com/articles/2011/01/lasers-bring-gesture-recognition-to-the-home.html

62. 3D computer vision techniques v.4b262 Real time Virtual 3D Scanner - Structured Light Technology Demo http://www.youtube.com/watch?v=a6pgzNUjh_s

63. 3D computer vision techniques v.4b263 Time of flight laser method Send the IR-laser light to different directions and sense how each beam is delayed. Use the delay to calculate the distance of the object point http://www.laserfocusworld.com/articles/2011/01/lasers-bring-gesture-recognition-to-the-home.html

64. 3D computer vision techniques v.4b264 LIDAR light detection and ranging scanner http://hodcivil.edublogs.org/2011/11/06/lidar-%E2%80%93-light-detection-and-ranging/ http://commons.wikimedia.org/wiki/File:Lidar_P1270901.jpg LeicaLeica terrestrial lidar (light detection and ranging) scanner lidar http://www.youtube.com/watch?v=MuwQTc8KK44

65. 3D computer vision techniques v.4b265 3D Laser Scanning - Underground Mine Mapping Demo http://www.youtube.com/watch?v=BZbvz8fePeQ

66. 3D computer vision techniques v.4b266 Motion capture for film production (MOCAP) http://upload.wikimedia.org/wikipedia/commons/7/73/MotionCapture.jpg http://www.naturalpoint.com/optitrack/products/s250e/indepth.html IR light emitter and camera http://www.youtube.com/watch?v=IxJrhnynlN8

67. 3D computer vision techniques v.4b267 3D body scanner http://www.cyberware.com/products/scanners/ps.html http://www.cyberware.com/products/scanners/wbx.html http://www.youtube.com/watch?v=86hN0x9RycM

68. 3D computer vision techniques v.4b268 3-D Face capture http://www.captivemotion.com/products/ http://www.youtube.com/watch?v=-TTR0JrocsI&feature=related

69. 3D computer vision techniques v.4b269 Dimensional Imaging 4D Video Face Capture with Textures Dimensional Imaging 4D Video Face Capture with Textures http://www.youtube.com/watch?v=XtTN7tWaXTM&feature=related

70. 3D computer vision techniques v.4b270 Kinect Another structure light method Use dost rather than strips http://www.laserfocusworld.com/articles/2011/01/lasers-bring-gesture-recognition-to-the-home.html

71. 3D computer vision techniques v.4b271 Kinect Hardware

72. 3D computer vision techniques v.4b272 See the IR-dots emitted by KINECT http://www.youtube.com/watch?v=-gbzXjdHfJA http://www.youtube.com/watch?v=dTKlNGSH9Po&feature=related

73. 3D computer vision techniques v.4b273 Novel sensors : light field camera Spin off from Stanford camera array light field camera : LYTRO camera Be able to refocus after the picture is taken https://www.lytro.com/camera http://www.youtube.com/watch?v=7QV152jc3Ac

74. 3D computer vision techniques v.4b274 light field camera How does it work http://www.quora.com/Lytro/How-does-the-new-Lytro-camera-work

75. 3D computer vision techniques v.4b275 3D (Volumetric) display Rendering for an Interactive 360º Light Field Display SIGGRAPH 2007 Papers Proceedings http://gl.ict.usc.edu/Research/3DDisplay/ http://www.youtube.com/watch?v=h6aUIS44ezE

76. 3D computer vision techniques v.4b276 Occlusion-Capable Volumetric 3D Display by Cossairt,etal. Actuality Systems, Inc http://www.3dcgi.com/cooltech/displays/displays.htm http://www.youtube.com/watch?v=8KaQmn2VTzs

77. 3D computer vision techniques v.4b277 3D display Using a lattice with thin slits, viewer's eyes see different pixels on the screen to create 3d perception http://www.televisions.com/tv-articles/TV-in-3D/Displaying-3D-Without-Glasses.php

78. 3D computer vision techniques v.4b278 The future Content search in 3D video data bases Shot boundary detection Video data mining Video classification

79. 3D computer vision techniques v.4b279 Appendix

80. 3D computer vision techniques v.4b280 Essential matrix E (a 3x3 matrix) P.110[2] X 1 is 3-D X in left camera (reference) system X 2 is 3-D X in right camera system Exercise1: Draw vectors T  X2 or T  X1 in the diagram

81. 3D computer vision techniques v.4b281 Essential Matrix E Right_image_point T *E*left_image_point=0