1. 3D computer vision techniques
KH Wong
3D computer vision techniques v.6f

2. Seminar Title: 3D computer vision techniques.
Abstract In this talk, the ideas of obtaining 3D information of objects (or called 3D reconstruction) using different techniques are discussed. One popular approach is the image feature based method that uses 2D camera. In particular 3D reconstruction methods based on one-image, two-image and multiple-image are discussed. Moreover, batch and sequential treatments of input data are studied. I will also talk about novel techniques, such as using multiple cameras and laser based methods to obtain 3D information. And I will discuss how 3D computer vision is used in film and game production. Finally naked-eye 3D display technologies will be mentioned.
3D computer vision techniques v.6f

3. 3D computer vision techniques v.6f
Overview (part1)
Introduction and motivation
From 2D to 3D
Camera systems/calibration
What are the problems to be solved: pose estimation and structure from motion SFM
Feature extraction/correspondence
Reconstruction algorithms
1 view, 2 views, 3 views ,
N views
(i) Batch method: slow
(ii) sequential method : fast Real-time algorithms/Kalman filter
Direct 3D approach using 3D scanners
Projector-camera systems
Applications example
Recent project
Wearable glasses
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction
Robot Avatar: a virtual tourism robot for people with disabilities
An efficient 3-D environment scanning method
3D computer vision techniques v.6f

4. 3D computer vision techniques v.6f
Overview (part 2)
Applications
Photos from tourists (photo tourism)
3D displays
Possible future research
Classification based on 3D information
Content search 3D based on 3D keys
Merging with sound information
3D computer vision techniques v.6f

5. Introduction and motivation
We live in a 3D world
We see 2D images but perceive the world in 3D
Intelligent robot should have this 3D reconstruction capability
3D computer vision techniques v.6f

6. How to obtain 3D information?
Cameras-2D
Range sensors-3D
3D computer vision techniques v.6f

7. 3D computer vision techniques v.6f
Challenges
Obtain 3D information for tasks in a 3D world.
2D-to-3D reconstruction from a camera
3D directly— laser range sensor, kinect sensor
Novel sensors
Camera array/ multiple camera
One pixel camera
light field camera
3D computer vision techniques v.6f

8. From 2D to 3D: 2D-to-3D reconstruction (feature based method)
Camera (perspective projection)
Features-extraction and correspondences
Methods
One-image method
Two-image (Stereo) method
Three-image method
N-image method
Bundle adjustment
Kalman filter
3D computer vision techniques v.6f

9. Camera system: 3D to 2D projection
Camera system: 3D to 2D projection
Pinhole
Camera
Perspective model
u=F*X/Z (nonlinear relation)
v=F*Y/Z
Virtual
Screen
or CCD
sensor
World
center Y v F Z F
Real
Screen Or
CCD
sensor
Thin lens
or a pin hole
3D computer vision techniques v.6f

10. Perspective Projective
World
Coordinates Yw Zw
Rc,Tc Xw
Model Mx at t=1
c (Image center, ox,oy)
F=focal length
image
Oc=
(0,0,0)
(Camera center)
Xc-axis
Zc-axis
Yc-axis
v-axis
u-axis
X,Y,Z
Camera
Coordinates.
(u,v)
Principal axis
(0,0) of image plane
3D computer vision techniques v.6f

11. Define: Pose and model structurey z
Based on the reference coordinate system
Model: Mx is a feature point, the model of Mx is the 3D position of Mx.
Pose of the camera: Pose of image 1 is R1,T1. pose of camera2 is RF2,T2 etc. x
The reference coronate Mx
Object M v1
[u,v]1 v2 v3
[u,v]2 vm O1
Image
t=1
[u,v]3 …
[u,v]m …
R1,T1
Image
t=2
Image
t=3
Image
t=m O2 Om
Rm,Tm
R2,T2 O3
3D computer vision techniques v.6f
Camera motion
R3,T3

12. What are the problems to be solved ?
Assume you have a set of images taken by a moving camera (e.g. a video by a motiving camera)
Pose estimation and
Assume the model (3D positions of each feature point) of the object is known, from the sequence of images of the object find the pose (rotation R and translation T ) of each camera position.
Structure from motion (SfM)
From the sequence of images of the object find (i) the pose (rotation R and translation T ) of each camera position and (ii) also the model (3D positions of each feature point) of the object.
3D computer vision techniques v.6f

13. 3D perception In paintings
3D computer vision techniques v.6f
3D perception In paintings
Western:Fresco by Raphael, , Stanza della Signatura, Vatican Palace, Rome.
Chinese: Dwelling in the Fuchun Mountains (富春山居圖) by Huang Gongwang (1269–1354) -- 《富春山居圖》是元朝畫家黃公望的作品，創作於1347年至1350年

14. Camera calibration of a camera
To find the camera parameters
Focal length (f) , CCD alignment position (Ox,Oy) etc.
Method:
Take pictures of a known object (calibration object, e.g. checker board) at different positions
Use mathematical method to find focal length etc.
Popular tool: MATLAB camera calibration toolbox, or OPENCV cam cal tool
3D computer vision techniques v.6f

15. 3D computer vision techniques v.6f
Features extraction & correspondence --Camera moved, find correspondences for neighboring images --We can use feature to identify the motions of projected 3D features in 2D.
Area a
Image at t=t0+dt,
(or right image)
Image at t=t0 (or left image)

16. 3D computer vision techniques v.6f
The Harris algorithm for feature extraction ref:
%For each N(x,y)
%in the 12x12 neighborhood image N(x,y)
For (i=1, i<=12 i++)
{ For (j=1, j<=12 j++)
{ find
I(i,j)/x %convolve I(i,j) with x_edge mask
I(i,j) /y %convolve I(i,j) with y_edge mask }
% then find a 2x2 matrix A(x,y) for N(x,y)
The 12x12 window
N(x,y)
(i=1,j=1)
(x,y)
i=12,j=12
3D computer vision techniques v.6f

17. Demo : feature extraction and tracking
Youtube Moviehttp://
3D computer vision techniques v.6f

18. 2D-to-3D reconstruction
One-image
2D-to-3D reconstruction
3D computer vision techniques v.6f

19. One image 2D-to-3D reconstruction method
Difficult and with ambiguity
3D computer vision techniques v.6f

20. Photometric stereo method: One image 2D-to-3D
Using prior knowledge (e.g. face)
The idea:
Position the light source at different directions.
The normal vector of the object surface at each pixel can be found by the reflected intensity.
Based on the normal vectors, find the 3D model of the whole object
Hassner Basri - Example Based 3D Reconstruction from Single 2D Images.pdf
3D computer vision techniques v.6f

21. 2D-to-3D reconstruction
Two-image
2D-to-3D reconstruction
3D computer vision techniques v.6f

22. Two-image 2D-to-3D reconstruction method: stereo vision
Objectives:
Basic idea of stereo vision
Stereo reconstruction by epipolar geometry
Stereo camera pair calibration (find Fundamental matrix F)
Construct the 3D (graphic) model from 2 images
Inside a computer
Graphic
model
3D computer vision techniques v.6f

23. 3D computer vision techniques v.6f
Two-view stereo vision method with parallel cameras:
If camera motion is pure translation: Triangular calculation
X’l
X’r
Focal
Length f
Object
Px(x,y,z) z
Left camera center
(reference point)
Horizontal
Disparity=xL-xR
b (Baseline)
Left
Camera
Principle
axis
Right
Image
plane
By similar triangle,
w.r.t left camera lens center
By similar triangle,
w.r.t right camera lens center
3D computer vision techniques v.6f
One major problem is to locate x’l and x’r The correspondence problem

24. 3D computer vision techniques v.6f
If camera motion is NOT pure translation (with some rotation) : Use Epipolar Geometry X
Right_image_pointT*E*left_image_point=0
Left side is the reference
Plane-3 3
Perpendicular to
TX2 or TX1
(x2,y2)
Right epipolar line
Left epipolar line
(x1,y1)
left
Frame
Plane-1 1
right Frame
Plane-2 2 O2 O1 e1 e2
R,T
Focal
length=f1
Base line=||T||
Focal
length=f2
3D computer vision techniques v.6f

25. 3D computer vision techniques v.6f
Method: 8-point algorithm Ref:
Find 8 point corresponded (
Map 8 Right_image_points to left_image_point
Solve the epeiolar formula
Right_image_pointT*E*left_image_point=0
Find E.
From E we can find camera R (rotation) ,T (translation)
From R,T we can find model (3D positions of the left feature points (using left as reference)
Details see
3D computer vision techniques v.6f

26. 3D computer vision techniques v.6f
The 8-point algorithm idea
Output of demo_stereo5i_for_ex4.m, from
Cam1 (reference)
Epipolar lines and epipole at cam2
3D computer vision techniques v.6f

27. 3D computer vision techniques v.6f
Example stereo images with epipolar lines The epipolar lines of each image will join at the epipole e (it may be outside the image frame)
Real image example
Left
Right
RANSAC idea:
Random select 8 point pairs (corresponding point pairs) find their Fundamental matrix (Rotation (R) and translation (T) between cameras) , Use voting to find final results to reduce noise.
Then use R,T to reconstruct the 3D positions of these feature points
3D computer vision techniques v.6f

28. An example of stereo reconstruction
Short-Baseline Stereo Systems for Mobile Devices
3D computer vision techniques v.6f

29. Stereo-based Free-space Estimation
Another example
3D computer vision techniques v.6f

30. Stereo hardware approach
Qualcomm Research
Stereo camera pair
3D computer vision techniques v.6f

31. 2D-to-3D reconstruction
Three-image
2D-to-3D reconstruction
3D computer vision techniques v.6f

32. Three-image 2D-to-3D reconstruction method
More robust using 3 views
It contains 3 epipolar relations
Stereo1: view1,2 ,
Stereo2: view2,3,
Stereo3 :view 3,1.
Combine 3 epipolar geometry information together.
Similar to the algorithm in stereo / epipolar geometry method but (apply 3 times, each for a stereo pair)
M=3-D model point
M, m’, m” are image points
C,C’,C” are camera centers
3D computer vision techniques v.6f

33. Example of 3-image reconstruction
LIBVISO: Feature Matching for Visual Odometry
3D computer vision techniques v.6f

34. 3D computer vision techniques v.6f
N-image
2D-to-3D reconstruction
(i) Batch method: slow
(ii) sequential method : fast Real-time algorithms/Kalman filter
3D computer vision techniques v.6f

35. Batched method: order of images can be random
Use all images collected for reconstruction
Order of images is not important
From measurement [u,v]I find X X v1
[u,v]1 v2 v3
[u,v]2 vm O1
Image
t=1
[u,v]3 …
R1,T1
[u,v]m …
Image
t=2
Image
t=3
Image
t=m Om O2
Rm,Tm
R2,T2 O3
3D computer vision techniques v.6f
R3,T3
Camera motion

36. 3D computer vision techniques v.6f
(i) Batch method: N-image 2D-to-3D Use all images collected for reconstruction
Procedure
Feature extraction,
Run bundle adjustment
Result in 3D
Details in
3D computer vision techniques v.6f

37. 3D computer vision techniques v.6f
N-image ii) sequential method : fast Real-time algorithms/Kalman filter
3D computer vision techniques v.6f

38. 3D computer vision techniques v.6f
Sequential method: order of images is used, it cannot be random. E.g. use a video as the input
Each time only consider one new input image
So it is faster and can run in real time X v1
[u,v]1 v2 v3
[u,v]2 vm O1
Image
t=1
[u,v]3 …
R1,T1
[u,v]m …
Image
t=2
Image
t=3
Image
t=m Om O2
Rm,Tm
R2,T2 O3
3D computer vision techniques v.6f
R3,T3
Camera motion

39. 3D computer vision techniques v.6f
Kalman Filter
Prediction
Correction
The Kalman filter has two stages: Prediction stage and correction stage.
I will use an example to demonstrate how Kalman filter work.
pictures by Ko Hoi Fung
3D computer vision techniques v.6f 39 39

40. 3D computer vision techniques v.6f
Kalman filter example
t’ = 1:
Position = x1’
Velocity = v1’
x1’ = v0 * t + x0
Prediction
t = 0:
Position = x0
Velocity = v0
t = 1:
Position = x1
Let position and velocity be the state.
Usually, not all the states can be measured or observed.
Assume the car is moving in 1D and its acceleration is constant.
Also, we can only measure the position of the car.
Also assume the initial state is x0 and v0.
Then Kalman filter will make a prediction of the state at next time frame according to the current state
Since the car has a constant acceleration, we can find a linear relation between the position and the velocity.
We measure the position the car at the next time frame.
This measurement will be used to update the Kalman filter so that the prediction will get better in the next frame if the filter converge.
States:
Position
Velocity
Measurements:
Update
3D computer vision techniques v.6f 40 40

41. 3D computer vision techniques v.6f
Example
Hernan Badino and Takeo Kanade: "A Head-Wearable Short-Baseline Stereo System for the Simultaneous Estimation of Structure and Motion". IAPR Conference on Machine Vision Applications (MVA), Nara, Japan, June 2011
3D computer vision techniques v.6f

42. Novel sensors : Camera array/ Multiple camera systems
Camera array/ multiple camera: High Performance Imaging - Using Large Camera Array
3D computer vision techniques v.6f

43. The Self-Reconfigurable Camera Array
Each camera
Demo movie
MOV
3D computer vision techniques v.6f

44. Direct 3D approach using 3D scanners: the Microsoft Kinect or Polhemus
Polhemus
3D computer vision techniques v.6f

45. Projector-camera systems
Application of computer vision
3D computer vision techniques v.6f

46. A Projector-Camera system
3D computer vision techniques v.6f

47. Projector-Camera calibration
3D computer vision techniques v.6f

48. 3D computer vision techniques v.6f
Our setup
3D computer vision techniques v.6f

49. Calibration procedure
3D computer vision techniques v.6f

50. 3D computer vision techniques v.6f
Quadrangle tracking
3D computer vision techniques v.6f

51. 3D computer vision techniques v.6f
Experiments
3D computer vision techniques v.6f

52. 3D computer vision techniques v.6f
Projection result
3D computer vision techniques v.6f

53. 3D computer vision techniques v.6f
Results
3D computer vision techniques v.6f

54. Hand held direct manipulation 3D Display
3D computer vision techniques v.6f

55. 3D computer vision techniques v.6f
Keystone correction
Configuration
3D computer vision techniques v.6f

56. 3D computer vision techniques v.6f
Aim of this work
Desired Results
Keystoned projection
Corrected projection
3D computer vision techniques v.6f

57. 3D computer vision techniques v.6f
Overview
Three major modules
Projector-camera pair calibration
Projection region detection and tracking
Automatic keystone correction
Flow chart
3D computer vision techniques v.6f

58. Pre-warp projection image
Pre-warped projection image
Display result
3D computer vision techniques v.6f

59. 3D computer vision techniques v.6f
Keystone correction
Some real correction results
3D computer vision techniques v.6f

60. 3D computer vision techniques v.6f
Application examples
and demos
3D computer vision techniques v.6f

61. 3D computer vision techniques v.6f
Photo tourism
Youtube
3D computer vision techniques v.6f

62. 3D computer vision Applications
3D models from images
Game development
Robot navigation
3G Mobile phone applications,
Location systems
User input
3D computer vision techniques v.6f

63. 3D computer vision techniques v.6f
Demo A1: 3D reconstruction (see also (Click picture to see movie)
Grand Canyon Demo
Flask
Robot
3D computer vision techniques v.6f

64. Demo A2: augmented reality (Click picture to see movie)
Augmented reality demo
3D computer vision techniques v.6f

65. Demo A3 Projector camera system (PROCAM) Click pictures to see movies
CVPR 09
A Projector-based Movable
Hand-held Display System
VRCAI09:A Hand-held 3D Display
System that facilities direct
manipulation of 3D virtual
objects
3D computer vision techniques v.6f

66. 3D computer vision techniques v.6f
Demo A4
Flexible projected surface
3D computer vision techniques v.6f

67. 3D computer vision techniques v.6f
Demo A5
3-D display without the use of spectacles.
3D computer vision techniques v.6f

68. 3D computer vision techniques v.6f
Demo A6
Spherical projected surface for 3D viewing without spectacles.
3D computer vision techniques v.6f

69. 3D computer vision techniques v.6f
Demo A7
A KEYSTONE-FREE HAND-HELD MOBILE PROJECTION
3D computer vision techniques v.6f

70. Recent computer vision projects
Wearable glasses
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction
Robot Avatar: a virtual tourism robot for people with disabilities
An efficient 3-D environment scanning method
3D computer vision techniques v.6f

71. 1)Project 1: Wearable glasses
Can see through the display
Overlay text/images with the surroundings
3D computer vision techniques v.6f

72. 3D computer vision techniques v.6f
Project 2: Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction
2016 the first International Workshop on Pattern Recognition (IWPR 2016) Tokyo, Japan during May 11-13, Kwan Pang Tsuia, Kin Hong Wongb*, Changling Wanga, Ho Chuen Kamb, Hing Tuen Yaub, and Ying Kin Yu aDepartment of Mechanical Engineering, The Chinese University of Hong Kong, Hong Kong bDepartment of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong
3D computer vision techniques v.6f

73. 3D computer vision techniques v.6f
Overview
Objectives
Theory
Implementation
Results
Conclusion
3D computer vision techniques v.6f

74. 3D computer vision techniques v.6f
Objectives
Investigate different methods to calibrate a 3-D scanning system consisting of multiple Kinect sensors
Reconstruct the full surface model of an object
3D computer vision techniques v.6f

75. 3D Scanning Sensor Types
Laser
Very Accurate
Expensive ( up to US$50,000)
Bulky
LED
Accurate
Mid-range price ( > US$ 2000)
Infra-red
Microsoft Kinect, Structure Sensor
Cheap ( US$100~200)
Less accurate
3D computer vision techniques v.6f

76. 3D computer vision techniques v.6f
Motivations
3-D scanning is important in many scientific and industrial applications
3-D scanners are consumer grade products which are available at low cost
Previous work:
Single Kinect to move around the object to obtain result -- Relatively difficult to be handled and carried out perfectly
Iterative Closest Point (ICP)
Kinect Fusion
Target surface too smooth
Identifiable features too little
3D computer vision techniques v.6f

77. 3D computer vision techniques v.6f
Theory for method 1 Calibrate the pose between Kinect J and K first. Then K and L etc.
Kinect L
Kinect M
Move the sphere and take samples
Kinect J
Kinect K
3D computer vision techniques v.6f

78. 3D computer vision techniques v.6f
Theory for method 1
Overview of the system
Method 1 :
Using Sphere as a calibration object (Staranowicz et. al. paper)
3D computer vision techniques v.6f

79. 3D computer vision techniques v.6f
Theory for method 2 Calibrate the pose between Kinect J and K first. Then K and L etc.
Kinect M
Kinect L
Move the checker board and take samples
Kinect J
Kinect K
3D computer vision techniques v.6f

80. 3D computer vision techniques v.6f
Result
Models obtained by multiple Kinects are combined to become a full 360 degrees Omni model
Video
3D computer vision techniques v.6f

81. 3D computer vision techniques v.6f
Project 3: Robot Avatar: a virtual tourism robot for people with disabilities Chong Wing Cheung, Tai Ip Tsang, Kin Hong Wong*
The 2nd International Conference on Virtual Reality ICVR16, May 20-22, 2016, Chengdu, China.
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong
*corresponding author
Download link
3D computer vision techniques v.6f

82. 3D computer vision techniques v.6f
Introduction
To create a robot agent aimed to help the disabled person experiencing the physical environment
Generic Robot that used in Virtual Tourism or Bomb detonation, etc.
Using Cutting Edge Virtual Reality (VR) Technology
Chose Oculus Rift owing to the features, cost and programmability
Story-telling +1
3D computer vision techniques v.6f

83. 3D computer vision techniques v.6f
System overview
Story-telling +1
3D computer vision techniques v.6f

84. The robot with a stereo camera pair
Head Mount Device
HMD
Hand gesture recognition
Robot avatar
3D computer vision techniques v.6f

85. 3D computer vision techniques v.6f
Demos
Utube :
icvr16_avatar_video.MP4
3D computer vision techniques v.6f

86. Project 4: An efficient 3-D environment scanning method
24th International Conference in Central Europe on  Computer Graphics, Visualization and Computer Vision 2016
(WSCG 2016) Venue : May 30 - June 3, Kin Hong Wonga, Ho Chuen Kamaa, Ying Kin Yua, Sheung Lai Loa, Kwan Pang Tsuib, and Hing Tuen Yaua aDepartment of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong
bDepartment of Mechanical Engineering, The Chinese University of Hong Kong, Hong Kong
3D computer vision techniques v.6f

87. 3D computer vision techniques v.6f
Introduction
Develop a complete system for building the 3D map of a large area e.g. a shopping mall or a museum
3D computer vision techniques v.6f

88. 3D computer vision techniques v.6f
Motivations
Huge demand on the 3-D digitization of larger environment for virtual reality or 3-D navigation applications 3-D scanners are now available at low cost (e.g Kinect )
3D computer vision techniques v.6f

89. 3D computer vision techniques v.6f
The idea
Kinect sensor placed at different positions at different times to cover a large target area
One master camera determines the kinect pose
Pose estimation achieved by placing a dual-face checkerboard on the top of the Kinect
Using mirror to tackle Kinect calibration issue
Data fusion:
LIDAR (Light Detection And Ranging)
Back-to-back Kinects
3D computer vision techniques v.6f

90. Our approach: Data fusion
LIDAR (Light Detection And Ranging)
Sparse 16 or 32 lin3s of accuracy range measurements.
Kinects: Dense, but scanning range is small (70 degrees)
Solution
LIDAR + 2 back-to-back Kinects
3D computer vision techniques v.6f

91. 3D computer vision techniques v.6f
The idea
3D computer vision techniques v.6f

92. The scanning head and scanning robot
The scanning robot
The scanning head has a LIDAR and 2 back-to-back Kinects
Back-to-back
Kinects
LIDAR
3D computer vision techniques v.6f

93. 3D computer vision techniques v.6f
Calibration using a mirror to calibrate the two Kinects without a shared view
To find relative position (pose) between the two Kinects
Use a mirror at front and then back position, so the two Kinects can see the same dual-face (front back pattern are same) checker board
The checker board becomes the calibrated object , so the problem is solved
back Kinect
front Kinect
The calibration task: to find relative position (pose) translation and rotation between the two Kinects .
3D computer vision techniques v.6f

94. Rotating head with back-to-back Kinect
Each Kinect can only scan 70 degrees
Need to combine multiple models to become a full 360 degrees model
Solution:
Rotates 45 degrees each step
Capture 3D models (front, back) at each step
Combine results at the end
70 degrees scanning range
back Kinect
front Kinect
3D computer vision techniques v.6f

95. 3D computer vision techniques v.6f
Combined results using a rotating head with back-to-back Kinect cameras
3D computer vision techniques v.6f

96. 3D computer vision techniques v.6f
END
3D computer vision techniques v.6f