1. Panoramic Stereo Vision of Cooperative Mobile Robots for Localizing 3D Moving Objects
Zhigang Zhu, K. Deepak Rajasekar
Allen R. Hanson, Edward M. Riseman
Department of Computer Science
University of Massachusetts at Amherst
Funded by
AFRL/IFTD F (SAFER)
DARPA/ITO DABT (SDR Multi-Robot)
DARPA/ITO Mobile Autonomous Robot S/W (MARS)
2018/11/10
Omnidirectional Vision UMASS-Amherst

2. Cooperative Panoramic Stereo
Outline
Motivation
Panoramic Vision Sensor
Cooperative Panoramic Stereo
3D Match Algorithm
Experimental Results
Conclusion & Discussion
2018/11/10
Omnidirectional Vision UMASS-Amherst

3. Omnidirectional Vision 2000 UMASS-Amherst
1. Motivation
Task: Human Search and Rescue
multiple moving platforms - cooperation
distributed sensors- self calibration
multiple moving objects & rapid response
unknown environments
Research Issues: Panoramic Stereo
panoramic stereo geometry
dynamic calibration of cameras on moving platforms
view planning - adaptive viewpoints and baselines
real-time detection and 3D localization
match primitives and algorithms btw. 2 views with large perspective distortions
2018/11/10
Omnidirectional Vision UMASS-Amherst

4. 2. Panoramic Imaging Sensor
- geometric mathematical model
for image transform & calibration
p p1
pinhole P1 P B O C
Ellipsoidal mirror
Hyperboloidal mirror
panoramic annular lens (PAL)
* 40 mm in diameter, C-mount
* view: H: 360, V: -15 ~ +20
* single view point (O)
2018/11/10
Omnidirectional Vision UMASS-Amherst

5. Omnidirectional Vision 2000 UMASS-Amherst
2.1 Cylindrical panoramic un-warping
Two Steps:
(1). Center determination
(2) Distortion rectification
2-order polynomial approximation
2018/11/10
Omnidirectional Vision UMASS-Amherst

6. 2.2. Virtual cylindrical camera calibration
Find three parameters by three points:
- Effective Focal Length Fv
- Horizon Circle v0
- Camera Height: H0 H v D O v0
ground plane Fv H0
2018/11/10
Omnidirectional Vision UMASS-Amherst

7. Omnidirectional Vision 2000 UMASS-Amherst
3. Panoramic Stereo
Cooperative panoramic stereo is constructed by cameras on two moving platforms
Four Issues:
(1) Self-Calibration by seeing each other
_ Baseline B and relative viewpoints
(2) Distance by Triangulation
- Match the images of a target
(3) Error Analysis and View Planning
- What is the best view configuration?
(4) Size-Ratio Method
- Alternative way in co-linearity
Target
Baseline
Camera 1
Camera 2
Image 2
Image 1
2018/11/10
Omnidirectional Vision UMASS-Amherst

8. Omnidirectional Vision 2000 UMASS-Amherst
3.1. Dynamic Calibration
Image size of known cylinder determines baseline
Image bearings determine relative orientation
PAL 2
PAL 1
Why cylinder?
- view-invariant
- easy to detect
2018/11/10
Omnidirectional Vision UMASS-Amherst

9. An example: self-calibration and triangulation
Pano 1:
Image of the 2nd robot
Images of a person
Pano 2:
Image of the 1st robot
Results: B = 180 cm, D1 = 359 cm, D2 = 208 cm
2018/11/10
Omnidirectional Vision UMASS-Amherst

10. 3.2. Error Analysis and View Planning
Three error sources
Calibration error:
the baseline error is roughly proportional to the square of the baseline itself
Matching error:
view difference in O1 and O2 will introduce a "matching error"
Triangulation error
decreases for long baseline and good viewing geometry, but meanwhile the calibration error and matching error may increase
QUESTION:
For a given camera/target distance,where should the second camera be placed to minimize triangulation error?
2018/11/10
Omnidirectional Vision UMASS-Amherst

11. Best viewpoint and baseline: error map
Optimization problem of two variables: B and 1
-8 m
8 m 10 20
>45
O2? 30
Error level
Error map when O2 in different locations
2018/11/10
Omnidirectional Vision UMASS-Amherst

12. Omnidirectional Vision 2000 UMASS-Amherst
Best viewpoint and baseline : three conclusions
(1) Accuracy in distance is not simply proportional to baseline in the case of dynamic calibration
m m m
(11.5R) (34R) (64R) D1 1
68.0
59.0
27.8 D1 B
0 2 m m
2.9 m
2.1 m
1.2 m
(2) The two curves can be used to find the best viewpoints & baselines for difference distances
(3) There is a relatively large region with errors that are less than twice the minimum error
2018/11/10
Omnidirectional Vision UMASS-Amherst

13. 3.3. Size-Ratio method in co-linearity
Width D2 D1
Camera 1
Camera 2
Object
Sizes of an object in a pair of images tell the distances
2018/11/10
Omnidirectional Vision UMASS-Amherst

14. Omnidirectional Vision 2000 UMASS-Amherst
4. 3D Match Algorithm
Performed on objects extracted from images
Four steps:
(1) Moving object detection and tracking
(2) Head detection and localization
(3) Stereo match based on 3D features
(4) Improving match by temporal tracking
2018/11/10
Omnidirectional Vision UMASS-Amherst

15. Omnidirectional Vision 2000 UMASS-Amherst
4.1. Object Detection &Tracking
Detection and tracking multiple moving people
by motion analysis and region grouping
2018/11/10
Omnidirectional Vision UMASS-Amherst

16. 4.2. Head detection and localization
Head location is one of the reliable primitives for 3D match
- usually visible
- easy to detect
- symmetric
>>Further extension:
appearance-based partial match
2018/11/10
Omnidirectional Vision UMASS-Amherst

17. 4.3. Stereo Match based on 3D features
Why not 2D match? (1) large perspective distortion
(2) low image resolution
Matching primitives of an object blob :
(1) Intensity of blob (2) bearing of head ->D
(3) width of blob ->W (4) point at top of head -> H
2018/11/10
Omnidirectional Vision UMASS-Amherst

18. Omnidirectional Vision 2000 UMASS-Amherst
[3D match algorithm]
(1) Measures for each assumed match:
- Intensity similarity: ris
Are image intensities consistent?
- Ray convergence: rrc
Do rays thru. two images converge in 3D space?
- Width consistency: rwc
Are image widths consistent with assumed geometry?
- Height consistency: rhc
Are image heights consistent with assumed geometry? O2 O1
Image 1
Image 2
object 2
object 1
(2). Overall Match “Goodness”
Choose maximum
remove object images from match hypothesis
repeat
(3). Match Selection
2018/11/10
Omnidirectional Vision UMASS-Amherst

19. 4.4. Current work: Improving match using temporal match consistency
Time T
m(i’,j’) in
Frame t-1
Best temporal match rt(i,i') i' j' i
Spatio match rs(i,j) in frame t
Best temporal match rt(j,j') j
Improved measure:
2018/11/10
Omnidirectional Vision UMASS-Amherst

20. Omnidirectional Vision 2000 UMASS-Amherst
5. Experimental Results
Stationary platforms but dynamic calibration
Three processes:
two clients: people detection & tracking
the server: 3D match & GUI
Synchronized image capture
network time
light signal and camera sync clock
Tests: people walked on a pre-defined track
one person: performance of localization using triangulation & size-ratio
two person: performance of 3D match for multi-objects
Platform1 :
Detection
& Tracking
Platform 2 :
Server:
3D Match & GUI
3D estimates
Camera 1
Camera 2
internet
2018/11/10
Omnidirectional Vision UMASS-Amherst

21. Omnidirectional Vision 2000 UMASS-Amherst
One -person sequence (256 frames, 5 Hz)
Image 1
camera1
camera 2
2D map of 50cm grid
Walked along a rectangular path
6 turn around
Distance estimates using two methods
Co-linearity
Triangulation
Better estimation here, error ~ ±10 cm
Image 2
2018/11/10
Omnidirectional Vision UMASS-Amherst

22. Omnidirectional Vision 2000 UMASS-Amherst
two -person sequence (200+ frames )
Image 1
camera1
camera 2
Two people
opposite direction
the same path
move forward,
turn, meet,
apart...
- 3D match
- inaccurate OK
- track when meet
- exclude false
- consistent !
5% match error
Image 2
2018/11/10
Omnidirectional Vision UMASS-Amherst

23. Omnidirectional Vision 2000 UMASS-Amherst
Error case 1: detecting error causes inaccurate size measure
Image 1
camera1
camera 2
Match “goodness”
Image 2
Image 1
Improvement:
Appearance-based partial match?
Image 2
2018/11/10
Omnidirectional Vision UMASS-Amherst

24. Omnidirectional Vision 2000 UMASS-Amherst
Error case 2: “Greedy” Match causes “winner takes all”
Image 1
camera1
camera 2
Match “Goodness”
Image 2
Image 1
A global optimazation algorithm (e.g. Dynamic Programming) will fix this kind of error
Image 2
2018/11/10
Omnidirectional Vision UMASS-Amherst

25. Omnidirectional Vision 2000 UMASS-Amherst
6. Conclusion & Discussions
Conclusion:
cooperative panoramic stereo : view planning & adaptive baseline
dynamic calibration method : moving platforms
3D match algorithm: in perspective distortion and low resolution
real-time implementation: Now 5Hz
Further work:
view planning : moving platforms or multiple camera scheduling
general framework for match and track: Bayesian Network?
incorporate pan/tilt/zoom cameras with panoramic cameras
accurate self-calibration
accurate localization
human identification by face recognition
2018/11/10
Omnidirectional Vision UMASS-Amherst