Panoramic Stereo Vision of Cooperative Mobile Robots for Localizing 3D Moving Objects Zhigang Zhu, K. Deepak Rajasekar Allen R. Hanson, Edward M. Riseman Department of Computer Science University of Massachusetts at Amherst zhu@cs.umass.edu http://www.cs.umass.edu/~zhu Funded by AFRL/IFTD F30602-97-2-0032 (SAFER) DARPA/ITO DABT63-99-1-0022 (SDR Multi-Robot) DARPA/ITO Mobile Autonomous Robot S/W (MARS) 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Cooperative Panoramic Stereo Outline Motivation Panoramic Vision Sensor Cooperative Panoramic Stereo 3D Match Algorithm Experimental Results Conclusion & Discussion 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Omnidirectional Vision 2000 UMASS-Amherst 1. Motivation Task: Human Search and Rescue multiple moving platforms - cooperation distributed sensors- self calibration multiple moving objects & rapid response unknown environments Research Issues: Panoramic Stereo panoramic stereo geometry dynamic calibration of cameras on moving platforms view planning - adaptive viewpoints and baselines real-time detection and 3D localization match primitives and algorithms btw. 2 views with large perspective distortions 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

2. Panoramic Imaging Sensor - geometric mathematical model for image transform & calibration p p1 pinhole P1 P B O C Ellipsoidal mirror Hyperboloidal mirror panoramic annular lens (PAL) * 40 mm in diameter, C-mount * view: H: 360, V: -15 ~ +20 * single view point (O) 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Omnidirectional Vision 2000 UMASS-Amherst 2.1 Cylindrical panoramic un-warping Two Steps: (1). Center determination (2) Distortion rectification 2-order polynomial approximation 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

2.2. Virtual cylindrical camera calibration Find three parameters by three points: - Effective Focal Length Fv - Horizon Circle v0 - Camera Height: H0 H v D O v0 ground plane Fv H0 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Omnidirectional Vision 2000 UMASS-Amherst 3. Panoramic Stereo Cooperative panoramic stereo is constructed by cameras on two moving platforms Four Issues: (1) Self-Calibration by seeing each other _ - Baseline B and relative viewpoints (2) Distance by Triangulation - Match the images of a target (3) Error Analysis and View Planning - What is the best view configuration? (4) Size-Ratio Method - Alternative way in co-linearity Target Baseline Camera 1 Camera 2 Image 2 Image 1 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Omnidirectional Vision 2000 UMASS-Amherst 3.1. Dynamic Calibration Image size of known cylinder determines baseline Image bearings determine relative orientation PAL 2 PAL 1 Why cylinder? - view-invariant - easy to detect 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

An example: self-calibration and triangulation Pano 1: Image of the 2nd robot Images of a person Pano 2: Image of the 1st robot Results: B = 180 cm, D1 = 359 cm, D2 = 208 cm 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

3.2. Error Analysis and View Planning Three error sources Calibration error: the baseline error is roughly proportional to the square of the baseline itself Matching error: view difference in O1 and O2 will introduce a "matching error" Triangulation error decreases for long baseline and good viewing geometry, but meanwhile the calibration error and matching error may increase QUESTION: For a given camera/target distance,where should the second camera be placed to minimize triangulation error? 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Best viewpoint and baseline: error map Optimization problem of two variables: B and 1 -8 m 8 m 10 20 >45 O2? 30 Error level Error map when O2 in different locations 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Omnidirectional Vision 2000 UMASS-Amherst Best viewpoint and baseline : three conclusions (1) Accuracy in distance is not simply proportional to baseline in the case of dynamic calibration 0 2 m 6 m 11.5 m (11.5R) (34R) (64R) D1 1 68.0 59.0 27.8 D1 B 0 2 m 6 m 11.5 2.9 m 2.1 m 1.2 m (2) The two curves can be used to find the best viewpoints & baselines for difference distances (3) There is a relatively large region with errors that are less than twice the minimum error 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

3.3. Size-Ratio method in co-linearity Width D2 D1 Camera 1 Camera 2 Object Sizes of an object in a pair of images tell the distances 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Omnidirectional Vision 2000 UMASS-Amherst 4. 3D Match Algorithm Performed on objects extracted from images Four steps: (1) Moving object detection and tracking (2) Head detection and localization (3) Stereo match based on 3D features (4) Improving match by temporal tracking 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Omnidirectional Vision 2000 UMASS-Amherst 4.1. Object Detection &Tracking Detection and tracking multiple moving people by motion analysis and region grouping 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

4.2. Head detection and localization Head location is one of the reliable primitives for 3D match - usually visible - easy to detect - symmetric >>Further extension: appearance-based partial match 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

4.3. Stereo Match based on 3D features Why not 2D match? (1) large perspective distortion (2) low image resolution Matching primitives of an object blob : (1) Intensity of blob (2) bearing of head ->D (3) width of blob ->W (4) point at top of head -> H 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Omnidirectional Vision 2000 UMASS-Amherst [3D match algorithm] (1) Measures for each assumed match: - Intensity similarity: ris Are image intensities consistent? - Ray convergence: rrc Do rays thru. two images converge in 3D space? - Width consistency: rwc Are image widths consistent with assumed geometry? - Height consistency: rhc Are image heights consistent with assumed geometry? O2 O1 Image 1 Image 2 object 2 object 1 (2). Overall Match “Goodness” Choose maximum remove object images from match hypothesis repeat (3). Match Selection 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

4.4. Current work: Improving match using temporal match consistency Time T m(i’,j’) in Frame t-1 Best temporal match rt(i,i') i' j' i Spatio match rs(i,j) in frame t Best temporal match rt(j,j') j Improved measure: 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Omnidirectional Vision 2000 UMASS-Amherst 5. Experimental Results Stationary platforms but dynamic calibration Three processes: two clients: people detection & tracking the server: 3D match & GUI Synchronized image capture network time light signal and camera sync clock Tests: people walked on a pre-defined track one person: performance of localization using triangulation & size-ratio two person: performance of 3D match for multi-objects Platform1 : Detection & Tracking Platform 2 : Server: 3D Match & GUI 3D estimates Camera 1 Camera 2 internet 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Omnidirectional Vision 2000 UMASS-Amherst One -person sequence (256 frames, 5 Hz) Image 1 camera1 camera 2 2D map of 50cm grid Walked along a rectangular path 6 turn around Distance estimates using two methods Co-linearity Triangulation Better estimation here, error ~ ±10 cm Image 2 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Omnidirectional Vision 2000 UMASS-Amherst two -person sequence (200+ frames ) Image 1 camera1 camera 2 Two people opposite direction the same path move forward, turn, meet, apart... - 3D match - inaccurate OK - track when meet - exclude false - consistent ! 5% match error Image 2 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Omnidirectional Vision 2000 UMASS-Amherst Error case 1: detecting error causes inaccurate size measure Image 1 camera1 camera 2 Match “goodness” 1 2 3 1 0.37 0.43 0.62 2 0.00 0.00 1.00 Image 2 Image 1 Improvement: Appearance-based partial match? Image 2 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Omnidirectional Vision 2000 UMASS-Amherst Error case 2: “Greedy” Match causes “winner takes all” Image 1 camera1 camera 2 Match “Goodness” 1 2 1 0.00 0.92 2 0.89 1.00 Image 2 Image 1 A global optimazation algorithm (e.g. Dynamic Programming) will fix this kind of error Image 2 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst

Omnidirectional Vision 2000 UMASS-Amherst 6. Conclusion & Discussions Conclusion: cooperative panoramic stereo : view planning & adaptive baseline dynamic calibration method : moving platforms 3D match algorithm: in perspective distortion and low resolution real-time implementation: Now 5Hz Further work: view planning : moving platforms or multiple camera scheduling general framework for match and track: Bayesian Network? incorporate pan/tilt/zoom cameras with panoramic cameras accurate self-calibration accurate localization human identification by face recognition 2018/11/10 Omnidirectional Vision 2000 UMASS-Amherst