Pursuit Evasion Games and Multiple View Geometry René Vidal UC Berkeley Part I: Motivation, 10 minutes, 9 slides -> 8 Part II: Previous work, 35 minutes, 21 slides -> 20 Part II: Future work, 15 minutes, 8 slides -> 10

“Research Trends” Control Computer Vision Hybrid Systems, Embedded Systems Quantum Control, Multi-Agent Control Computer Vision Multi-body Structure from Motion Omnidirectional Vision Many more … Texture, segmentation and grouping, shape, stereo, illumination, recognition, real-time vision

Previous Work Robotics and Control Computer Vision Pursuit-Evasion Games (ICRA’01-CDC’01-TRA’01) Decidable Classes of Discrete Time Hybrid Systems (HSCC’00-CDC’01) Computer Vision Multiple View Geometry (IJCV’01-ECCV’02) Optimal Motion Estimation (ICCV’01) Planar motions with Small Baselines (ECCV’02) Camera Self-Calibration (ECCV’00-IJCV’01)

Pursuit-Evasion Games Cory Sharp, Omid Shakernia, David Shim, Jin Kim Shankar Sastry

Pursuit-Evasion Game Scenario Evade!

PEG: Map Building (Hespanha CDC’99) Given measurements build a probabilistic map Measurement step: model for sensor detection Prediction step: model for evader motion

PEG: Pursuit Policies (Hespanha CDC’99) Greedy Policy Pursuer moves to the reachable cell with the highest probability of having an evader The probability of the capture time being finite is equal to one (Hespanha CDC ’99) The expected value of the capture time is finite (Hespanha CDC ’99) Global-Max Policy Pursuer moves to the reachable cell which is closest to the cell in the whole map with the highest probability of having an evader May not take advantage of multiple pursuers (all the pursuers could move to the same place)

PEG: Control Architecture (ICRA’01) position of evader(s) position of obstacles strategy planner position of pursuers map builder communications network evaders detected obstacles pursuers positions Desired pursuers positions tactical planner trajectory regulation tactical planner & regulation actuator positions [4n] lin. accel. & ang. vel. [6n] inertial [3n] height over terrain [n] obstacles detected evaders detected vehicle-level sensor fusion state of helicopter & height over terrain obstacles detected control signals [4n] agent dynamics actuator encoders INS GPS ultrasonic altimeter vision Exogenous disturbances terrain evader

PEG: Vision System

PEG: Experimental Results (Summer’00)

PEG: Architecture Implementation (CDC’01) Navigation Computer Serial Vision Computer Strategic Planner Helicopter Control GPS: Position INS: Orientation Camera Control Color Tracking UGV Position Estimation Communication UAV Pursuer Map Building Pursuit Policies Communication Runs in Simulink Same for Simulation and Experiments TCP/IP Serial Robot Micro Controller Robot Computer Robot Control DeadReck: Position Compass: Heading Camera Control Color Tracking GPS: Position Communication UGV Pursuer UGV Evader

PEG: Experimental Results (Spring’01)

PEG: Experimental Results (Spring’01) Add simulation video before

PEG: Experimental Results (Spring’01) Pursuit Policy v.s. Vision System

Vision Based Landing

Landing: (Sharp & Shakernia ICRA’01)

PEG: Summary Proposed a probabilistic framework and a hierarchical control architecture for pursuit-evasion games Developed algorithms for Sensor fusion, helicopter control, vision based detection, vision based landing Implemented architecture with UGVs and UAVs in real-time

Multiple View Geometry The Multiple View Matrix Yi Ma (UIUC), Jana Kosecka (GMU), Shankar Sastry (UCB)

Multiple View Geometry (MVG)

Multiple View Geometry (MVG) Obtain camera motion and scene structure from multiple images of a cloud of 3D feature points

MVG: Literature Review Theory Two views: Longuet-Higgins’81, Huang & Faugeras’89, … Three views: Spetsakis et.al.’90, Shashua’94, Hartley’94, … Four views: Triggs’95, Shashua’00, … Multi views: Faugeras et.al.’95, Heyden et.al.’97,… Algorithms Euclidean: Maybank’93, Weng, Ahuja & Huang’93, … Affine: Quan & Kanade’96, … Projective: Triggs’96, … Orthographic:Tomasi & Kanade’92, …

MVG: Anatomy of cases (state of the art) surface curve line point theory algorithm practice Euclidean affine projective 2 views 3 views 4 views m views algebra geometry optimization

MVG: A need for unification Euclidean surface curve line point 2 views 3 views 4 views m views theory algorithm practice affine projective algebra geometry optimization rank deficiency of Multiple View Matrix

MVG: Formulation Homogeneous coordinates of a 3-D point Homogeneous coordinates of its 2-D image Projection of a 3-D point to an image plane Either Euclidean or Projective

MVG: Classical Approach Given corresponding images of points recover motion, structure and calibration from This set of equations is equivalent to

MVG: The Multiple View Matrix WLOG choose frame 1 as reference Theorem: [Rank deficiency of Multiple View Matrix] (generic) (degenerate)

MVG: Bilinear and Trilinear Constraints Multiple View Matrix implies bilinear constraints Multiple View Matrix implies trilinear constraints Constraints among more than three views are algebraically dependent (quadrilinear in particular)

MVG: Degenerate Cases Theorem [Uniqueness of the pre-image]: Given vectors with respect to camera frames, they correspond to a unique point in the 3-D space if rank(M) = 1. If rank(M) = 0, the point is determined up to a line on which all the camera centers must lie.

MGV: Line Features Point Features Line Features

MVG: Mixed Point and Line Features . . .

MGV: Planar Features Point Features Line Features Besides multilinear constraints, it simultaneously gives homography:

Multiple View Landing (ICRA’02)

MGV: Multiple View Planar Algorithm Results

Optimal Motion Estimation (ICCV’01) Noisy measurements: Minimize error: Subject to: Lagrange optimization + algebraic elimination:

Optimal Motion Estimation (ICCV’01) M is not a regular Euclidean space Can do Euclidean optimization, but need to project onto M There are optimization techniques for Stiefel manifolds Quadratic convergence is guaranteed if Hessian is non-degenerate [Smith and Brockett ’93]

Multiple View Matrix: Summary Points M implies bilinear and trilinear constraints Quadrilinear constraints do not exist Lines All indep. constraints are among 3 views Some are trilinear, some are nonlinear Mixed points and lines There are multilinear constraints among 2-3 views There are nonlinear constraints among 3-4 views

Current Work: SFM for curves Differentiating of a point on a curve intensity level sets region boundaries . . .

Current Work: SFM for curves Multiple view matrix for curves Gives rise to rank condition Gives rise to set of ODE’s

Current Work: Multi-Body SFM Geometry of multiple moving objects Given a set of corresponding image points, obtain: Number of evaders Evader to which each point belongs to (segmentation) Depth of each point Motion of each pursuer and evader Orthographic case (Costeira-Kanade’95) Multiple moving points (Shashua-Levin’01) Can the multiple view matrix approach be used for motion estimation and segmentation?

Current Work: Multi-Body SFM Generalized Multiple View Matrix [Ma et.al. ’01] Multiple moving points Projection

Current Work: Multi-Body SFM Linked Multiple Body Motion Estimation Infinitesimal motion case Ursella, Soatto and Perona ‘95 Bregler and Malik ‘98 Discrete motion case Generalized multiple view matrix Paden-Kahan inverse kinematic subproblems

Current Work: Multi-Body SFM [Vidal’01] Costeira-Kanade in perspective Form optical flow matrices Obtain number of objects Segment the points

Conclusions Computer Vision for Real-Time Control Pursuit Evasion Games Vision Based Landing A new approach to Multiple View Geometry Simple: just linear algebra Unifying: Euclidean, projective, 2 views, 3 views, multiple views, points, lines, curves, surface? New Constraints: There are nonlinear constraints