EN94941 Follow The Leader Using Vision Only Bax Smith, BSc., BEng., MSc. Candidate
EN94942 Introduction The purpose of this presentation is to introduce and give some background information on my project for this course
EN94943 Topics of Discussion Project Description Project Background The Guts of the Project Current State of Things Literature Review Input From Audience
EN94944 Project Description Implementation of a Real-Time follow-the-leader vehicle Two vehicles: one leads, one follows Has many of the elements for this course: real-time, discrete-event, continuous dynamics, multi-agent, hierarchical and time-critical
EN94945 Project Background – Why this project? A subset of my Masters Project: Follow-the-Leader Using Vision under different weather conditions Has aspects of Image Processing, A.I. and Control It’s Cool!
EN94946 Project Background - Applications My purpose: Mining Vehicles Automated Highways – Smart Cars Automatic Target Recognition – Like they use in Smart Missiles!! Security Systems – Identify the intruder The Ultimate Paint-Ball Weapon!!
EN94947 The Guts of the Project Hardware: Jagan Wagons, PIII 600 Laptop, Webcam Software: Win2000, VC++6.0 Program Cycle Grab an image from Webcam – Small Colored! Use LVQ Neural Net to isolate vehicle from other objects using color Use Min Bounding Box to scale to classifier input size Use Support Vector Machines (SVM) to recognize object – NECEC 2001 Plug! Call appropriate motor commands – Maybe PID, but I doubt it
EN94948 Current State of Things Term 8 Project was similar…sort of. Have developed/implemented LVQ Nets for object detection based on color – worked very well Have developed/implemented SVMs for 3D Object Recognition – NECEC 2001 Plug #2
EN94949 Term 8 Project
EN LVQ Neural Net Input Map to Subclass Map to Class Output R G B dR mR lR dG mG lG dB mB lB Y R G B Y
EN Why LVQ? Each color is a 3D vector ie RGB All Reds are not the same LVQ stores general direction of a color
EN Training Extract color from sample images of environment
EN Support Vector Machines
EN Literature Review General ntroip/title.html Neural Nets nn_all/ann_all.html Support Vector Machines
EN Input From Audience Suggestions Comments Critisms
EN Follow The Leader Using Vision Only Bax Smith, BSc., BEng., MSc. Candidate Part II – Midterm Progress
EN Review of Project Hardware: Jagan Wagons, PIII 600 Laptop, Webcam Software: Win2000, VC++6.0 Program Cycle Grab an image from Webcam – Small Colored! Use LVQ Neural Net to isolate vehicle from other objects using color Use Min Bounding Box to scale to classifier input size Use Support Vector Machines (SVM) to recognize object Call appropriate motor commands
EN The Setup
EN Automatic Target Recognition Imaging Sensors Image Preprocessing Object Segmentation Feature Extraction Object Classification
EN Imaging Sensor Imaging Sensors Image Preprocessing Object Segmentation Feature Extraction Object Classification CCD TV Low-light Level TV (LLTV) Forward Looking Infrared (FLIR) Synthetic Aperture Radar (SAR) Moving Target Indicator Radar (MTI Radar) Laser Radar (LADAR)
EN Webcam Pros: Cheap Easy to Use Free Software!!!!! Cons: Low Resolution Slow Software HARD to Use/Buggy
EN Webcams - Software OpenCV Video For Windows (VFW) API Ton of Wrappers for VFW Microsoft Vision SDK Easy to use Low Level – Made for Realtime Apps
EN Object Segmentation Imaging Sensors Image Preprocessing Object Segmentation Feature Extraction Object Classification Preprocessing: Focus Contrast Enhancement Detection/Segmentation: Foreground/Background Separation Silhouetting
EN LVQ Neural Network
EN Inside LVQ Net
EN LVQ – Competitive Layer
EN LVQ Competitive Layer - Training CL = - || w – p || If p classified correctly w(q) = w(q-1) + a( p(q) – w(q-1) ) Else w(q) = w(q-1) - a( p(q) – w(q-1) ) w1w1 w2w2 w3w3
EN LVQ – Linear Layer Non-Target Target
EN After LVQ Target Non-Target
EN Feature Extraction Imaging Sensors Image Preprocessing Object Segmentation Feature Extraction Object Classification Transforms the segmented image into a form that can be used as input to classifier – Feature Vector
EN Minimum Bounding Box
EN Object Classification Imaging Sensors Image Preprocessing Object Segmentation Feature Extraction Object Classification Segmented Objects are classified based on their associated feature vector Object Classification
EN Support Vector Machines Optimal Separating Hyperplane for linearly separable problems Extending OSH to Nonlinear Using Kernels
EN Optimal Separating Hyperplane
EN Optimal Separating Hyperplane
EN OSH - Mathematics Given: (x 1,y 1 ) … (x n, y n ) | x i is vector of reals, y = {-1, +1} Construct Hyperplanes: H 1 : + b = +1 H 2 : + b = -1 OSH: + b = 0
EN Support Vectors
EN Formulating Constraints From: H 1 : + b = +1 H 2 : + b = -1 We get: y i ( + b) >= 1
EN Distance Between Hyperplanes Just subtract H 2 from H 1 : = 2 And normalize: = 2/||w||
EN Bigger Margin = Better Classifier Therefore: Maximize: 2/||w|| Constrained by: y i ( + b) >= 1 Or equivalently: Minimize: ½ ||w|| 2 Constrained by: y i ( + b) >= 1
EN QP Problem - Lagrange Saddle Point Minimize wrt w and b Maximize wrt alphas At Saddle Point dL/dw, dL/db = 0:
EN The Dual Problem Substituting in the derivatives: Maximize, subject to: alphas>0 and Solve for alphas
EN Only Support Vectors Matter Lagrange also gives us: Which shows that only the data points that lie on the outer hyperplanes will have non-zero lagrange multipliers
EN Solution to QP Classification of vector, v, is the sign of:
EN Extending OSH to SVM OSH works for linearly separable data therefore transform data into a space where it is linearly separable
EN Using a Kernel Function Kernel Functions are easily added to the Lagrangian: And classification function
EN Don’t Need to Compute Transform We just need the Kernel Function: For example: x = (x 1, x 2 ); z = (z 1, z 2 ); 2 = (x 1 z 1 + x 2 z 2 ) 2 = x 1 2 z x 2 2 z x 1 z 1 x 2 z 2 =
EN Examples of Kernels K(x,z) = d K(x,z) = exp(-||x-z|| 2 /2 )
EN Wheelchair Control Send drive commands to Wheelchair
EN Follow The Leader Using Vision Only Bax Smith, BSc., BEng., MSc. Candidate Episode III – The Final Presentation
EN Today Quick Review Demo Setup Agents Supervisory Controller
EN Review of Project Hardware: Jagan Wagons, PIII 600 Laptop, Webcam Software: Win2000, VC++6.0 Program Cycle Grab an image from Webcam – Small Colored! Use LVQ Neural Net to isolate vehicle from other objects using color Use Min Bounding Box to scale to classifier input size Use Support Vector Machines (SVM) to recognize object Call appropriate motor commands
EN The Setup Supervisor
EN The Agents Robot1 Robot2
EN Agents in MATLAB function state = getRobotState(state, sl, dsl, sr, dsr) idle = 1; fol_leader = 2; fol_r = 3; switch state case idle, if sl == 1, state = fol_leader; end; case fol_leader, if sr == 1, state = fol_r; elseif dsl == 1, state = idle; end; case fol_r, if dsr == 1, state = idle; end;
EN Object Segmentation Imaging Sensors Image Preprocessing Object Segmentation Feature Extraction Object Classification Preprocessing: Focus Contrast Enhancement Detection/Segmentation: Foreground/Background Separation Silhouetting
EN Segmentation via LVQ
EN Object Classification Imaging Sensors Image Preprocessing Object Segmentation Feature Extraction Object Classification Segmented Objects are classified based on their associated feature vector Object Classification
EN Classification via SVM
EN Creating the Plant Sync Robot1Robot2
EN Robert Plant
EN Creating the Spec
EN Creating the Supervisor Sync PlantSpec
EN The Supervisor
EN CONTROL.TXT c1 Control data are displayed by listing the plant state followed by the supervisor state (if different from the plant state) where disabling and/or forcing occur, together with the events which must be disabled or forced there. Control Data: PLANT: [fol_leader1,fol_r1] SUPER: [[fol_leader1,fol_r1],1] DELAY: sr2 PLANT: [fol_r2,fol_leader2] SUPER: [[fol_r2,fol_leader2],1] DELAY: sr1 PLANT: [fol_leader1,idle_r2] SUPER: [[fol_leader1,idle_r2],1] DELAY: sl2 PLANT: [idle_r1,fol_leader2] SUPER: [[idle_r1,fol_leader2],1] DELAY: sl1 c1 printed.
EN Supervisor in MATLAB function [sl1, dsl1, sr2, dsr2, sl2, dsl2, sr1, dsr1] = runSupervisor(stateOfR1, sl1, dsl1, sr2, dsr2, stateOfR2, sl2, dsl2, sr1, dsr1) idle = 1; fol_leader = 2; fol_r = 3; if (stateOfR1 == fol_leader) & (stateOfR2 == idle), sl2 = 0; dsl2 = 1; end; if (stateOfR1 == idle) & (stateOfR2 == fol_leader), sl1 = 0; dsl1 = 1; end; if (stateOfR1 == fol_leader) & (stateOfR2 == fol_r), sr2 = 0; dsr2 = 1; end; if (stateOfR1 == fol_r) & (stateOfR2 == fol_leader), sr1 = 0; dsr1 = 1; end;
EN Demos W/O Supervisory Control With Supervisory Control