1. EN94941 Follow The Leader Using Vision Only Bax Smith, BSc., BEng., MSc. Candidate

2. EN94942 Introduction The purpose of this presentation is to introduce and give some background information on my project for this course

3. EN94943 Topics of Discussion Project Description Project Background The Guts of the Project Current State of Things Literature Review Input From Audience

4. EN94944 Project Description Implementation of a Real-Time follow-the-leader vehicle Two vehicles: one leads, one follows Has many of the elements for this course: real-time, discrete-event, continuous dynamics, multi-agent, hierarchical and time-critical

5. EN94945 Project Background – Why this project? A subset of my Masters Project:  Follow-the-Leader Using Vision under different weather conditions Has aspects of Image Processing, A.I. and Control It’s Cool!

6. EN94946 Project Background - Applications My purpose: Mining Vehicles Automated Highways – Smart Cars Automatic Target Recognition – Like they use in Smart Missiles!! Security Systems – Identify the intruder The Ultimate Paint-Ball Weapon!!

7. EN94947 The Guts of the Project Hardware: Jagan Wagons, PIII 600 Laptop, Webcam Software: Win2000, VC++6.0 Program Cycle  Grab an image from Webcam – Small Colored!  Use LVQ Neural Net to isolate vehicle from other objects using color  Use Min Bounding Box to scale to classifier input size  Use Support Vector Machines (SVM) to recognize object – NECEC 2001 Plug!  Call appropriate motor commands – Maybe PID, but I doubt it

8. EN94948 Current State of Things Term 8 Project was similar…sort of. Have developed/implemented LVQ Nets for object detection based on color – worked very well Have developed/implemented SVMs for 3D Object Recognition – NECEC 2001 Plug #2

9. EN94949 Term 8 Project

10. EN949410 LVQ Neural Net Input Map to Subclass Map to Class Output R G B dR mR lR dG mG lG dB mB lB Y R G B Y

11. EN949411 Why LVQ? Each color is a 3D vector ie RGB All Reds are not the same LVQ stores general direction of a color

12. EN949412 Training Extract color from sample images of environment

13. EN949413 Support Vector Machines

14. EN949414 Literature Review General  http://www-2.cs.cmu.edu/~cil/v-pubs.html  http://www.ece.ubc.ca/~irenek/techpaps/i ntroip/title.html  http://www.itsonline.com/index.htm Neural Nets  http://www.maths.uwa.edu.au/~rkealley/a nn_all/ann_all.html Support Vector Machines  http://www.kernel-machines.org/

15. EN949415 Input From Audience Suggestions Comments Critisms

16. EN949416 Follow The Leader Using Vision Only Bax Smith, BSc., BEng., MSc. Candidate Part II – Midterm Progress

17. EN949417 Review of Project Hardware: Jagan Wagons, PIII 600 Laptop, Webcam Software: Win2000, VC++6.0 Program Cycle  Grab an image from Webcam – Small Colored!  Use LVQ Neural Net to isolate vehicle from other objects using color  Use Min Bounding Box to scale to classifier input size  Use Support Vector Machines (SVM) to recognize object  Call appropriate motor commands

18. EN949418 The Setup

19. EN949419 Automatic Target Recognition Imaging Sensors Image Preprocessing Object Segmentation Feature Extraction Object Classification

20. EN949420 Imaging Sensor Imaging Sensors Image Preprocessing Object Segmentation Feature Extraction Object Classification CCD TV Low-light Level TV (LLTV) Forward Looking Infrared (FLIR) Synthetic Aperture Radar (SAR) Moving Target Indicator Radar (MTI Radar) Laser Radar (LADAR)

21. EN949421 Webcam Pros:  Cheap  Easy to Use  Free Software!!!!! Cons:  Low Resolution  Slow  Software HARD to Use/Buggy

22. EN949422 Webcams - Software OpenCV Video For Windows (VFW) API Ton of Wrappers for VFW Microsoft Vision SDK  Easy to use  Low Level – Made for Realtime Apps

23. EN949423 Object Segmentation Imaging Sensors Image Preprocessing Object Segmentation Feature Extraction Object Classification Preprocessing: Focus Contrast Enhancement Detection/Segmentation: Foreground/Background Separation Silhouetting

24. EN949424 LVQ Neural Network

25. EN949425 Inside LVQ Net

26. EN949426 LVQ – Competitive Layer

27. EN949427 LVQ Competitive Layer - Training CL = - || w – p || If p classified correctly  w(q) = w(q-1) + a( p(q) – w(q-1) ) Else  w(q) = w(q-1) - a( p(q) – w(q-1) ) w1w1 w2w2 w3w3

28. EN949428 LVQ – Linear Layer Non-Target Target

29. EN949429 After LVQ Target Non-Target

30. EN949430 Feature Extraction Imaging Sensors Image Preprocessing Object Segmentation Feature Extraction Object Classification Transforms the segmented image into a form that can be used as input to classifier – Feature Vector

31. EN949431 Minimum Bounding Box

32. EN949432 Object Classification Imaging Sensors Image Preprocessing Object Segmentation Feature Extraction Object Classification Segmented Objects are classified based on their associated feature vector Object Classification

33. EN949433 Support Vector Machines Optimal Separating Hyperplane for linearly separable problems Extending OSH to Nonlinear Using Kernels

34. EN949434 Optimal Separating Hyperplane

35. EN949435 Optimal Separating Hyperplane

36. EN949436 OSH - Mathematics Given:  (x 1,y 1 ) … (x n, y n ) | x i is vector of reals, y = {-1, +1} Construct Hyperplanes:  H 1 : + b = +1  H 2 : + b = -1  OSH: + b = 0

37. EN949437 Support Vectors

38. EN949438 Formulating Constraints From:  H 1 : + b = +1  H 2 : + b = -1 We get:  y i ( + b) >= 1

39. EN949439 Distance Between Hyperplanes Just subtract H 2 from H 1 :  = 2 And normalize:  = 2/||w||

40. EN949440 Bigger Margin = Better Classifier Therefore:  Maximize: 2/||w||  Constrained by: y i ( + b) >= 1 Or equivalently:  Minimize: ½ ||w|| 2  Constrained by: y i ( + b) >= 1

41. EN949441 QP Problem - Lagrange Saddle Point  Minimize wrt w and b  Maximize wrt alphas At Saddle Point dL/dw, dL/db = 0:

42. EN949442 The Dual Problem Substituting in the derivatives: Maximize, subject to:  alphas>0 and Solve for alphas

43. EN949443 Only Support Vectors Matter Lagrange also gives us: Which shows that only the data points that lie on the outer hyperplanes will have non-zero lagrange multipliers

44. EN949444 Solution to QP Classification of vector, v, is the sign of:

45. EN949445 Extending OSH to SVM OSH works for linearly separable data therefore transform data into a space where it is linearly separable

46. EN949446 Using a Kernel Function Kernel Functions are easily added to the Lagrangian: And classification function

47. EN949447 Don’t Need to Compute Transform We just need the Kernel Function: For example:  x = (x 1, x 2 ); z = (z 1, z 2 );  2 = (x 1 z 1 + x 2 z 2 ) 2 = x 1 2 z 1 2 + x 2 2 z 2 2 +2x 1 z 1 x 2 z 2  =

48. EN949448 Examples of Kernels K(x,z) = d K(x,z) = exp(-||x-z|| 2 /2  )

49. EN949449 Wheelchair Control Send drive commands to Wheelchair

50. EN949450 Follow The Leader Using Vision Only Bax Smith, BSc., BEng., MSc. Candidate Episode III – The Final Presentation

51. EN949451 Today Quick Review Demo Setup Agents Supervisory Controller

52. EN949452 Review of Project Hardware: Jagan Wagons, PIII 600 Laptop, Webcam Software: Win2000, VC++6.0 Program Cycle  Grab an image from Webcam – Small Colored!  Use LVQ Neural Net to isolate vehicle from other objects using color  Use Min Bounding Box to scale to classifier input size  Use Support Vector Machines (SVM) to recognize object  Call appropriate motor commands

53. EN949453 The Setup Supervisor

54. EN949454 The Agents Robot1 Robot2

55. EN949455 Agents in MATLAB function state = getRobotState(state, sl, dsl, sr, dsr) idle = 1; fol_leader = 2; fol_r = 3; switch state case idle, if sl == 1, state = fol_leader; end; case fol_leader, if sr == 1, state = fol_r; elseif dsl == 1, state = idle; end; case fol_r, if dsr == 1, state = idle; end;

56. EN949456 Object Segmentation Imaging Sensors Image Preprocessing Object Segmentation Feature Extraction Object Classification Preprocessing: Focus Contrast Enhancement Detection/Segmentation: Foreground/Background Separation Silhouetting

57. EN949457 Segmentation via LVQ

58. EN949458 Object Classification Imaging Sensors Image Preprocessing Object Segmentation Feature Extraction Object Classification Segmented Objects are classified based on their associated feature vector Object Classification

59. EN949459 Classification via SVM

60. EN949460 Creating the Plant Sync Robot1Robot2

61. EN949461 Robert Plant

62. EN949462 Creating the Spec

63. EN949463 Creating the Supervisor Sync PlantSpec

64. EN949464 The Supervisor

65. EN949465 CONTROL.TXT c1 Control data are displayed by listing the plant state followed by the supervisor state (if different from the plant state) where disabling and/or forcing occur, together with the events which must be disabled or forced there. Control Data: PLANT: [fol_leader1,fol_r1] SUPER: [[fol_leader1,fol_r1],1] DELAY: sr2 PLANT: [fol_r2,fol_leader2] SUPER: [[fol_r2,fol_leader2],1] DELAY: sr1 PLANT: [fol_leader1,idle_r2] SUPER: [[fol_leader1,idle_r2],1] DELAY: sl2 PLANT: [idle_r1,fol_leader2] SUPER: [[idle_r1,fol_leader2],1] DELAY: sl1 c1 printed.

66. EN949466 Supervisor in MATLAB function [sl1, dsl1, sr2, dsr2, sl2, dsl2, sr1, dsr1] = runSupervisor(stateOfR1, sl1, dsl1, sr2, dsr2, stateOfR2, sl2, dsl2, sr1, dsr1) idle = 1; fol_leader = 2; fol_r = 3; if (stateOfR1 == fol_leader) & (stateOfR2 == idle), sl2 = 0; dsl2 = 1; end; if (stateOfR1 == idle) & (stateOfR2 == fol_leader), sl1 = 0; dsl1 = 1; end; if (stateOfR1 == fol_leader) & (stateOfR2 == fol_r), sr2 = 0; dsr2 = 1; end; if (stateOfR1 == fol_r) & (stateOfR2 == fol_leader), sr1 = 0; dsr1 = 1; end;

67. EN949467 Demos W/O Supervisory Control With Supervisory Control