ASU MAT 591 Image Processing Science and Robotic Vision Rod Pickens Principal Research Engineer Lockheed Martin, Incorporated

Signals and Processing Analog and discrete signals Dimensionality of signals 1-D signals Sounds (temporal), echocardiogram, seismic signal 2-D signals (this presentation) Images (spatial) 3-D signals Video sequences of images (spatial and temporal) Signal processing Synthesize and analyze signals Filter signals using low-pass, band-pass, and high-pass filter Modify signals such as warp, delay, stretch, rotate, shrink, … Restore and enhance signals Recognize patterns and detect signals

Signal Processing: Now Animal Robotic Touch Touch Vision Vision Taste Taste Hearing Hearing Smell Smell

The Processing Analogy

Analysis and Synthesis of Light Fourier Synthesis White Light In Fourier Analysis White Light Out Inverse Functions

Fourier Transforms are Inverse Functions

Inverse Functions Derivative Inv Fourier Trans Inv Radon Trans Warp Correction Integral Fourier Transform Radon Transform Warp Data

Filtering White Light In Red Light Out Filtering removes all but red colors Red Light Out

Television Television Stations 3, 5, 6, 13, 15, … Channel 6 Television Filtering removes all but Channel 6 Channel 6

Television Television Stations 3, 5, 6, 13, 15, … Channel 15 Filtering removes all but Channel 15 Channel 15

Radio Radio Stations Radio Stations 91.5, 96.9, 100.7 Station 100.7 Filtering removes all but Station 100.7

Radio Radio Stations Radio Stations 91.5, 96.9, 100.7 Station 96.9 Filtering removes all but Station 96.9

Scene of a Room: walls, books, desks, chairs, windows,… Vision Scene of a Room: walls, books, desks, chairs, windows,… Robot vision Book Filtering removes all but a book

Scene of a Room: walls, books, desks, chairs, windows,… Vision Scene of a Room: walls, books, desks, chairs, windows,… Scene of a Room Robot vision Table Filtering removes all but a table

Graphics to build a scene Synthesis Descriptor of scene is D(w) All Room Contents Scene of a Room

Filter that eliminates less important data. Data compression Signal Filter that eliminates less important data. Approximation of Signal

Filter that eliminates less important data. Data compression goal Signal Approximation of Signal Filter that eliminates less important data.

An Example of a Processing Architecture

The Example Architecture Data Format Format Correct Errors Communications Correct Errors Preprocess Preprocess Normalize Remove Noise Remove Distortions Restore Remove Sensor Effects Restore Analyze Decompose Signals Analyze Recognize Label Signals Recognize Descriptions Will Discuss in more detail!

Preprocess Data Format Correct Errors Preprocess Preprocess Restore Normalize Remove Noise Remove Distortions Analyze Recognize Descriptions

Fourier Based Noise Filtering Noisy Input Image Fourier Analysis Mostly Noise so is Zeroed Clearer Output Image Mostly Signal Fourier Synthesis and Filter the Noise Fourier Transform From Jason Plumb at http://noisybox.net/weblog/

Filtering and Enhancing Data Math to follow From Mathworks homepage at http://www.mathworks.com/

Filtering: Analysis Image Analysis

Filtering: Removing Noise Image Filtering: removes noise

Filtering: Synthesis Image Synthesis Enhanced

Filtering Enhanced Filtering: removes noise Image Synthesis Analysis

Enhancing the Data: Linear map I2 = m* I1 Enhance (stretch) Using Linear Mapping I=Intensity I1 p(I1) Input Image Intensity Histogram I2 p(I2) Output Image Intensity Histogram (more contrast)

Warping data Suppose we have unwanted camera motion. From Mathworks homepage at http://www.mathworks.com/

We can correct motion errors if we know motion model. Warping data We can correct motion errors if we know motion model. From Mathworks homepage at http://www.mathworks.com/

Warping data From Mathworks homepage at http://www.mathworks.com/

Warping Correction is an Inverse Function

Linear Algebra to Flip x1 y1 x2 y2

Linear Algebra to Flip x1 y1 x1 x2 x2=- x1 y1 y2 y2=y1 x2 y2

Linear Algebra to Flip y1 I(x1,y1) y2 y2=y1 x1 y1 x2 x2 y2 x1 x2=- x1

Linear Algebra to Flip y1 I(x1,y1) y2 y2=y1 x1 y1 x2 y2 x1 x2=- x1 x2

Linear Algebra to Flip y1 I(x1,y1) y2 y2=y1 x1 y1 x2 y2 y2 y2 y2 x1 x2=- x1 x2 x2 x2 x2 I(x2,y2)=I(f(x1),g(y1))

Linear Algebra to Flip y1 y1 y1=y2 x1 y2 x1 y2 x2 x1=- x2 x2 I(x2,y2)

Linear Algebra to Flip y1 I(f-1(x2), g-1(y2)) y1 y1=y2 x1 y2 x1 y2 x2 x1=- x2 x2 I(x2,y2)

Linear Algebra to Flip y1 y2 y2=y1 x1 y1 x2 y2 x1 x2=- x1 x2 I (x2,y2) I (x1,y1)=I(f-1(x2), g-1(y2)) y2 y2=y1 x1 y1 x2 y2 x1 x2=- x1 x2 I (x2,y2)

Linear Algebra to Flip and Shrink y1 x1 y2 x2

Linear Algebra to Flip and Shrink y1 y2 y2 = -0.5 * y1 x1 y1 x2 y2 x2 = 0.5 * x1 x1 x2

Correcting warped data (camera motion) If we can determine f(), g(), f-1(), and g-1(), then we can correct camera motion! From Mathworks homepage at http://www.mathworks.com/

Restoration Data Format Correct Errors Preprocess Restore Restore Remove Sensor Effects Analyze Recognize Descriptions

Restoring data for smear, optics,… From Mathworks homepage at http://www.mathworks.com/ Smear and optics can be viewed as filters that can degrade an image! Uses Linear Systems Theory Next

Restoring data for smear, optics,… From Mathworks homepage at http://www.mathworks.com/ Restoration Uses Linear Systems Theory Next

Restoration: Analysis Image Analysis

Filtering: Removing Smear Image Smr-1(wx,wy) is a filter that removes smear or restores the original object.

Filtering Smear inverted as a filter Image Object Image Restored to best look like original Object

Restoring data for smear, optics,… From Mathworks homepage at http://www.mathworks.com/ Uses Linear Systems Theory Image(wx,wy) Next

Restoring data for smear, optics,… From Mathworks homepage at http://www.mathworks.com/ Smr(wx,wy)*Image(wx,wy) Uses Linear Systems Theory Image(wx,wy) Next

Restoring data for smear, optics,… From Mathworks homepage at http://www.mathworks.com/ Smr(wx,wy)*Image(wx,wy) Uses Linear Systems Theory Image(wx,wy) *Smr-1(wx,wy)* Smr(wx,wy) Image(wx,wy)= Image(wx,wy) *1(wx,wy ) Image(wx,wy)

Synthesis and Analysis Data Format Correct Errors Preprocess Restore Synthesize Analyze Analyze Recognize Descriptions Decompose / Compose Signals - Transforms: Fourier, SVD, Wavelets - Statistical Analysis: parametric and non-parametric

Fourier Transform Fourier Synthesis White Light In White Light Out Fourier Analysis White Light Out

Fourier Transform Magnitude Phase From Wolfram homepage at http://documents.wolfram.com Magnitude Phase

Radon Transform From Mathworks homepage at http://www.mathworks.com/

Wavelet Transform From Wolfram homepage at http://documents.wolfram.com

Many kinds of transforms Common Transforms Fourier Discrete fourier Cosine Sine Hough Hadamard Slant Karhunen-Loeve Fast KL SVD Sinusoidal Many kinds of transforms

Statistics From Mathworks homepage at http://www.mathworks.com/

Recognition Data Format Correct Errors Preprocess Restore Analyze Recognize Recognize Descriptions Label Signals - Signal Detection - Pattern Recognition - Artificial Intelligence

* Features are mathematical measurements Pattern Recognition Analysis * Features are mathematical measurements Class 2 (rose) Class 1 (daisy) Feature 1 Feature 1 Class 3 (sun flower) Feature 2 * Feature 2 Transforms: Fourier, Wavelet, … Statistics: mean, st. dev, … Shape: Fourier, Hough, Moments Texture: Cooccurrence, Eigen Filters, … Analysis Tools Features Feature 1: Hough measure Feature 2: 3rd Eigen Filter Classification Bayesian Neural nets Nearest neighbors Linear

Mathematical Decisions Class 1 is z f2 z z z z z z o How do we separate the classes? z z z o z z z o z o o z z o o o o o o f1 o o o o o o o Class 2 is o

Mathematical Decisions Class 1 is z f2 z z z z z z o z z z o z z Linear decision z o z o o z z o o o o o o f1 o o o o o o o Class 2 is o

Mathematical Decision Class 1 is z f2 z z z z z z o z z z o z z Linear decision z o z o o z z o o o o o o f1 o o o o o o o Class 2 is o

Mathematical Decision Class 1 is z f2 z z z z z z o z z z o z z Quadratic decision z o z o o z z o o o o o o f1 o o o o o o o Class 2 is o

Mathematical Decision Class 1 is z f2 z z z z z z z z z z z z z z z f1

Mathematical Decision f2 o o o o o o o o o o f1 o o o o o o o Class 2 is o

Mathematical Decision Class 1 is z f2 z z z z z z o z z z o z z z o z o o -1 3 z z o o o o o o f1 o o o o o o o Class 2 is o

Isolate Object: Segmentation Analysis Synthesis From Mathworks homepage at http://www.mathworks.com/

Analyze Object: Features - Length - Width - Contour - Orientation - Edges Skeleton - Texture Details - Intensity From Mathworks homepage at http://www.mathworks.com/

Matched Filtering (registration) Input Image or Iin(x,y) From Mathworks homepage at http://www.mathworks.com/

Matched Filtering (registration) Input Image or Iin(x,y) Exemplar (reference) or Iref(x,y) From Mathworks homepage at http://www.mathworks.com/

Matched Filtering (registration) Input Image or Iin(x,y) Exemplar (reference) or Iref(x,y) error x x2 From Mathworks homepage at http://www.mathworks.com/

Matched Filtering (registration) Input Image or Iin(x,y) Exemplar (reference) or Iref(x,y) error x Actually search form min of x,y simultaneously! x2 From Mathworks homepage at http://www.mathworks.com/

Image Processing: Summary Data Format Format Correct Errors Communications Correct Errors Preprocess Preprocess Normalize Remove Noise Remove Distortions Restore Remove Sensor Effects Restore Analyze Decompose Signals Analyze Recognize Label Signals Recognize Descriptions

References Fundamentals of Image Processing by Jain Digital Image Analysis by Gonzalez and Wintz Pattern Recognition by Fukunaga Pattern Recognition Principles Tou and Gonzalez Detection, Estimation, and Modulation Theory by Van Trees Pattern Classification by Duda and Hart Robot by Hans Moravec (graphics from www.amazon.com)

Signal Processing: 50 years from now Evolved Robotic Touch Touch Hmmm. Vision Vision Vision Taste Taste Hearing Hearing Smell Smell

Signal Processing: 50 years from now Evolved Robotic Touch Touch Wow! Vision Vision Vision Taste Taste Hearing Hearing Smell Smell

Signal Processing: 50 years from now Evolved Robotic Touch Touch I see, therefore, am I? Hmmm. Vision Vision Vision Taste Taste Hearing Hearing Smell Smell