Image Processing Goal –Start simple: look at small windows –Identify useful image structures (‘Clues’ useful for recognizing objects) –Eliminate irrelevant aspects of image appearance (neglect appearance variations that don’t help to explain object identity) First step (usually) in any algorithm

We don’t “see” most information in image

+ Can add invisible change

We don’t “see” most information in image + = Can add invisible change

Important information small fraction of total Initial Image

Important information small fraction of total Initial Image Important information Boundary locations boundary sharpness size/direction of brightness change

Important information small fraction of total Initial Image Important information Boundary locations edge sharpness; size/direction of brightness change Reconstruct good semblance of original image just using info at boundaries Reconstruction Elder: IJCV 34(2/3), 97–122 (1999)

Image Processing Goal –Small windows –Identify useful image structures (`clues’) –Eliminate irrelevant aspects Example 1 – Emphasize signal; suppress noise

Image Processing Goal –Small windows –Identify useful image structures (`clues’) –Eliminate irrelevant aspects Example 2 –Detect “boundaries” (jumps in brightness) –De-emphasize slow variations in brightness

Why look at small windows? Local redundancy –Neighboring image points strongly related (points on same object have similar color, texture)  Don’t need to know brightness at every pixel  Isolate just useful information (eg, average brightness in window) What’s behind the patch?

Why look at small windows? Details are important! Boundary signals presence of some object

Note All image locations are equal –Mannequin equally likely to be anywhere in image

Note All image locations are equal* –Mannequin equally likely to be anywhere in image *Not quite true for photographs (people center and compose images)

Conclusion Analyze image over every small window...

Conclusion Analyze image over every small window Easiest: linear weighted sum of brightness in each window...

Conclusion Analyze image over every small window Easiest: linear weighted sum of brightness in each window  Convolution (filtering)...

What is image filtering? For each pixel, modify value based on values of pixels nearby

Linear Filtering New pixel value = weighted sum of nearby pixels Uses: –Integrate information over regions –Clean up noisy images –Analyze image at different resolutions –Detect image patterns, brightness boundaries –Connects to Fourier analysis

Example (animated) 1D “image” Average Sum nearby pixels with weights Filtered Image I' Averaging makes filtered curve smoother Called mask, kernel, filter…

2D Linear Filtering x 3 filter (averaging or box filter) Image Note: computer vision filters usually have small number of pixels

Image 2D Linear Filtering Filtered Image

Image 2D Linear Filtering Filtered Image

Image 2D Linear Filtering Filtered Image

Image 2D Linear Filtering Filtered Image

Image 2D Linear Filtering Filtered Image

Image 2D Linear Filtering Filtered Image

Image 2D Linear Filtering Filtered Image

Image 2D Linear Filtering Filtered Image

Filtering Equations (correlation!) Original image filter (or mask, kernel…) Filtered image

Filtering Equations (correlation!) Original image filter Filtered image is a “little image” containing the weights with which the pixels of are summed Procedure For each filter position Multiply filter and image entries in corresponding positions Sum and record result at position under filter center

Filtering Equations (correlation!) Original image Filtered image filter

Filtering Equations (correlation!) Original image Filtered image filter Index ranges give filter size, here (2N+1) x (2N+1)

Filtering Equations (correlation!) Original image Filtered image Easiest: use odd sized filters, symmetric index range [–N,N] so filter center at (0,0) Index ranges give filter size, here (2N+1) x (2N+1) filter

One Dimension (correlation!) Filtered image For filter of size (2N+1) centered on (0,0)

Convolution Like Correlation with Filter Reversed 1D 2D

Convolution Like Correlation with Filter Reversed 1D 2D ‘-’ instead of ‘+’ crucial change!

Convolution Like Correlation with Filter Reversed Many nice properties-a kind of multiplication 1D 2D

Convolution From now on, linear filtering equals convolution (unless I say otherwise)

Convolution procedure Original image filter Convolved image Procedure For each filter position Flip (reflect) filter in both x and y directions Multiply filter and image entries in corresponding positions Sum and record result at position under filter center

Convolution: symmetric form (doesn’t work for correlation) 1D

Convolution: symmetric form (doesn’t work for correlation) 1D {

Convolution: symmetric form (doesn’t work for correlation) 1D {

Convolution: symmetric form (doesn’t work for correlation) 1D Convention: extend filter. Assign W(i)=0 for out-of-range

Convolution: symmetric form (doesn’t work for correlation) 1D Convention: extend filter. Assign W(i)=0 for out-of-range

Convolution: symmetric form (doesn’t work for correlation) 1D This extends for any number of convolutions (again, with convention that everything out of range is zero for W and V)

Convolution: symmetric form (doesn’t work for correlation) 2D

Convolution: symmetric form (doesn’t work for correlation) 2D

Convolution like multiplication! Commutative Associative Distributive (linear)

Convolution like multiplication! Commutative

Convolution like multiplication! Commutative

Convolution like multiplication! Associative

0 Convolution: More properties Shift invariance Compare vs

Convolution: More properties Shift invariance  Fourier Transform connection  Convolution equivalent* to multiplication after Fourier Transform!!

Some Examples Filter mask 1.0 Filtered image

Some Examples Filter mask 1.0 Plot of mask (weights) Filtered image

Some Examples Filter mask 1.0 Filtered image

Some Examples In 1D the plot of W would look like this. Filter mask Filtered image

Filter mask 1.0

Filter mask 1.0

Next example

Filter mask 1.0 Shifted from (0,0)

Filter mask 1.0

Filter mask 1.0

Filter mask New example

Filter mask

Averaging filter  Blur

OriginalFiltered (1D) Filter mask

OriginalFiltered (1D) Filter mask

OriginalFiltered (1D) Filter mask

OriginalFiltered (1D) Filter mask

OriginalFiltered (1D) Filter mask Note how the original sharp transition gets blurred

Filter mask Warm up…

1 Filter mask Equivalent

Filter mask Warm up…

2.0 Filter mask

2.0 Filter mask -

1.89 (a peak in a trough)

Original in 1D Filter mask

Original in 1D Filter mask

Original in 1D Filter mask

Original in 1D Filter mask

Original in 1D Filter mask

? Different example

Different example

1D example 1 =? o

1D example o 1 = Only the jump survives!

A Detail: how to deal with border

Dealing with image border What happens here? Trying to sum over pixels outside the image

Dealing with image border Various choices: 1) Pad with zeros…

Dealing with image border Various choices: 1)Pad with zeros 2)Duplicate border

Dealing with image border Various choices: 1)Pad with zeros 2)Duplicate border

Dealing with image border Various choices: 1)Pad with zeros 2)Duplicate border 3)Wrap (not so important in this class)

Dealing with image border Various choices: 1) Pad with zeros 2) Duplicate border 3) Wrap 4) Or just crop (don’t try to extend original image; filtered image is smaller than original) Image Filtered Image

Filtering to reduce noise Noise = what we don’t care about –Assume random noise added at each pixel –Reduce noise by averaging over windows Random noise from different pixels tends to cancel Signal not much affected (image is redundant---pixel’s neighbors have similar brightness)

Simple Additive Noise

( Image = signal + random noise) Assume –No dependence of noise size on signal –Expected value of noise is zero –Noise added at each pixel independently –Type of noise is the same at all pixels. or, more precisely:

Averaging filter to reduce noise

Averaging Filter: Definition Mask has positive weights summing to 1 Replaces each pixel with weighted average over its neighborhood Example: BOX filter has all weights equal /9

Averaging several times...

Image gets smoother; noise in patches instead of speckles

For smooth signals, averaging doesn’t affect the signal much and improves Signal/Noise Example: Image = Constant + Noise –Average image over n-pixel window

For smooth signals, averaging doesn’t affect the signal much and improves Signal/Noise Example: Image = Constant + Noise –Average image over n-pixel window Smaller because of cancellations

Averaging a noisy image You can copy this code to matlab ( substitute your image for prowler.pgm) I=imread('prowler.pgm'); imagesc(I), figure(gcf ) % “,” separates commands, figure(gcf) brings current figure to front (gcf ==“get current figure”) colormap gray % shows image as black and white S= size(I); % size of image I In=double(I)+randn(S(1),S(2))*50; % Adds noise; randn creats a matrix of the given size with Gaussian random entries N(0,1) % `double’ forces image I to type double imagesc(In),figure(gcf) Is= conv2(double(In),ones(3,3)/9);imagesc(Is),figure(gcf) % convolve image with box filter. ``ones” creates a matrix % of given size containing ones Is= conv2(double(Is),ones(3,3)/9);imagesc(Is),figure(gcf) pause % Wait for keyboard input before going on % what does averaging again do? REPEAT Is= conv2(double(Is),ones(3,3)/9);imagesc(Is),figure(gcf), pause close all %close all figure displays

Averaging reduces noise k=100; T=1000; % k= image length (for 1D noise image) % T = number of trials RandIm =randn(k,1); plot(RandIm), figure(1), title('1D Noise Image') AverageNoise = sum((1/k)*RandIm), pause figure(2) hist(sum((1/k)*randn(k,T))); axis([ inf]); xlabel([int2str(T),' trials of noise averaged over ', int2str(k),' pixels']); title('histogram')

Averaging for a normal image... (no noise)

What else is averaging good for? Remember original goal of image processing –Identify useful image structures (`clues to objects’) –Throw away less useful parts of image

What else is averaging good for? Remember original goal of image processing –Identify useful image structures (`clues to objects’) –Throw away less useful parts of image After smoothing… – Each pixel contains average brightness in its neighborhood. –This may be all you need to know about the neighborhood  To save only important information, keep just one pixel per neighborhood.

Original Smoothed with Keeping only every other pixel (along x and y directions) New image ¼ size All important information preserved

Image Compression Original Subsampled (every 7 th pixel)

Image Compression Original Subsampled (every 7 th pixel) Smoothed, then sampled

Averaging: problems

Example: Averaging/smoothing with Box Filter Star Star “averaged” with Box

Example: Averaging/smoothing with Box Filter Star Star “averaged” with Box  Not very smooth!

Example: Averaging/smoothing with Box Filter

Artifact from sharp edge of box filter

Improved averaging filter

Smoothing as Inference of Signal True signal is smooth. To infer a pixel’s “true” brightness without noise, look at brightness of nearby pixels Signal + Noise signal plus noise

Smoothing as Inference of Signal Infer brightness at central pixel only using nearby pixels Signal + Noise signal plus noise Mask matched to signal

Smoothing as Inference of Signal Infer brightness at central pixel only using nearby pixels  Adjust size of averaging mask (Match signal smoothness: reduce noise without oversmoothing signal) Signal + Noise signal plus noise Mask matched to signal

Smoothing as Inference of Signal Mask size should match signal smoothness Signal signal plus noise Mask + Noise

Smoothing as Inference of Signal mask size should match signal smoothness Closer pixels are more similar Similarity falls off smoothly with distance Signal + Noise signal plus noise mask

Smoothing as Inference of Signal mask size should match signal smoothness Closer pixels are more similar Similarity falls off smoothly with distance  Make mask weights decrease smoothly with distance from filter center Signal + Noise signal plus noise mask

From box filter to Gaussian

Averaging with Gaussian mask Gaussian weights nearby pixels more Smooth roll off in weights

Gaussian Smoother (Rotationally Symmetric) Gaussian filter gives reasonable model of blurring (eg from lens)

Star Smoothing with Gaussian Blurry Star (Gaussian Smoothed)

Smoothing with Gaussian

Box Smoothing Gaussian Smoothing Artifact from sharpness of box filter No artifact

Gaussian Filter: useful property Convolving 2 Gaussians yields* new Gaussian! Recall Gaussian Definition:

Gaussian Filter: useful property Convolving 2 Gaussians yields* new Gaussian! Recall Gaussian Definition: New Gaussian is wider, with (more blur)

Gaussian Filter: useful property Convolving 2 Gaussians yields* new Gaussian! Recall Gaussian Definition: New Gaussian is wider, with (more blur) * Strictly true only for continuous functions, not discrete filters defined on pixels

Gaussian Filter: useful property Convolving 2 Gaussians yields* new Gaussian! Recall Gaussian Definition: New Gaussian is wider, with (more blur) * Strictly true only for continuous functions, not discrete filters defined on pixels See later for equations

Gaussian Filter Associativity of convolution implies Several filterings with small Gaussians equivalent to single filtering with larger Gaussian

This generalizes… Convolving two smoothing filters gives a smoothing filter with more blur. Can get a lot of smoothing by convolving many times with a small (low blur) filter This is faster to compute…

Gaussian Filter and Scale Space Want to consider image at different resolutions (levels of detail, different scales)  Smooth image with different sized smoothers Branches,leaves Bushes Tree, ground, sky foreground, background

Gaussian Filter and Scale Space Want to consider image at different resolutions (levels of detail, different scales)  Smooth image with different sized smoothers Branches,leaves Bushes Tree, ground, sky foreground, background Human vision does this! (double face)

Gaussian Filter and Scale Space Want to consider image at different resolutions (levels of detail, different scales)  Smooth image with different sized smoothers Scale Space = same image viewed at different resolutions (size scales) Branches,leaves Bushes Tree, ground, sky foreground, background

Gaussian Filter and Scale Space Want to consider image at different resolutions (levels of detail, different scales)  Smooth image with different sized smoothers Scale Space = same image viewed at different resolutions (size scales) With Gaussians, only resolution matters… Whatever the sequence of Gaussian smoothing operations applied to an image, the result is the same as smoothing once with a single Gaussian of the appropriate size/resolution. (With other filters, two different series of smoothing operations can give different results at the same resolution) Branches,leaves Bushes Tree, ground, sky foreground, background

Gaussian Filter and Scale Space Scale Space (same image viewed at different resolutions) Many applications! –Fast image download ( Transmit low resolution, progressively upgrade to higher resolution) –Image editing/blending –Fast image searching ( search image first at low resolution, refine at higher resolution) –Image compression ….

Example: Image Blending

Implementation notes Gaussian has infinite size (G(x)>0 for all x) For efficiency, cut off filter at large x. What x? Rule of thumb: Use window with each side

Implementation note 2 Gaussian and BOX filters are separable

Implementation note 2 Gaussian and BOX filters are separable

Implementation note 2 Gaussian and BOX filters are separable  Can replace 2D convolution with two 1D ones: Much faster! 1. Convolve each row with 1D filter 2.Then convolve each column with 1D filter Reduces computation by factor width of filter

Image Separable Filtering (animated!) Filtered Image =

Image Separable Filtering (animated!) Filtered Image =

Aside: convolution for continuous functions discrete continuous

Aside: Continuous convolution Discrete (closer analog) continuous

Constants normalize the “weights” to 1, so Gaussian is an averaging filter. Above is for continuous functions. When you approximate Gaussian on grid of pixels, normalize so Note: normalization

Convolving 2 Gaussians See next page for equations for 1D convolution

Equations (Gaussian convolution in 1D) Completing square in exponent Change variable From normalization of Gaussian

Summary Filtering: analyze small image windows Linear filtering and correlation/convolution –Uses Smoothing/averaging/blurring Sharpening Boundary (edge) detection Next set of slides: pattern detection –Convolution: Technicalities Symmetric form Commutative, associative, etc. Dealing with boundary

Summary Smoothing –Uses Noise reduction (not so important for this class) Compression Scale space: analyze image at different resolutions (see later) Edge detection (see later) –Technicalities Gaussians (special nice properties) Normalize sum of weights to 1! Size of Gaussian mask (note: smaller gives faster computation) Separability Blurring a little many times  blurring a lot once