Image Processing - in short Image in  image out image = 2D array of pixels (pixel = img[x,y]) color : 3 values / pixel gray : 1 value / pixel value usually 0.0 to 1.0 (float) or 0 to 255 (byte) No 3D scene no sub-pixel sampling must work with given pixels (no extra info)

3 Kinds of Processes Local Neighborhood Global out[x,y] depends on in[x,y] only Neighborhood out[x,y] depends on in[x,y] and pixels near in[x,y] Global out[x,y] depends on ALL pixels in out

Local Processes out[x,y] = f ( in[x,y] ) adjust colors, brightness correct for acquisition bias eg, camera sensor with non-linear response no spatial filtering no sharpening, blurring

Brightness adjustment out[x,y] = in[x,y] +  (clamp to 0-1 range) in K > 0 K < 0

Quantization out[x,y] = f( in[x,y] ) fixed threshold ordered dither 1 if in[x,y]>thresh[x,y] then out=1 else out=0 out in 1

Contrast stretch out[x,y] = f( in[x,y] ) more contrast less contrast 1 shallow slope steep slope in 1 in 1

Gamma Correction out[x,y] = in[x,y] in >1 <1 1 1 out out in 1 in 1 in 1

Histogram Equalization Frequent pixels  higher contrast original (low contrast) high contrast (detail lost) histogram equalized (more details)

Histogram Equalization 1 : get color frequencies p(v) = (#pixels with color v) / (#total pixels) 2 : compute cummulative probability C(v) = fraction of colors  v = p(x) , for x v 3 : apply equalization out = C(in)

Histogram Equalization more frequent in Pixel frequency less frequent less steep Cummulative pixel probability out more steep

Luminance (colorgrey) Y-component of CIE X-Y-Z color in: 3 values/pixel (R,G,B) out: 1 value/pixel Y  0.3 R + 0.6 G + 0.1 B

Hue Rotation 1: in(R,G,B) (H,S,V) 2: H  H +  3: (H,S,V)  out(R,G,B)

Adjust Saturation 1: in(R,G,B) (H,S,V) 2: S  S + S 3: (H,S,V)  out(R,G,B) S < 0 in S > 0

Neighborhood Processes out[x,y] = f (pixels near in[x,y]) linear f( ) = linear combination of pixel colors non-linear f( ) = other function non-linear combination procedure

Linear filtering Equivalent to convolution Integral form: out(x) = integral{ in(x - t) × g(t) × dt } out = in * g Discrete form: out[x,y] = i,j in[x + i, y + j] × g[i, j] out[x,y] = weighted sum of pixels g( ) = “filter kernel”

Filter Kernels Matrix of weights example: box filter 1D: g(x) = 1/a if (0  x  a) 2D: g(x,y) = 1/a2 if (0  x  a) and (0  y  a) discrete box filter (3 × 3), a=3, a2=9 g[x,y] =  1 1 1   1 1 1  ÷ 9  1 1 1  1/a a

Discrete 3 × 3 Box Filter in out ×1/9 × 1/9 × 1/9 × 1/9 × 1/9 × 1/9 out[x,y] × 1/9 × 1/9 × 1/9

Discrete Convolution (2n+1) × (2n+1) out[x,y] = in[x,y] * g[x,y] s = 0; for (i=-n; i<=n; i++) for (j=-n; j<=n; j++) s += in[x+i, y+j] * g[i,j]; out[x,y] = s; 3 × 3 : n=1

Convolution examples Smoothing (box, gaussian filters) Edge detection (derivative, Sobel, laplacian) Edge enhancement Gradient

Remapping Convolution may change range Example : edge detection in : originally 0 to 1 out : should be 0 to 1 too Example : edge detection in pixel: 0 to 1 filtered pixel: -1 to +1 out = (filtered + 1) / 2

Smoothing sharp edge brighter darker 10 11 10 9 10 5 6 5 4 5 scanline in: 1 × 3 box filter scanline out: (details lost) 10 10 10 10 8 7 5 5 5 5 softened edge

5 × 5 Box Filter in out Boundaries blurred, detail lost

Gaussian Filter 1D: g(x)= 1/(2) × e-x2/22 2D: g(x,y)= 1/(2) × e- (x2+y2) /22 4 Discrete 5x5 gaussian: (=1.4)

Gaussian Filter in 5x5 box filter 5x5 gaussian (edges sharper)

Edge Detection edge f(x) df(x)  f(x+h) - f(x) dx h scanline out[x,y] = in[x+1,y] - in[x,y] g[x,y] =  0 0 0  0 -1 1   0 0 0 df(x) dx (must remap result to range 0-1)

Derivative sharp edges brighter darker brighter 10 11 10 9 10 5 6 5 10 scanline in: values 0-10 -1 1 g[x]= = {in[x] - in[x-1]}/2 + 5 scanline out: 5 5 4 4 5 2 5 4 7 5 dark pixel bright pixel

Better Edge Detection Use neighboring scanlines avoid single-pixel edges g[x,y] =  -1 0 1  -1 0 1  (vertical edges)  -1 0 1 g[x,y] =  1 1 1   0 0 0  (horizontal edges)  -1 -1 -1

Sobel Edge Detection Neighboring scanlines have less weight g[x,y] =  -1 0 1  -2 0 2  (vertical edges)  -1 0 1 g[x,y] =  1 2 1   0 0 0  (horizontal edges)  -1 -2 -1 endpoints of edges less blurred

Horizontal Derivative in simple df dx derivative amplifies noise! (why?) Sobel df dx

Gradient P = [P/x, P/y] P/x = horizontal derivative P is height of terrain P is “uphill” vector |P| is terrain slope P/x = horizontal derivative P/y = vertical derivative |P| = gradient strength ignores orientation uphill P/y P/x

Oriented Edges in |P| , Sobel (all edges) simple df dx (vertical edges only)

Gradient + Blur in |P| , Sobel features lost gaussian blur, then |P| Sobel less noise

Laplacian Non-oriented edges 2P = 2P/x2 + 2P/y2 (2nd derivatives) scalar, not a vector Estimate for 2nd derivative g[x,y] =  0 1 0  1 -4 1|  0 1 0 where did this come from?

Estimated 2nd Derivative df = f '  f(x+h) - f(x) dx h d2f = f ''  f '(x+h) - f '(x) dx2 h  f(x+2h) - 2f(x+h) + f(x) h2  f(x+h) - 2f(x) + f(x-h) (shifted left) h2 d2f  f(y+h) - 2f(y) + f(y-h) (in Y direction) dy2 h2

2nd Derivative Edges original: two edges gaussian blur inflection points: f '' = 0 first derivative second derivative: zero crossing at edges

2nd Derivative Edges More robust to noise Smoothing : removes noise but blurs edge 2nd derivative: curvature edge at inflection point zero crossing

Edge Enhancement AKA “Sharpening” out[x,y] = out[x,y] - α × edge[x,y] Find edges, amplify them, add them back out[x,y] = out[x,y] - α × edge[x,y] g[x,y] =  0 -α 0 -α 1+4α -α | (laplacian)  0 -α 0

Edge Enhancement in out

Non-linear Processes de-speckle median bilateral filter

De-speckle get in[x,y] and neighbours compute standard deviation  if ( < threshold) out[x,y] = mean value (ie, apply blur) removes single-pixel noise dust spots noise removal

Despeckle in out

Median Filter get in[x,y] and neighbours out[x,y] = median value also removes noise preserves edges (does not bring in new colors) better than blurring

Median Filter median, 3x3 neighbourhood in median, 5x5 neighbourhood

Fourier Filtering Fourier Transform modify in[], apply inverse F.T. in[x]  F.T.  in[] convert to frequency image in[x] = color of xth pixel in[] = strength of  frequency modify in[], apply inverse F.T. uses WHOLE image per pixel global process

Frequency filtering sinusoid in(x,y) = 1+sin(1/x) filter: low-pass high-pass band-pass

Fourier Transform F.T. in two copies of frequency zero frequency at center repeating bands higher freqs. on edges

Fourier Low-pass filter F.T. in high freqs suppressed out inverse F.T. blurred, ringing

Fourier High-pass filter F.T. in low freqs suppressed out inverse F.T. edges detected

Remove Periodic Artifacts frequency peaks in F.T. inverse F.T. out peaks zeroed out

Arithmetic operations (Binary images) a b NOT b a OR b a AND b a XOR b a - b