Algorithm For Image Flow Extraction- Using The Frequency Domain

Image flow extraction what it is good for ? • early-warning systems • tracking & reconstructing • egomotion • visual segmentation • super resolution

Several methods: • differential based • region matching based • phase based • frequency based - spatiotemporal filtering

Motion in the frequency domain • the power spectrum of translating texture at velocity v = ( u,v) occupies a tilted plane in the frequency domain t = ux + vy • motion-sensitive gabor energy filters sample these planes efficiently Wt = UWx + VWy

3D Gabor Energy Filters • a Gausssian multiplied by sine wave or cosine wave (BPF) • (t0 x0 y0) - the center frequency, maximum filter output • maximum filter output for translating texture with spatial frequency (x0 y0) and such velocity v = ( u,v ) that the temporal frequency is t = ux0 + vy0 = t0

• a family of filters : 12 filters with same bandwith, (x02 + y02)0 • a family of filters : 12 filters with same bandwith, (x02 + y02)0.5= const, but different spatial and temporal frequencies • each velocity corresponds to different tilt of the plane thus to different distribution of filters outputs

The algorithm gaussian pyramid HPF Image sequence gabor filtering motion energy convolution measured energy theoretical energy pattern flow Pattern flow

Gaussian Pyramid • level L : sample density reduction by 2L & convolution with gaussian of size (2L+2 -3)x (2L+2 -3) • sample reduction- expands image spectrum • convolution- lower levels higher frequencies enhancement, higher levels lower frequencies enhancement • high level-high velocity, low level-low velocity (t = ux) 0-1.25 level 0 1.25-2.25 level 1 2.25-5 level 2

FFT of image FFT of pyramided image

HPF Filtering • image motion characterized by changes • changes-higher frequencies, enhance changes by HPF FFT

Gabor Filtering • filtering with 12 gabor sine phase filters & 12 gabor cosine phase filters

Motion Energy • the sum of the squared output of sine phase filter plus the squared output of sine phase filter • measure of Gabor energy that is invariant to the phase of the signal

Convolution with Gaussian • gabor filters are localized both in space-time and frequency domains, thus the motion energy is also localized • convolution with gaussian - enhancing center values (most reliable) decreasing far away from center values (least reliable)

Measured Energy • convolution with gaussian (25x25x7) divide the image into 25x25 sized segments, each of one can move in a different velocity • the center value in the convulved motion energy segment is the energy caused by a moving object in that segment

Theoretical Energy • Parseval’s theorem is used to derive equation that predicts the energy of gabor filter in response to moving texture , Ri(u,v)

Pattren Flow • for each moving object : 12 measured energies & 12 predicted energies • least squares estimate for u and v minimizes the difference between the predicted and measured energies • minimizing : Gauss-Newton method ...

Results • accurate for textures in motion containing spatiotemporal frequencies near the center frequencies

pattern flow moving gaussians

Car moving at velocity (2,-0 Car moving at velocity (2,-0.2) pixel per frame, the algorithm result (1.93,-0.16)

The End Future Improvements • adaptive center frequencies choosing • combining pyramid levels • substitute gaussian convolution with different method - e.g relaxation labeling • different numerical methods for minimization • DSP realization The End