1. Design for Embedded Image Processing on FPGAs
Chapter 9
Geometric Transformations

2. Outline Two approaches to geometric transformation
Forward mapping
Reverse mapping
Separable mapping techniques
Performs mapping in two passes
Interpolation
Bilinear, bicubic, and cubic spline
Image registration
Feature based methods, SIFT
Area based methods
Applications

3. Geometric Transformation
Redefines the geometry of pixels in an image
Forward mapping
Output locations are defined as a function of the input locations
Reverse mapping
Input locations are a function of the output locations
Forward mapping
Reverse mapping
Input Image
Output Image

4. Geometric Transformations
Zoom
Rotation
Perspective
Distortion

5. Geometric Transformation
Requires a frame buffer to manage changed geometry
Cannot have both streamed input and output
Forward mapping:
Can work with streamed input
Reverse mapping:
Can produce a streamed output

6. Problems with forward mapping Two pass algorithms

7. Problems with Forward Mapping
Mapping specifies where each input pixel goes to
Mapped location may not be on an integer grid
Write the pixel value to the nearest location
Problem
Cannot just round mapped coordinates
Some output pixels would have no inputs, some have multiple

8. Problems with Forward Mapping
Need to map the whole pixel
Each output pixel is a weighted sum of all inputs which overlap it
Requires maintaining an output accumulator for each pixel to add all the input contributions
Problem for streamed input
Each input can map to many output pixels
Several clock cycles may be required to distribute input to all affected output pixels
This requires a complex caching arrangement
Determining the intersection between input and output pixels is complex and time consuming

9. Separable Mapping Transform image in two 1D passes
Row operation (each row)
1D transformation to get each pixel into the correct column
Column operation (each column)
1D transformation to get each pixel into the correct row

10. Two Pass Implementation
Input is row warped into one frame buffer
Column warp is applied to first image
Second image is loaded into second frame buffer
Image is streamed out in row order
Third image is warped into the same buffer
Requires bank-switched dual port memory

11. Two Pass Implementation
Hardware requirements may be reduced by processing every second image
Reuse warping engine for rows and columns
Only a single frame buffer is required
Otherwise same as previous slide
Each row or column processed independently
Can process multiple rows or columns in parallel
Limited by memory bandwidth

12. Limitations of Two Pass Approach
Cannot rotate close to 90°
Bottleneck problem: All pixels map to same column
Cannot be separated out again in second pass
Fold-over when mappings are not monotonic
Mapping is many-to-one
Limits range of mappings
Second mapping (column mapping) uses a mix of input and output coordinates
May not be possible to derive closed form mapping
Separable 1-D warps not equivalent to 2-D warp
Assumes pixels are infinitely thin

13. 1-D Forward Mapping Requires interpolation
Especially where scale factor is greater than 1
Otherwise get flat plateaus where a single input pixel spans several output pixels
Requires anti-alias filtering
Especially where scale factor is less than 1
Filter is spatially dependent on magnification
Algorithm:
Keep track of fractions of input and output pixels used
Each cycle either reads an input or writes an output
Based on if an output pixel is complete
FIFO on input and output smooth data flow for streamed operation

14. 1-D Forward Mapping

15. 1-D Forward Mapping
Speed operation by allowing both input and output cycles if input pixel completes the current output pixel

16. Alternative Algorithm
Output is a sum over fractions of pixels
Maintain a running sum (row integral)
Interpolate to give integral at output edges
Difference between these (normalised by width) is the output pixel value

17. Running Two Passes at Once
First pass as before
Result streamed to second pass
All columns operated on at once
Maintains state for each column in a data row buffer
Output to frame buffer is not in raster order
Depends on column warp

18. Avoiding aliasing Interpolation Bilinear Bicubic Cubic spline
Reverse Mapping
Avoiding aliasing
Interpolation
Bilinear
Bicubic
Cubic spline

19. Reverse Mapping Most common mapping when warping in software
Producing a streamed output
Want to know where output pixel comes from
Mapped pixel does not necessarily fall on grid
Read nearest pixel from frame buffer
Interpolate between available pixels
Some form of caching is required
Also require some form of anti-alias filter
Especially where magnification is less than 1
Frame buffer
Output
stream
Reverse mapping
Address

20. Using a Pyramid for Anti-aliasing
There is insufficient time to filter input while performing warp
Filtering is spatially dependent
Image pyramid filters and down- samples each layer by successive powers of 2
Requires 33% more storage
Can be constructed in a single pass through the image
Using for anti-aliasing
Select level for desired magnification
Tri-linear interpolation between levels

21. Summed Area Tables Table contains sum of all pixels above and left
May be formed recursively from a streamed input
Can give sum of any rectangular region with 4 accesses

22. Reverse Mapping
Generally mapping requires whole image in frame buffer first
If distortion is small, can begin producing output before complete image is loaded
Only a partial frame buffer required (using row buffers)
Keeping number small reduces both resources and latency

23. Interpolation Determines values at locations between pixels
Like filtering, but with spatially variant filter kernel

24. Bilinear Interpolation
Output value is weighted sum of 4 nearest pixels
Integer part gives pixel locations
Fractional part gives weights
A form of 2×2 filter except
Weights vary with location in image
Input is not streamed

25. Preloading the Cache
Accessing 4 pixels per clock cycle requires careful cache design
Preload cache before use
Load one row in advance
Either buffer mapping for reuse on next row
Or recalculate the mapping on the next row
Trade-off between computation and storage

26. Cache Preload Alternative

27. Bicubic Interpolation
Requires a 4×4 interpolation kernel
Kernel is separable, which reduces computation
In general this separability cannot be exploited by stream processing
In 1-D have piecewise cubic kernel with 4 sections
16 values must be read from cache or frame buffer for every output pixel
Generally requires partitioning of cache over several block RAMs to overcome bandwidth limitations
Kernel weights
Either calculated as needed
Precalculated and stored in a lookup table

28. Bicubic Interpolation

29. Cubic Spline Interpolation
Piecewise cubic kernel
Continuous in first and second derivatives
This gives large region of support
Although it does decay exponentially
Not a simple convolution
Convert to B-splines which have compact support
Conversion requires two passes of a recursive pre-filter
One pass in forward direction and one in reverse direction
Follow by cubic B-spline filter

30. 2-5-2 Spline Interpolation
Pre-filter has power of 2 coefficients
Eliminates multiplication from the pre-filter
Length of filter, k, governed by wordlength
Can be implemented recursively
Requires only single pass enabling a stream implementation
Final filter is a 4×4 piecewise cubic kernel

31. Mapping Optimisations
Incremental calculation can be used with streamed input or output
Successive pixels are in successive locations
Example: affine mapping
Moving to next row or column is a simple addition

32. Review of a range of image methods and possible FPGA implementations
Image Registration
Review of a range of image methods and possible FPGA implementations

33. Image Registration
Process of determining the mapping or transformation between two images
Four steps
Feature detection
Feature matching
Transform model estimation
Image transformation and resampling
Two broad classes of methods
Feature based
Matches corresponding points between images
Area based
Matches areas of pixels (correlation, etc)

34. Feature Detection Salient or distinctive points within images
Should be distributed throughout image
Ideally easy to detect
Invariant to expected geometric transformation
Commonly used features
Spot centre of gravity
Corners
Lines or edges and intersections
Points of strong response to band-pass filters
Scale invariant feature transform

35. Scale Invariant Feature Transform
Image examined at a range of scales
Typically 4 per octave, over several octaves
Filtered using difference of Gaussian filters
Can exploit fact that Gaussian filter is separable
Successive iterations of Gaussian filter reduce the scale
Can downsample as resolution is reduced
Determines peaks both spatially and in scale
Eliminate peaks along edges
Position along an edge is sensitive to noise
Feature also uses gradient orientation at peaks
Feature vector extracted for each peak for matching

36. Scale Invariant Feature Transform

37. Feature Matching
Determines correspondence between feature points in two images
Or between image and model
Combinatorial problem
Requires efficient search techniques
Tree based search methods
Hashing of feature vectors
Model based matching
Uses few key features to create an initial model
Model guides matching of other features
Model can be updated as more features are matched

38. Model Estimation
Given set of corresponding points determines best transformation
An optimisation problem
Linear models can be solved directly (least squares)
Non-linear models usually use iterative refinement
Usually optimised in software rather than hardware

39. Area Based Methods Matches images without explicitly detecting objects
Performed on whole image
Or on small patches extracted from the image
Limitations:
Larger patches can be sensitive to distortion
Sensitive to varying illumination or different sensors
Requires visual texture
Smooth textureless patch will match just about any other smooth area

40. Correlation Classical area matching algorithm Reducing complexity
Searches for offset where the cross correlation is maximum
Generally restricted to translation only
Reducing complexity
Instead of correlation, sum of squares or absolute differences are often used
Search simplified using hierarchical search
Uses image pyramid
Performs a coarse to fine search
Next slide calculates a 4×4 search window in parallel

41. Correlation

42. Fourier Methods Based on phase correlation
Shift in image domain corresponds to linear phase shift
Limited to pure translation
Combine with log-polar map to give rotation and scale invariance

43. Gradient Based Methods
Based on Taylor series, estimates offset based on gradient
Gradient obtained using matched prefilter and derivative
Offset solved using least squares
Can iterate solution to make it more accurate

44. Optimal Filtering Rather than interpolate and find best offset
Finds best filter to give one image from reference
Force sum of coefficients to equal 1
Filter coefficients found by least squares
In matrix form:
Offset then estimated from optimal filter coefficients

45. Optimal Filtering Shown here for 2×2 window
Number of MACs grows quickly with window size
When registering multiple images FTF only needed once

46. Applications of Registration
Camera calibration
Determines imaging model parameters of camera
Video compression
Motion compensation
Optical flow
Estimations motion field
Stereo Matching
Build disparity field, associated with depth or range
Image super-resolution
Combines several low resolution images to create a single high resolution image of scene

47. Summary Two approaches to geometric transformation
Forward mapping works from streamed input
Reverse mapping can provide streamed output
Separable mapping techniques
Converts mapping to sequence of 1D mappings
Interpolation
Required when reverse map samples do not fall on integer pixel locations
Image registration
Used to determine image transformation
Feature and area based approaches