Design for Embedded Image Processing on FPGAs Chapter 9 Geometric Transformations

Outline Two approaches to geometric transformation Forward mapping Reverse mapping Separable mapping techniques Performs mapping in two passes Interpolation Bilinear, bicubic, and cubic spline Image registration Feature based methods, SIFT Area based methods Applications

Geometric Transformation Redefines the geometry of pixels in an image Forward mapping Output locations are defined as a function of the input locations Reverse mapping Input locations are a function of the output locations Forward mapping Reverse mapping Input Image Output Image

Geometric Transformations Zoom Rotation Perspective Distortion

Geometric Transformation Requires a frame buffer to manage changed geometry Cannot have both streamed input and output Forward mapping: Can work with streamed input Reverse mapping: Can produce a streamed output

Problems with forward mapping Two pass algorithms

Problems with Forward Mapping Mapping specifies where each input pixel goes to Mapped location may not be on an integer grid Write the pixel value to the nearest location Problem Cannot just round mapped coordinates Some output pixels would have no inputs, some have multiple

Problems with Forward Mapping Need to map the whole pixel Each output pixel is a weighted sum of all inputs which overlap it Requires maintaining an output accumulator for each pixel to add all the input contributions Problem for streamed input Each input can map to many output pixels Several clock cycles may be required to distribute input to all affected output pixels This requires a complex caching arrangement Determining the intersection between input and output pixels is complex and time consuming

Separable Mapping Transform image in two 1D passes Row operation (each row) 1D transformation to get each pixel into the correct column Column operation (each column) 1D transformation to get each pixel into the correct row

Two Pass Implementation Input is row warped into one frame buffer Column warp is applied to first image Second image is loaded into second frame buffer Image is streamed out in row order Third image is warped into the same buffer Requires bank-switched dual port memory

Two Pass Implementation Hardware requirements may be reduced by processing every second image Reuse warping engine for rows and columns Only a single frame buffer is required Otherwise same as previous slide Each row or column processed independently Can process multiple rows or columns in parallel Limited by memory bandwidth

Limitations of Two Pass Approach Cannot rotate close to 90° Bottleneck problem: All pixels map to same column Cannot be separated out again in second pass Fold-over when mappings are not monotonic Mapping is many-to-one Limits range of mappings Second mapping (column mapping) uses a mix of input and output coordinates May not be possible to derive closed form mapping Separable 1-D warps not equivalent to 2-D warp Assumes pixels are infinitely thin

1-D Forward Mapping Requires interpolation Especially where scale factor is greater than 1 Otherwise get flat plateaus where a single input pixel spans several output pixels Requires anti-alias filtering Especially where scale factor is less than 1 Filter is spatially dependent on magnification Algorithm: Keep track of fractions of input and output pixels used Each cycle either reads an input or writes an output Based on if an output pixel is complete FIFO on input and output smooth data flow for streamed operation

1-D Forward Mapping

1-D Forward Mapping Speed operation by allowing both input and output cycles if input pixel completes the current output pixel

Alternative Algorithm Output is a sum over fractions of pixels Maintain a running sum (row integral) Interpolate to give integral at output edges Difference between these (normalised by width) is the output pixel value

Running Two Passes at Once First pass as before Result streamed to second pass All columns operated on at once Maintains state for each column in a data row buffer Output to frame buffer is not in raster order Depends on column warp

Avoiding aliasing Interpolation Bilinear Bicubic Cubic spline Reverse Mapping Avoiding aliasing Interpolation Bilinear Bicubic Cubic spline

Reverse Mapping Most common mapping when warping in software Producing a streamed output Want to know where output pixel comes from Mapped pixel does not necessarily fall on grid Read nearest pixel from frame buffer Interpolate between available pixels Some form of caching is required Also require some form of anti-alias filter Especially where magnification is less than 1 Frame buffer Output stream Reverse mapping Address

Using a Pyramid for Anti-aliasing There is insufficient time to filter input while performing warp Filtering is spatially dependent Image pyramid filters and down- samples each layer by successive powers of 2 Requires 33% more storage Can be constructed in a single pass through the image Using for anti-aliasing Select level for desired magnification Tri-linear interpolation between levels

Summed Area Tables Table contains sum of all pixels above and left May be formed recursively from a streamed input Can give sum of any rectangular region with 4 accesses

Reverse Mapping Generally mapping requires whole image in frame buffer first If distortion is small, can begin producing output before complete image is loaded Only a partial frame buffer required (using row buffers) Keeping number small reduces both resources and latency

Interpolation Determines values at locations between pixels Like filtering, but with spatially variant filter kernel

Bilinear Interpolation Output value is weighted sum of 4 nearest pixels Integer part gives pixel locations Fractional part gives weights A form of 2×2 filter except Weights vary with location in image Input is not streamed

Preloading the Cache Accessing 4 pixels per clock cycle requires careful cache design Preload cache before use Load one row in advance Either buffer mapping for reuse on next row Or recalculate the mapping on the next row Trade-off between computation and storage

Cache Preload Alternative

Bicubic Interpolation Requires a 4×4 interpolation kernel Kernel is separable, which reduces computation In general this separability cannot be exploited by stream processing In 1-D have piecewise cubic kernel with 4 sections 16 values must be read from cache or frame buffer for every output pixel Generally requires partitioning of cache over several block RAMs to overcome bandwidth limitations Kernel weights Either calculated as needed Precalculated and stored in a lookup table

Bicubic Interpolation

Cubic Spline Interpolation Piecewise cubic kernel Continuous in first and second derivatives This gives large region of support Although it does decay exponentially Not a simple convolution Convert to B-splines which have compact support Conversion requires two passes of a recursive pre-filter One pass in forward direction and one in reverse direction Follow by cubic B-spline filter

2-5-2 Spline Interpolation Pre-filter has power of 2 coefficients Eliminates multiplication from the pre-filter Length of filter, k, governed by wordlength Can be implemented recursively Requires only single pass enabling a stream implementation Final filter is a 4×4 piecewise cubic kernel

Mapping Optimisations Incremental calculation can be used with streamed input or output Successive pixels are in successive locations Example: affine mapping Moving to next row or column is a simple addition

Review of a range of image methods and possible FPGA implementations Image Registration Review of a range of image methods and possible FPGA implementations

Image Registration Process of determining the mapping or transformation between two images Four steps Feature detection Feature matching Transform model estimation Image transformation and resampling Two broad classes of methods Feature based Matches corresponding points between images Area based Matches areas of pixels (correlation, etc)

Feature Detection Salient or distinctive points within images Should be distributed throughout image Ideally easy to detect Invariant to expected geometric transformation Commonly used features Spot centre of gravity Corners Lines or edges and intersections Points of strong response to band-pass filters Scale invariant feature transform

Scale Invariant Feature Transform Image examined at a range of scales Typically 4 per octave, over several octaves Filtered using difference of Gaussian filters Can exploit fact that Gaussian filter is separable Successive iterations of Gaussian filter reduce the scale Can downsample as resolution is reduced Determines peaks both spatially and in scale Eliminate peaks along edges Position along an edge is sensitive to noise Feature also uses gradient orientation at peaks Feature vector extracted for each peak for matching

Scale Invariant Feature Transform

Feature Matching Determines correspondence between feature points in two images Or between image and model Combinatorial problem Requires efficient search techniques Tree based search methods Hashing of feature vectors Model based matching Uses few key features to create an initial model Model guides matching of other features Model can be updated as more features are matched

Model Estimation Given set of corresponding points determines best transformation An optimisation problem Linear models can be solved directly (least squares) Non-linear models usually use iterative refinement Usually optimised in software rather than hardware

Area Based Methods Matches images without explicitly detecting objects Performed on whole image Or on small patches extracted from the image Limitations: Larger patches can be sensitive to distortion Sensitive to varying illumination or different sensors Requires visual texture Smooth textureless patch will match just about any other smooth area

Correlation Classical area matching algorithm Reducing complexity Searches for offset where the cross correlation is maximum Generally restricted to translation only Reducing complexity Instead of correlation, sum of squares or absolute differences are often used Search simplified using hierarchical search Uses image pyramid Performs a coarse to fine search Next slide calculates a 4×4 search window in parallel

Correlation

Fourier Methods Based on phase correlation Shift in image domain corresponds to linear phase shift Limited to pure translation Combine with log-polar map to give rotation and scale invariance

Gradient Based Methods Based on Taylor series, estimates offset based on gradient Gradient obtained using matched prefilter and derivative Offset solved using least squares Can iterate solution to make it more accurate

Optimal Filtering Rather than interpolate and find best offset Finds best filter to give one image from reference Force sum of coefficients to equal 1 Filter coefficients found by least squares In matrix form: Offset then estimated from optimal filter coefficients

Optimal Filtering Shown here for 2×2 window Number of MACs grows quickly with window size When registering multiple images FTF only needed once

Applications of Registration Camera calibration Determines imaging model parameters of camera Video compression Motion compensation Optical flow Estimations motion field Stereo Matching Build disparity field, associated with depth or range Image super-resolution Combines several low resolution images to create a single high resolution image of scene

Summary Two approaches to geometric transformation Forward mapping works from streamed input Reverse mapping can provide streamed output Separable mapping techniques Converts mapping to sequence of 1D mappings Interpolation Required when reverse map samples do not fall on integer pixel locations Image registration Used to determine image transformation Feature and area based approaches