1. Design for Embedded Image Processing on FPGAs
Chapter 5
Mapping Techniques

2. Outline Timing constraints Memory bandwidth constraints
Pipelining, synchronisation, clock domains
Memory bandwidth constraints
Caching and row buffering
Resource contention
Busses, multiplexing, controllers, reconfigurability
Computational techniques
Number systems
Lookup tables
CORDIC
Other techniques

3. Constraints Real-time implementation imposes constraints
Video rate input or output (timing) constraints
Streamed processing
VGA output must produce 1 pixel every 40 ns
Memory bandwidth
Memory can only be accessed once per clock cycle
Requires data caching
Resource constraints
Contention between parallel hardware blocks
Logic block usage (limited resources available)

4. Problem Pipelining Process synchronisation Multiple clock domains
Timing Constraints
Problem
Pipelining
Process synchronisation
Multiple clock domains

5. Timing Constraints Particularly important with stream processing
Source driven processes
Example: processing data from a camera
Timing governed by rate data produced at the source
Processing hardware has little control over arrival rate
Must respond to data and events as they arrive
Sink driven processes
Example: processing data to be displayed on screen
Timing governed by rate data is consumed
Processor must be scheduled to provide data and events at the right times

6. Stream Processing All calculations performed at pixel clock rate
Delay can easily exceed a single clock cycle
Pipelining: spread operation over several cycles
Achieve any throughput with sufficient resources
Uses intermediate registers to hold temporary results
Introduces control complications
Priming, stalling, flushing the pipeline, synchronisation issues
Input
stream
Output
Function
block
Function
block
Input
stream
Output
Register

7. Low-Level Pipelining Splits a calculation over several clock cycles
Example:
2 stage pipeline:
Register intermediate results
4 stage pipeline:
Each stage has smaller propagation delay
Reduce further with retiming

8. Retiming Uneven timing issues Retiming balances propagation delays
In last case, multiplication takes longer than addition
Propagation delay of multiply dominates
Clock speed limited by slowest stage
Retiming balances propagation delays
Moves some of the multiplication logic to after the registers
Speeds up the overall design

9. Low-Level Pipelining Throughput Latency Priming Flushing Stalling
1 value per clock cycle
Latency
Time from input to corresponding output
4 clock cycles here
Priming
Period of invalid output while waiting for first y
Flushing
Waiting after last valid input for the last y
Stalling
Stopping the clock for invalid input 1 2 3 4 x x1 y1 x2 y2 y3 y - 5 6 8 7 12 11 14 9 15 44 18 90 48 94
Time

10. Synchronisation
Operations in an application level algorithm are often pipelined
Easy if all of the operations are streamed
More difficult for random access and hybrid modes
Inputs and outputs of each operation in the pipeline must be synchronised
Different latencies
Different priming requirements for each operation
Operation 1
Operation 2
Operation 3

11. Synchronisation Methods
Global scheduling
Determine which operation requires data when
Usually have a global counter and schedule events by matching count
Effective if all events are synchronous

12. Synchronisation Methods
Data valid flags
Each operation runs independently
Associate with each output a data valid flag
Flag controls operation of downstream function blocks
Best suited for source driven processes
Sink driven processes may also have a “wait” flag propagating upstream
Operation 1 2
Data
Data valid
Enable

13. Synchronisation Methods
CSP channels
Guarantees synchronous communication
Both transmitter and receiver must be ready for data transfer
One operation will block (stall) if the other is not ready
Suitable for either input or output driven processes
Channel may have an associated FIFO buffer
Relaxes timing at expense of resources
Operation 1 2
Channel

14. CSP Channels Synchronisation and transfer logic
Transfer takes place when both sender and receiver are ready

15. Multiple Clock Domains
Not all external devices will operate at the same clock frequency
Interfaces may need to be built in different clock domains
Different domains are asynchronous relative to one another
Communicating between clock domains
Channels
Shared memory
FIFO buffers
Signals between domains require synchronisation
Failure to synchronise can result in metastability

16. Cross Domain Synchronisation
Synchroniser with bi-directional handshaking
Ensures data is stable before enabling clock in receiver
Signals by toggling Request and Acknowledge lines
Complete transfer takes 3 or 4 clock cycles
Throughput limited to this rate

17. Cross Domain Synchronisation
Higher throughput requires a FIFO type buffer
Build using small dual-port memory
One port is in each domain, so can be clocked independently
When buffer is half-full, synchronisation signal is sent
While one half is being unloaded, other half is being filled

18. Memory Bandwidth Constraints
Memory architectures
Caching
Row buffering
FIFOs

19. Memory Bandwidth Constraints
Software architecture
Read from memory, process, write results to memory
Memory bandwidth is often the bottleneck
Streamed pipeline architecture
Rather than write results to memory, results are passed immediately to next operation
Many memory accesses are eliminated
Where possible, as much processing as possible should be performed while data is passing through the FPGA

20. Memory Bandwidth Constraints
Some operations require image buffering
When computation order does not follow a raster scan
Required buffer size depends on operation
Potentially whole frame (e.g. rotation by 90°)
Large amount of memory required for frame buffer
Buffering single PAL frame: 768 x 576 x 24 bpp = 1.2 MB
Off-chip memory typically used for frame buffer
Usually single access per clock cycle
Problem accessing more than 1 pixel per clock cycle
Higher speed pipelined memory complicates design

21. Parallel Memory Architectures
Multiple copies in parallel banks
Each bank accessed separately
Duplicate contents if necessary
Partition odd and even banks
Distrubutes data over multiple banks
Allows 2×2 blocks to be accessible
Scramble addresses
Allows 1×4 and 4×1 blocks to be accessible

22. Parallel Memory Architectures
Data packing
Increase word width
Each access reads or writes several pixels
Bank switching
Double buffering
Upstream process writes to one bank
Downstream process reads from the other
Swap banks at end of frame
Frame buffer
Frame buffer
Bank 1
Frame buffer
Bank 2

23. Multi-port frame buffer
Increasing Bandwidth
Multi-port memory
2 or more parallel accesses
R/R, R/W, W/W
Higher speed RAM clock
Multiple sequential accesses (multi-phase design)
DDR RAM uses both rising and falling clock edges
Multi-port frame buffer
System clock
RAM clock (×3)
RAM accesses

24. Caching Stores data that will be used again in a secondary memory
Data from cache can be read in parallel with data from frame buffer
Data accessed from memory
Reduces memory bandwidth
Previous results that will be used again
Reduces computation expense
On-chip RAM often used for cache
Small blocks
Multiple parallel blocks gives wide bandwidth
Offchip
RAM
Cache

25. Cache Architectures Sequential cache Parallel cache
Cache controller responsible for ensuring data is available
Usually a FIFO when interfacing with slow or burst memory
Parallel cache
Data can be read from cache in parallel with memory

26. Row Buffering A common form of caching in image processing
Consider 3×3 window filter
Each output pixel value is a function of 9 input pixels
Direct implementation
Each window position requires reading 9 pixels from memory
Each pixel is read 9 times
For different window positions
Resulting memory bandwidth limits processing speed
Overcome with caching

27. Row Buffering for Filters
Streamed processing
Significant overlap between windows
Reuse data by shifting it along
Only need to load new pixels
Use row buffers to cache incoming stream
Only need to load one pixel for each window position
From row buffer
New pixels
Window registers
Row buffer
Input stream
New pixels

28. Row Buffers Implementation
In series or parallel with window (series shown here)
Shift register
Circular memory (dual-port)
Often indexed by pixel x address
Dual-port
memory
Shift register
Window
Filter function
Input
stream
Output
Row buffers
RAM
Address
counter
Address
Data

29. Circular Memory FIFO Address counter wraps around
Makes the memory addressing circular
Logic to detect buffer full and empty
Based on comparisons between address counters

30. Resource Constraints Contention Multiplexing and busses
Resource controllers
Conflict arbitration
Reconfigurability

31. Resource Constraints Resource contention
Finite system resources
Local memory, off-chip memory, implemented function blocks
Need scheduling if concurrent access required
Avoid concurrent processes writing to same resource
Often not detected in current programming languages
Must make efficient use of logic blocks
Not just for reducing usage but also propagation delay
Arithmetic operations consume significant resources
Multiplication, division, square roots, trig functions, etc
Use look-up tables and other computation techniques where appropriate

32. Resource Contention Only one process can access a resource at a time
Algorithm must be designed to handle potential conflicts
Reducing coupling between parallel sections
Arbitration logic must be built and respected
Contention strategies
Carry on regardless
Dangerous – danger of corrupting data for both processes
Wait for the resource to be free (block)
Need to stall any pipelines
Perform some other task while waiting
Hard to implement in hardware

33. Resource Sharing Expensive resources may be shared
Subject to bandwidth constraints
Simultaneous access to shared resources prohibited
Requires some form of arbitration
Sharing is implemented using multiplexers
To be practical, the resource must be more expensive than multiplexer and control logic
Alternative is to connect resources via a bus
Less expensive than multiplexers
Bus is resource in its own right

34. Busses Not all FPGAs have internal tristate buffers
Can implement using distributed multiplexers

35. Resource Manager Encapsulate resource with arbitration logic
Access via interface
Interface includes handshaking signals

36. Arbitration Methods Scheduled access Semaphore access Priority access
Algorithm is designed to separate accesses
Works okay for simple or synchronous algorithms
Difficult to handle asynchronous accesses
Semaphore access
Builds hardware to resolve conflicts
Access may not be guaranteed
Priority access
Different parts of the algorithm have different priority
High priority sections are guaranteed access
Low priority sections only have access when resource is free

37. Scheduled Access Example
Bank switched memory
Particular RAM used for reading or writing controlled by set State variable

38. Prioritised Access Based on a priority encoder
Can combine with multiplexer to give prioritised multiplexer

39. Semaphore
Required if the access lasts longer than one cycle

40. Reconfigurability Compile time reconfigurability
Complete functionality determined at compile time
Run time reconfigurability
Algorithm is split into multiple sequential sections
Complete FPGA reprogrammed as part of operation
Must maintain data off-chip for it not to be lost
Partial reconfigurability
Only part of FPGA is reprogrammed
Application is split into modules
These interface with a static core through predefined interfaces
Static core
Dynamic module

41. Computation Techniques
Number systems
Lookup tables
CORDIC
Approximations
Other techniques

42. Number Representation
Integers
Unsigned
Signed
Real numbers
Fixed point
Floating point
Logarithmic number system
Residue number systems
Redundant representations

43. Integers
Most common unsigned representation is based on binary numbers
Use only as many bits as necessary
Signed number formats
Sign magnitude
Separate sign bit (usually MSB)
Two’s complement
Gives MSB a negative weight
Offset binary
Adds an offset or bias to all numbers so that negative numbers are positive

44. Fixed Point Real Numbers
Scales an integer by a fraction, so the integer represents the number of fractions
Arithmetic operations are the same as for integers
Need to manually keep track of scale factor
Changes with multiplication
Need to align words with different fractions when adding
Alignment is by a fixed amount
A shift which is free in hardware
Dynamic range limited by size of the integer
Overcome by allowing fraction to vary (floating point)

45. Floating Point Numbers
Splits number into two parts
Exponent: position of the binary point
Represented using offset binary
Significand: binary fraction, normalised between 1 and 2
Most significant bit is always 1, so does not need to be represented
Two exponent values reserved for special cases
All zeros
Denormalised numbers (binary fraction is not normalised)
All ones
Overflow, infinity and NAN

46. Floating Point Numbers
Size of exponent and significand can be tailored to the accuracy of calculation being performed
IEEE standard representation
32 bit: 1 sign, 8 bits for exponent, 24 bits for significand
64 bit: 1 sign, 11 bits for exponent, 53 bits for significand
Also specifies the computation of arithmetic operations
Makes them processor independent
Logic for floating point significantly more than for fixed point
Performing normalisation
Detecting and managing all the different error conditions

47. Logarithmic Number System
Represents a number by its logarithm (base 2)
Uses a separate sign bit
Logarithms of negative numbers not defined
Similar precision and range to floating point
Attraction
Multiplication and division become addition and subtraction
Disadvantage
Addition and subtraction are significantly more complex
Second term implemented either directly or using a lookup table

48. Residue Number Systems
Represents integers by their residues with respect to a number of co-prime moduli
Example: moduli {13,15,17} can represent the range
Addition, subtraction and multiplication performed using modulo arithmetic on each residue
Shorter propagation delay
Less logic (especially for multiplication)
Convert back to binary using Chinese remainder theorem

49. Redundant Representations
Arithmetic speed is limited by carry propagation
Using a redundant representation limits length of carry propagation
Gives significant speed improvements
Standard binary:
Signed digit:
Asymmetric signed digit:
Disadvantages
Using additional digits requires wider signals
2 bits for radix-2 signed digit representation
This requires more logic to implement

50. Lookup Tables Many functions are expensive to calculate
Precalculate and store results in a lookup table
Can trade table size for accuracy
Drop least significant bits
Accuracy of approximation depends on slope
Don’t need to round the input value
Just adjust the value in the table … … …

51. Interpolated Lookup Tables
For “smooth” functions, can trade table size for computational resources and latency
Two truncated tables
Values, and slopes
Use slope to interpolate values
Accuracy depends on curvature
LUTs in parallel are a single LUT with wider data path
Slope

52. Interpolated Lookup Tables
Using dual-port RAM to reduce table width
Allows two table accesses
Calculate slope, rather than store in table
Higher order interpolation
More tables in parallel to give interpolation coefficients

53. Bipartite Lookup Tables
With smoothly varying functions, slopes are often similar
Rather than multiply by slope, results of multiplication are stored in a lookup table
Offsets from the slope are shared for several segments
Symmetry can also be exploited to further reduce the size of the offset LUT

54. Higher Order Approximations
Lookup tables
Table size grows exponentially with precision
Polynomial approximations
Coefficients chosen to minimise approximation error
Size grows approximately linearly with precision
Summary comparison
Precision
Best method
< 10 bits
Direct lookup table
8-14 bits
Bipartite lookup table
12-20 bits
Interpolated lookup table
> 16 bits
Polynomial approximation

55. CORDIC Method for calculating trigonometric functions
Based on incremental rotations
Trick is choosing factors to be powers of 2
Multiplications then become shifts
Where dk is direction of rotation and
Result rotates the input vector by
With gain of approximately

56. CORDIC Implementation
Add an additional accumulator z to hold the angle
Load signal resets counter and loads initial value
Mode signal selects the rotation direction
Small ROM contains the angles for each iteration

57. CORDIC Operation Modes
Rotation mode
Starts with angle in z
Chooses dk = sign(zk) to converge the angle to zero
Result is to rotate the vector by that angle
Can be used to calculate sine and cosine of an angle
Vectoring mode
Aligns the vector with the x axis
Chooses dk = -sign(yk) to converge y to zero
Result is magnitude and angle of the vector
Can be used to calculate arctangent
Each iteration gives 1 bit of the result

58. Unrolled CORDIC Builds separate hardware for each iteration
Avoids the need for barrel shift
This is an expensive operation
Shift is fixed for an iteration so may be hard wired
Angles added are constants so may be optimised
If necessary may be pipelined for speed

59. CORDIC Variations Problem with CORDIC is the gain factor Linear CORDIC
Compensated CORDIC introduces a scale factor with each iteration which makes the gain equal to 1
Scale factor require one extra addition for x and y per iteration
Linear CORDIC
Long multiplication
Non-restoring division
Hyperbolic CORDIC
Hyperbolic sine, cosine and arctangent
Related shift and add techniques can also be used for exponential and logarithm

60. Iterative Approximations
CORDIC and related algorithms converge slowly
One bit per iteration
Newton-Raphson gives quadratic convergence
Root finding algorithm
Number of bits doubles with each iteration
Example: square root
Form equation to solve
Form iteration
Determine initial approximation
Number of iterations depends on initial accuracy
Lookup table or polynomial approximation reduces iterations

61. Other Techniques Bit-serial processing Incremental update Separability
Useful with scarce resources and latency is not a problem
Processes one bit per iteration
Incremental update
Stream processing evaluates pixels successively
Reuse result from previous pixel with an appropriate adjustment
Separability
Separates 2D operations to separate operations in each of X and Y directions

62. Summary Mapping requires matching computation to resources available
Constrained by
Timing
Overcome with pipelining and appropriate synchronisation
Bandwidth
Overcome with memory architecture and caching
Resources
Share resources with arbitration and resource controllers
Reduce using appropriate computation techniques
A range of number systems and computational techniques have been reviewed