1. Distortion Correction ECE 6276 Project Review Team 5: Basit Memon Foti Kacani Jason Haedt Jin Joo Lee Peter Karasev

2. 2 ECE 6276 Final Project Team 5 7/14/2009 Outline Objective/Motivation Movement of Data C code Conversion to Catapult C Initial Results Current Issues Plans for Optimization Schedule Meeting with Mike Bradley

3. 3 ECE 6276 Final Project Team 5 7/14/2009 Objective Given a distorted image with known size and known lens distortion parameter, generate an undistorted image.

4. 4 ECE 6276 Final Project Team 5 7/14/2009 Motivation – Why? The formation of undistorted images can be described by a series of matrix multiplications Distortion makes it very difficult to calibrate a camera to measure geometry (depth, size, orientation, etc) Many applications in image processing and computer vision like structure estimation, image mosaicing, and ultimately vision-based control.

5. 5 ECE 6276 Final Project Team 5 7/14/2009 Motivation contd.. Application: Measure motion and geometry Problem: Known geometry in the scene is warped, relationship between 3D and 2D points is nonlinear. Solution: Undo the distortion, so x 2D = A * X 3D

6. 6 ECE 6276 Final Project Team 5 7/14/2009 Movement of Data Input RAM on Chip Input Buffer 1 Pixel/Line Random Read LUT Algorithm Output

7. 7 ECE 6276 Final Project Team 5 7/14/2009 C Code Reference Design const double powercoeffs[256][5] = { 0.0001676486936455572,0.9984675943966409,0.0 04271662030922045,- 0.004648466894072161,0.001731029514912795, 0.0001678607731736833,1.004564329741083,0.00 4318757090371574,- 0.008746108062078511,0.001856610695386462,… I/O scheme: Write to output in raster order, read input randomly (From buffer). New input put into buffer in raster order. Coordinate in buffer does not have closed form expression- approximate with power series created in matlab, store as lookup table. void undistort_ref( const unsigned char pixels_in[PIXELS], unsigned char pixels_out[PIXELS], bool reset, unsigned short kappa_idx ); Top-level Entry point- block of pixels in and out, reset, index of distortion level if( (iid >= 0) && (jjd >= 0) && (iid < HEIGHT) && (jjd < WIDTH) ) pixels_out[ i_*WIDTH + j_ ] = pixels_in[ (iid*WIDTH) + jjd ]; else pixels_out[ i_*WIDTH + j_ ] = 0; Conditional Write operation (ensure in- bounds) // read next frame n = fread(pix_in, 1, PIXELS, fin); // do the work for this block undistort_ref( pix_in, pix_out, 0, kappa_idx); // write what we undistorted n = fwrite( pix_out, 1, PIXELS, fout ); Code verification- load binary input vectors, run, save, compare to matlab generated binary files

8. 8 ECE 6276 Final Project Team 5 7/14/2009 Conversion to Catapult C Catapult C does not support dynamic memory allocation - use static arrays for memory - simplify C code structure to avoid pointers Any value that needs to be stored must be declared as static or could be synthesized out Top-level default instantiation assumes that input and output ports are wires, must explicitly declare memories Casting for printing is not automatically supported, must use.to_int() and.to_double() functions Had to simplify C code structure to not use as many pointers as concept is not as well defined in Catapult C Verify correctness in Visual Studio and Catapult environment

9. 9 ECE 6276 Final Project Team 5 7/14/2009 Test Vector – Original Image

10. 10 ECE 6276 Final Project Team 5 7/14/2009 Test Vector – Distorted Image

11. 11 ECE 6276 Final Project Team 5 7/14/2009 Test Vectors – Recovered Image

12. 12 ECE 6276 Final Project Team 5 7/14/2009 Initial Catapult C Synthesis (1/3) Image Size (Pixels) Optimization Parameters Optimization (Design Goal) Area ScoreLatency Cycles / Throughput Cycles Maximum Delay (ns) Slack (ns) 8x8No OptimizationArea14713911/125011.41-1.41 8x8No sqrtArea35231625/21549.270.73 8x8No sqrtLatency16125591/591011.41-1.41 32 x 32No sqrtArea4002.3325697/3190612.18-2.18 64 x 64No sqrtArea4077.14102593/12729812.18-2.18 256 x 256No sqrtArea3906.161639169/20984348.941.06 640 x 480No sqrtArea4392.457681921/95262429.490.51

13. 13 ECE 6276 Final Project Team 5 7/14/2009 Initial Catapult C Synthesis (2/3)

14. 14 ECE 6276 Final Project Team 5 7/14/2009 Initial Catapult C Synthesis (3/3)

15. 15 ECE 6276 Final Project Team 5 7/14/2009 Current Problems Top issue is Area – we can’t support large image sizes based on current algorithm implementation. How do we achieve the ability to stream images in and then pass through our algorithm? Need to segment our algorithm and stream in portions of a larger image. Need to allocate large arrays to RAMs, particularly our LUT.

16. 16 ECE 6276 Final Project Team 5 7/14/2009 Project Timeline

17. 17 ECE 6276 Final Project Team 5 7/14/2009 Plans for Optimization Reformulate problem slightly to avoid expensive sqrt() operation (requires a new LUT) Ensure that LUT is mapping to ROM and buffer is on-chip RAM, not I/O pins Determine required bit-width for math operation result and use minimum Zero-pad arrays to be a power of two (avoids extra MUX elements) Avoid division- add lookup for radius values Experiment with size of input blocks. rgb2yuv example seemed to indicate large blocks create low latency, high area designs. Choose pipelining interval, loop to unroll. Unroll low level, pipeline top?

18. 18 ECE 6276 Final Project Team 5 7/14/2009 Meeting with Mike Bradley He showed us how to ensure that LUT is mapping to ROM in Catapult C GUI Determined which for loops to unroll and which to pipeline Discussed different types of technologies ASIC vs FPGAs

19. 19 ECE 6276 Final Project Team 5 7/14/2009 Updated Catapult Synthesis -Catapult C no longer crashes for large input images -Significant improvement in area, latency, and throughput -Processes 1 pixel per clock cycle

20. 20 ECE 6276 Final Project Team 5 7/14/2009 Questions? ?