Institute of Information Sciences and Technology Towards a Visual Notation for Pipelining in a Visual Programming Language for Programming FPGAs Chris Johnston Supervisors: Donald Bailey, Paul Lyons Institute of Information Sciences and Technology Massey University, Palmerston North New Zealand

Institute of Information Sciences and Technology CHINZ July Outline Background to the problem Real time image processing using FPGAs Overview of VERTIPH Visual Environment for Real Time Image Processing in Hardware Visual notations for pipelining Pipelining is required to meet timing constraints Representing pipelining in a VPL Summary

Institute of Information Sciences and Technology CHINZ July Programmable hardware Configurable logic blocks (arbitrary logic functions) Programmable interconnects (routing switches) I/O RAM (Multipliers) (Processor cores) FPGA Architecture CLB RAMI/O CLB

Institute of Information Sciences and Technology CHINZ July Image processing Extracting information from or improving an Image

Institute of Information Sciences and Technology CHINZ July Why FPGAs? Image processing operations data intensive computationally expensive To do more work serial processors have to go faster FPGAs can process information in parallel FPGAs build custom hardware for each operation can be reconfigured as needs change

Institute of Information Sciences and Technology CHINZ July Hardware design is different from software design Issues With Programming FPGAs HardwareSoftware Parallelism All circuits activeParallelism through threads Resource sharing Need hardware arbitration Only one process running SpeedUse long pipelines, trade off with throughput Get a faster processor! (!)

Institute of Information Sciences and Technology CHINZ July Parallelism Temporal and spatial parallelism Function block Input data Output data Function block Register Function block Register Function block Register TemporalSpatial Input data Function block Register Output data Function block A pipeline Function block Input data Output data Function block Parallel operations Hybrid

Institute of Information Sciences and Technology CHINZ July Hardware Design Entry Schematic Handel-C code unsigned 1 x,y,z,C; par { C = x & y; z = x ^ y; } HLLs (High-level languages) VHDL code entity HA is port( x,y:in bit; z,C:out bit ); End HA; architecture BEHAVIOURAL of HA is begin C<=x and y; z<=x xor y; end HA; HDLs (Hardware description languages) Hard to reuse Not algorithmic Verbose Low level Very flexible Like software Higher level Have to work to a design model Increasing abstraction

Institute of Information Sciences and Technology CHINZ July Resource & scheduling FramePixelLinePixel VERTIPH Three Views of the System Architecture view If else If While Computational view

Institute of Information Sciences and Technology CHINZ July VERTIPH Shows components, data and control flow Allows for encapsulation of data structures and processes Architecture view Three Views of the System Barrel CorrectionKeyboard Interface Frame Buffer Manager RAM 1 RAM 2 Bilinear InterpolationVideo DriverCamera Interface Control flow Data flow (Architectural)

Institute of Information Sciences and Technology CHINZ July Barrel CorrectionKeyboard Interface Frame Buffer Manager RAM 1 RAM 2 Bilinear InterpolationVideo DriverCamera Interface Control flow Data flow Shows components, data and control flow Allows for encapsulation of data structures and processes Resource & scheduling FramePixelLinePixel VERTIPH Three Views of the System Architecture view If else If While Computational view

Institute of Information Sciences and Technology CHINZ July VERTIPH Three Views of the System Resource & scheduling FramePixelLinePixel Local and Global scheduling Avoids resource conflicts Shows when operations can run Architecture view If else If While Computational view (Resource & Scheduling)

Institute of Information Sciences and Technology CHINZ July Resource scheduling FramePixelLinePixel VERTIPH Three Views of the System Architecture view If else If While Computational view

Institute of Information Sciences and Technology CHINZ July While (true) If (videoScanX == VisibleCols) xc * xc sx xc x y + 1 y sy+2y+1 sy xadd sqrd x sx k y Kru1 Interpolated LUT mag correctx correcty Operators xadd: x + 1 sqrd: sx + 2x + 1 kru: (sx + sy) * k correctx: mag * x correcty: mag * y If (videoScanX == VisibleCols) xc * xc sx xc x y + 1 y sy+2y+1 sy Operators xadd: x + 1 sqrd: sx + 2x + 1 kru: (sx + sy) * k correctx: mag * x correcty: mag * y -3 Resource & scheduling FramePixelLinePixel If else If While Computational view VERTIPH Three Views of the System Architecture view Clock cycle (Computational)

Institute of Information Sciences and Technology CHINZ July Computational view shows concurrency, pipelining and latency no obvious way to show a pipeline Sequential Parallel If else If While Computational view VERTIPH Three Views of the System ABC Time B C Concurrency

Institute of Information Sciences and Technology CHINZ July illustrates the dataflow spacially inefficient shows that A, B & C run in separate clock cycles VERTIPH Pipelining notations Diagonal Gantt A B C

Institute of Information Sciences and Technology CHINZ July shows when all processors are active VERTIPH Pipelining notations Staggered Gantt A B C A B A

Institute of Information Sciences and Technology CHINZ July branching makes diagrams more complex cannot express multicycle pipelines VERTIPH Pipelining notations Staggered Gantt with Conditional Branching A C B B1B1 A B B1B1 A

Institute of Information Sciences and Technology CHINZ July VERTIPH A series of processors perform successive phases of a task processor passes data “down the line” when its phase is complete early processors accept more data while later ones handle early data A slider to change data interarrival times Notation is compact, but hides processor concurrency Pipelining notations Sequential Pipeline AB C C1C1 DEF

Institute of Information Sciences and Technology CHINZ July Pipelining notations Combines the benefits of Sequential and Gantt Bars indicate: phase, throughput & time when processors have valid data. VERTIPH Sequential Pipeline With Staggered Bars AB C C1C1 DEF

Institute of Information Sciences and Technology CHINZ July AB C C1C1 DEFAB C C1C1 DEF Pipelining notations VERTIPH Detailed bars show repeated hardware Staggered Sequential Pipeline With Staggered Bars extend clock lines down

Institute of Information Sciences and Technology CHINZ July Summary Staggered Gantt used too much screen space could not describe all options Sequential was not expressive enough Adding Staggered Bars gives an indication of data throughput Using Detailed Bars Balance between complexity and expressiveness. Shows hardware, phase and throughput

Institute of Information Sciences and Technology CHINZ July FIN

Institute of Information Sciences and Technology CHINZ July Computational