1. Eye-RIS

2. Vision System sense – process - control autonomous mode Program stora

3. Outline hardware  system overview  AnaFocus, SIS Q-Eye chip  DiCop  ALTERA NIOS II RISC uP, MLP software  development environment

4. System overview Early-processing  Extract information from the image flow  Output: reduced set of data  Pixel-wise simultaneous processing Not a loop! Post-processing  Inputs are abstract entities  Output complex decisions

5. System overview Two main parts:  Smart Image Sensor (SIS) – Q-EYE chip  RISC processor – Altera Nios

6. Q-Eye chip 176 x 144 (QCIF) resolution Each cell:  Multi-mode optical sensors  Pixel memories  Linear and non-linear analogue processors  Binary processors  Interconnected with 8 neighbours

7. Q-Eye chip „This means that each pixel can both sense the corresponding spatial sample of the image and process this data in close interaction and cooperation with other pixels.” „Smart Image Sensor Q-Eye is also called Focal-Plane Processor (FPP) because it processes images at the same physical layer where they are sensed.”

8. Conventional Vision System Be aware of the data size of F!

9. Eye-RIS Vision System In this case: f << F

10. Architecture of the Q-Eye chip

11. The cell array 176x144 interconnected sensing-processing cells. Each of these cells contains mixed-signal circuitry capable of:  Sensing images; including adaptive linear and high-dynamic range acquisition  Storing up to 7 grey scale images and up to 4 binary images  Performing grey-scale image additions, subtraction and multiplication by a constant  Performing spatio-temporal image filtering (low-pass, high pass, band pass and stop band) with programmable bandwidth.  Performing grey scale image convolutions  Performing grey scale to binary, and binary to grey scale conversions  Performing generic combinational and sequential binary operations  Performing Hit-and-Miss binary operations  Real-time loading and downloading of images  Ultra fast calculation and input/output of certain image features such as centroids, mean values, pixel coordinates…

12. Block Diagram of a Q-Eye cell

13. Altera Nios II General-purpose RISC processor core.  Harvard architecture.  Full 32-bit instruction set, data path, and address space.  32 general-purpose registers.  32 external interrupt sources.  Single-instruction 32 x 32 multiply and divide producing a 32-bit result.  Single-instruction barrel shifter.  Access to a variety of on-chip peripherals, and interfaces to off-chip memories and peripherals through Avalon Bus.  Hardware-assisted JTAG debug module enabling processor start, stop, step and trace.  Instruction and Data Cache Memory.  JTAG Debug Module.  Exception and interrupt controller.

14. Summary of Q-Eye chip Chip: Q-Eye Technology: 0,18 um Power Supply: Dual (1.8 V, 3.3 V) # of Processor Elements array: 176×144 Accuracy: ~8bit Density of Processor Elements: 422 PE × mm 2 Die Size: 60 mm 2 Package size: 64×67×63 mm Power Consumption of system: 700mW Maximum frame rate: > 10 000 I/O type: Binary or grayscale type images

15. Software Eye-RIS ADK  Nios II: ANSI C/C++  Q-EYE: CFPP

16. Programming

17. Programming Q-Eye chip The CFPP Image Processing Library (IPL)  Spatio-temporal filters  Arithmetic operation between images  Thresholding  Morphological and logic binary operations The Extended Image Processing Library (EIPL)  Blob management  Classifying functions  Linear and non-linear digital processing of grey-level images  Geometrical transformations

18. Programming Altera Nios II C,C++, Assembly The Eye-RIS Basic Library (EBL):  Control the execution of the CFPP code  I/O management  Sending and/or receiving images to/from the PC either to be displayed or saved to disk  Printing error or information messages to a console  Timer management

19. Programming Eye-RIS (sync)

20. Programming Eye-RIS (async)

21. CFPP global memories // Declaration extern fpp_int value; extern fpp_bool flag; extern fpp_time texp; // Access void foo() { value = 123; int foo = value; if(flag) { FPPTime_write(texp, 500); int readTExp = FPPTime_read(texp); }

22. DEMO