SIMD Image Processor Eric Liskay Andrew Northy Neraj Kumar 1

Features SIMD architecture Large array of cell processors operating in parallel Each processor performs the same function on a kernel -Kernel size is a 3x3 array of pixels Number of cell processors equal to (Tile Size in kernels) 2 -For example, if the tile size is 10 kernels, each tile will be 30x30 pixels Convolution Mode 3x3 pixel matrix read into the cell, only center pixel is written out Used for minimum, maximum, average, and edge detection operations User Inputs User can specify parameters for different functions User can specify a custom kernel for convolution 2

Top Level Diagram A B Cell Processo r x10 C Pixel Buffers A & B Pixel Buffer C (Output) 900x24-bits A - Pixel Data B - Pixel Data 900x24-bits Opcode XY Pixels User Inputs 24-bit Color RAW Image 3

Data Structures Pixel 3x3 Kernel Tile Red Channel Green Channel Blue Channel 8 bits 4

Image Processor Controller 4 Processes Load images from files into pixel buffers Positive Edge of Clock: Send data to cell processors Negative Edge of Clock: Read image from cell processors into Pixel Buffer C Write Output to file when finished processing the image

Cell Processor Each Cell processor operates on 3x3 pixel array, each pixel has 3 color channels -Two image inputs -Two user inputs -Opcode input -Kernel input (user’s filter) -One image output 6

Cell Functions Arithmetic and logical operations -Add, subtract, multiply divide -AND, OR, NOR, XOR -Operands may be two images, or one image and user input -Operation can be limited to one color channel or all channels 7

Cell Functions Darken Highlights -Compare sum of color channels with threshold value -Decrease value from all channels by set amount if sum is larger than threshold Brighten Shadows -Compare sum of color channels with threshold value -Increases value from all channels by set amount if sum is smaller than threshold 8

Cell Functions Convolution -Sobel, Prewitt, and Robinson Edge Detection -User specified Kernel -3x3 Minimum, Maximum, and Averaging Filters 9

Edge Detection Using the Sobel Operator The Sobel operator uses two 3x3 kernels which are convolved with a 3x3 array of pixels from the image A is the 3x3 array of pixels from the image G x and G y are the derivatives of the convolutions is the convolution operator G is the gradient magnitude 10

The Opcodes Regular Processing Mode (9 pixels in, 9 pixels out) MneumonicOpcodeDescription add0x00add pixel channel values in array A to array B addi0x01add userInputA to values in all channels from array A addir0x02add userInputA to value in red channel from array A addig0x03add userInputA to value in green channel from array A addib0x04add userInputA to value in blue channel from array A sub0x05subtract pixel channel values in array B from array A subi0x06subtract userInputA from values in all channels from array A subir0x07subtract userInputA from value in red channel from array A subig0x08subtract userInputA from value in green channel from array A subib0x09subtract userInputA from value in blue channel from array A mult0x0Amultiply pixel channel values in array A by array B mult20x0Bmultiply pixel channel values in array A by 2 multi0x0Cmultiply pixel channel values in array A by shifting left by userInputA div20x0Ddivide pixel channel values in array A by 2 divi0x0Edivide pixel channel values in array A by shifting right by userInputA inv0x0FInvert pixel channel values in array A and0x10AND pixel channel values in Array A with pixels in array B or0x11OR pixel channel values in Array A with pixels in array B nor0x12NOR pixel channel values in Array A with pixels in array B xor0x13XOR pixel channel values in Array A with pixels in array B drkn0x14Darken Highlights: If sum of pixel channel values in array A are above userInputA, subtract userInputB brtn0x15Brighten Shadows: If sum of pixel channel values in array A are below userInputA, add userInputB grayr0x16Grayscale based on Red Channel grayg0x17Grayscale based on Green Channel grayb0x18Grayscale based on Blue Channel pixflip0x19Pixels mirrored over middle pixel 11

The Opcodes Convolution Mode (9 pixels in, 1 pixels out) MneumonicOpcodeDescription sobel0x1ASobel sobel20x1BSobel Method 2 prewitt0x1CPrewitt robn0x1DRobinson North robnw0x1ERobinson Northwest robne0x1FRobinson Northeast robe0x20Robinson East robw0x21Robinson West robs0x22Robinson South robse0x23Robinson Southeast robsw0x24Robinson Southwest avggry0x253x3 pixel average in grayscale avgclr0x263x3 pixel average in color ukernel0x27User programmable filter min0x283x3 pixel minimum max0x293x3 pixel maximum 12

Color Codes

Regular Processing Mode 14

Red Image outputC(x,y)(0) <= B" ”; outputC(x,y)(1) <= inputA(x,y)(1); outputC(x,y)(2) <= inputA(x,y)(2); OriginalProcessed 15

Red Image outputC(x,y)(0) <= inputA(x,y)(0); outputC(x,y)(1) <= B" "; outputC(x,y)(2) <= inputA(x,y)(2); OriginalProcessed 16

Red Image outputC(x,y)(0) <= inputA(x,y)(0); outputC(x,y)(1) <= inputA(x,y)(1); outputC(x,y)(2) <= B" "; OriginalProcessed 17

Red = 0 OriginalProcessed 18

Green = 0 OriginalProcessed 19

Blue = 0 OriginalProcessed 20

Add: Image A + Image B OriginalProcessed 21

Add Immediate: Image A + User Input A OriginalProcessed 22

Add Immediate Red: Image A(red channel) + User Input A OriginalProcessed 23

Add Immediate Green: Image A(green channel) + User Input A OriginalProcessed 24

Add Immediate Blue: Image A(blue channel) + User Input A OriginalProcessed 25

Subtract: Image A - Image B OriginalProcessed 26

Subtract Immediate: Image A - User Input A OriginalProcessed 27

Subtract Immediate Red: Image A(red channel) - User Input A OriginalProcessed 28

Subtract Immediate Green: Image A(green channel) - User Input A OriginalProcessed 29

Subtract Immediate Blue: Image A(blue channel) - User Input A OriginalProcessed 30

OriginalProcessed Multiply: Image A * Image B 31

Multiply by 2: Image A * 2 OriginalProcessed 32

Multiply Immediate: Image A * User Input A OriginalProcessed 33

Divide by 2: Image A / 2 OriginalProcessed 34

Divide Immediate: Image A >> User Input A OriginalProcessed 35

Invert: Image A XOR “ ” OriginalProcessed 36

AND: Image A AND Image B OriginalProcessed 37

OR: Image A OR Image B OriginalProcessed 38

NOR: Image A NOR Image B OriginalProcessed 39

XOR: Image A XOR Image B OriginalProcessed 40

Darken Highlights: Decreases brightness by (User Input B) if average value of the pixels is above User Input A OriginalProcessed 41

Brighten Shadows: Increases brightness by (User Input B) if average value of the pixels is below User Input A OriginalProcessed 42

Convert to Grayscale using Red Channel OriginalProcessed 43

Convert to Grayscale using Green Channel OriginalProcessed 44

Convert to Grayscale using Blue Channel OriginalProcessed 45

Pixel Flip OriginalProcessed 46

Convolution Mode 47

Sobel 48

Sobel Method 2 49

Prewitt 50

Robinson North 51

Robinson Northwest 52

Robinson Northeast 53

Robinson East 54

Robinson West 55

Robinson South 56

Robinson Southeast 57

Robinson Southwest 58

3x3 Pixel Average Grayscale 59

3x3 Pixel Average Color 60

User Programmable Filter 61

3x3 Pixel Minimum OriginalProcessed 62

3x3 Pixel Maximum OriginalProcessed 63

Make Image Processor synthesizable Move loading files into testbench(HVL) and separate from image processor and Cells(HDL) Veloce? Enable the processing of larger images by loading and writing only part of the image at a time Other convolution ideas(e.g. Noise Reduction) Parameterized Kernel Size to enable larger kernels Gradient Threshold on “Darken Highlights” and “Brighten Shadows” for a less abrupt transition Multiple Operations done on image before writing it Future Work 64

Horizontal Flip OriginalProcessed 65

Vertical Flip OriginalProcessed 66

Rotate 180° Clockwise OriginalProcessed 67

All Photographs © Eric Liskay Lecture 10. Edge detection. Sobel and similar filters. Convolution Wikipedia - Sobel Operator References 68