3. ISP Hardware Design & Verification System Architecture for ISP Hardware FPGA Based ISP Simulator FPGA Based ISP Simulator GUI Componets Design & Verification for ISP Functional Blocks

System Architecture for ISP Hardware Total 12 functional blocks - Sytem clock : pixel clock Input : 8-bit bayer Output : 16-bit 4:2:2 YCbCr Functional block control through I2C Function selector generates “enable” singal to each fucnctional block of ISP Operational frequency for blocks : More than 100 MHz Hardware architectue for ISP sytem

Synthesis Results of ISP Hareware Design - Op. freq. : 105.85MHz - Critical path : AWB Item Value Total logic elements 10,338 Total registers 7,431 Total memory bits 316,896 Setup time of the ISP functional block measured in slow model. Resource usage

FPGA Based ISP Simulator Test Enveronment : 8M CIS PC DE2-70 FPGA board HyImage V4P board HyImage interface board : Communication bet. PC & FPGA Operation speed : In case of full resolution : 7~8 Frames per sec. In case of down scaling : 10 Frame per sec. FPGA Based Real-time ISP Simulator

FPGA Based ISP Simulator PCB board for FPGA-PC conncetion PCB Board Design for FPGA-PC Connection gpio0, gpio1 FPGA board GPIO Required of PCB design for FPGA board I/O with a fixed power pins(3.3V) Connection information for PCB mode

FPGA Based ISP Simulator GUI ① ② ③ ④ ⑤ ⑥ No. Main function Details ① Down scaling - Full/down scaling selection ② Color interpolation 5x5 adaptive interpolation 3x3 bilinear interpolation ③ Option Selection for Gamma value Setting weight for Edge enhancement ④ Auto focus Selection for operation ⑤ AE_Option - Setting for in-door/out-door ⑥ ISP Function Activate each functional block ISP Simulator GUI

FPGA Based ISP Simulator GUI Gamma value setting window Gamma value setting window - possible from 0.1 through 0.9 Edge enhancement weight value setting window Edge enhancement setting window - Value setting: 1 ~ 5 - Increase by 0.5

ISP Functional Block Design & Verification – Color Interpolation Applied Algorithm 3ⅹ3 bilinear color interpolation Process Line buffer : 3 line data store 3x3 data generator : 3x3 data matrix generation Bilinear operator : bilinear interpolation ⇒ Applying total 4-level pipeline

ISP Functional Block Design & Verification – Color Interpolation Program Source Analysis Hardware file configuration inter_3_3.v buffer_3_3_8bit.v : 3x3 buffer dpsram3200x8_2.v : 3x3 buffer memory simulation file generation (Modelsim file) buffer_3_3_8bit.v dpsram3200x8_2.v image_gen.v : 8bits bayer data generation fil inter_tb.v : Test-bench top file Simulation File Generation(Matlab file) rgbtogbrg.m : bayer pattern file generation from image-file input interpolation_3_3.m : Result image generation after performing 3x3 interpolation image_out.m : Converting hardware-based 3x3 interpolation results to a corresponding image file

ISP Functional Block Design & Verification - Color interpolation 5x5 adaptive color interpolation Hardware file composition color_inter_new.v : color interpolation top file buffer_5_5.v : 5 line buffer dpsram3200x16.v : memory for 5-line buffer pattern_gen.v : generation pattern value according to bayer pattern param.h : Store 8 masks Simulation file composition 4 higher level files inter_tb.v : Top test-bench file image_gen.v : bayer pattern input generation

ISP Functional Block Design & Verification -Color interpolation Simulation file configuration Matlab file rgbtogbrg.m : generate bayer pattern file from image file input interpolation_5_5.m : Result image generation after 5x5 interpolation image_out.m : Converting hardware based 5x5 interpolation results to image file

ISP Functional Block Design & Verification – Edge enhancement Applied algorithm Unsharp mask filter Performing process Buffer 3x3 Store 3-line data 3x3 data generator Generate 3x3 data matrix Sum 3-line summation Divider Get average data Add multiplication result of weight and subtracted value from original pixel data to the original pixel Total 19-stage pipeline

ISP Functional Block Design & Verification - Auto White Balance Gray World Algorithm Hardware file composition AWB.v : AWB top file divider.vhd : division calculation Simulation file composition awb_lee3.v : AWB top file awb_lee3_tb.v : Test bench top file divider.vhd: 21bit / 11bit dpsram3200x16.v : memory for 5-line buffer im_64x64.txt : Text image file RGB_image_gen.v : RGB pattern input generation RGB_image_write.v

ISP Functional Block Design & Verification - Auto White Balance Simulation file composition Matlab file * im2text.m : convert an image file to a text file * text2im.m : convert a text file to an image file * compare_result.m : result comparison

ISP Functional Block Design & Verification -Auto White Balance GWA AWB 하드웨어 구조 Applying 1-stage pipeline Consists of 5 sub-blocks Signal generator : generate a frame completion signal FIFO : Perform delay for Divider block Accumulator : perform total summation R, G, B pixels Shifter : perform average of R, G ,B Divider : divide R, G, B average total by each R, G, B average Source analysis See reference files

ISP Functional Block Design & Verification – Edge enhancement Program source analysis Hardware file composition Edge_enhance.v : edge_enhance.top file divider.vhd : division calculation dpsram3200x32.v : Line buffer memory FIFO.v : FIFO process Simulation file composition(Modelsim file) buffer3_3.v : 3 line buffer edge_en_tb.v : edge enhance top file FIFO.v dpsram3200x32.v edge_enhance.v divider.vhd YCbCr_image_gen.v : YCbCr image generation Reg_Nbits.v YCbCr_image_write.v Simulation file composition(Matlab file) im2textYCbCr.m : convert into test file after converting image file to YCbCr textYCbCr2im.m : convert text file to YCbCr image file compare_result.m : Result comparison

ISP Functional Block Design & Verification – Auto Focus Applied algorithm Tenengrad (calculating focusing value) Global search(Movement of focusing position) Performing process 3 line Buffer : store 3-line Y data 3x3 data generator : generate 3x3 data matrix Focus value calculating : calculate focus value per frame Main operator : setting the moving range for actuator step I2C Controller : activate actuator step movement 그림 틀

ISP Funcional Block Design & Verification – Auto Focus Program source analysis Hardware file composition global_se.v : autofocus.top file focus_val.v : calculate focus value I2C_AF.v : Moving Motor dpsram3200x16: Line buffer memory af_mem.v : memory access Simulation file composition(Modelsim file) Imposible to derive program by Modelsim (register control)

ISP Functional Block Design & Verification기 – Gamma correction Applied algorithm Piece-wise linear Gamma-correction Performing process Top block for gamma correction

ISP Functional Block Design & Verification – Gamma correction Performing process Step decision Gamma correction performing by various Linear partitions, not by gamma curve 𝑦= 𝑦 𝑛−1 + 𝑦 𝑛 − 𝑦 𝑛−1 𝑥 𝑛 − 𝑥 𝑛−1 ×(𝑥− 𝑥 𝑛−1 ) y ′ =( x n − x n−1 ) y n−1 +( y n − y n−1 )( 𝑥 𝑛 − 𝑥 𝑛−1 ) 𝑦= 𝑦 ′ 𝑥 𝑛− 𝑥 𝑛−1 Calculating 8-step output pixel using Gamma table Piece-wise linear gamma correction

4. Embedded System Design & Verification Software ISP Simulator Software ISP Simulator GUI Hardware ISP 설계 및 검증

Block diagram of an ISP Embedded System SoCBase 1.0 Based Design Process : ARM926EJS FPGA : Virtex4LV80 Design block : - ISP top ·I2C controller ·ISP functional block control ·ISP function blocks - Dualport SRAM controller - VGA controller Block diagram of an ISP Embedded System

Design Details System spec. Design Tool

SoCBase 1.0 1. Single-processor based SoC development application platform 2. Provided with total 26 IPs · AMBA bus · Memory controller · External unit controller SoC base 1.0

Hardware Design & Verification Dual port SRAM controller dual port SRAM controller design for high-speed store/output of image data Use dual port FIFO - 8192 x 32 bits Generate SRAM control data from Write/read data controller Required in case for output of high-resolution image Dual port SRAM controller Stucture

Hardware Design & Verification VGA controller Ics_307 block Vga data Ctroller Top_VGA iCLK_25 Ics_sclk ics_data ics_strobe oVGA_R[7:0] oVGA_G[7:0] oVGA_B[7:0] oVGA_HS oVGA_VS oVGA_BLANK oVGA_SYNK oVGA_CLOCK iRST Ics_307 chip Clk_108 Clock generation through control of ICS 307 chip input signal Clock generation of 25MHz for 640x480 resolution output VGA controller usage example : Refer to HBE-SoC-IPD Users Guide p.250 ICS 307 chip manual : HBE-SoC-IPD Users Guide p.265 참조

Dual port memory controller SRAM-VGA Simulation RGB Text Input Image generator Dual port memory controller VGA controller SRAM model Text Output 24bits RGB output YCbCr Color conversion Source file location : sram_vga_test folder Input file : ycbcr_in.txt Output file : image_out.txt SRAM model generation Confirm image through Matlab

SRAM-VGA Simulation Simulation result Down scaling Original image(3200x2408) Resulting image(800x600)

Pin Configuration

ISP embedded system verification environment Simulator Access slave address through Multi-ICE Control ISP functional blocks through embedded software VGA ARM FPGA UART CIS ISP embedded system verification environment

Future Study PCB Connection bet. CIS-IPD board Need to remove noise of output images SDRAM memory based dualport memory controller design Need to increase output resolution ISP functional block software/hardware partition SystemC based RT level simulation Use Seamless CVE based virtual core