An Introduction to Real-time Machine Vision in Mechatronics Dr. Onur TOKER

2 Outline RT Machine Vision ? Mechatronics ? Review of previous experiments Image sensors (CMOS versus CCD) CMUCam, and cwCAM Interfacing a CCD camera to an 8-bit uC Difficulties in real-time machine vision Conclusion

Dr. Onur TOKER3 RT Machine Vision ? Mechatronics ? Machine vision is the ability of a computation machine to "see." Visual object tracking Object recognition Automated inspection, sorting Pattern recognition, etc. RT: There is no strict real-time system. There are systems with very short event response latency times.

Dr. Onur TOKER4 Experiment #1 1-D tracking system Analog video camera & PCI grabber VB 6 & VFW based Simple algorithm PID control Pentium 2/350MHz

Dr. Onur TOKER5 Experiment #2 Line following Wireless video camera and ToyCar Processing on a remote PC VC++ & DirectX based Simple algorithm Pentium 3/1GHz

Dr. Onur TOKER6 Experiment #3 Intel 8051 Very primitive machine vision Rapid prototyping board LDR sensor MOSFET driver

Dr. Onur TOKER7 Experiment #3

Dr. Onur TOKER8 BOE-BOT kit Simple kit PBASIC Not very flexible Very small RAM IR LEDs & photo transistors

Dr. Onur TOKER9 BOE-BOT demo

Dr. Onur TOKER10 Other demos CMUCam demo (Color tracking) WAM demo (MIT 1995) (Tracking by stereo machine vision)

Dr. Onur TOKER11 CMOS image sensors A CMOS sensor (OV7620) CUMCam uses such a sensor 2 nd PCB has a Scenix uC Digital output Easy to interface DALSA CMOS Sensor

Dr. Onur TOKER12 CCD image sensors DALSA CCD Sensor Analog output Difficult to interface Require several support chips

Dr. Onur TOKER13 CMOS versus CCD CMOS sensor 640x480 mode CCD sensor 640x480 (NTSC output) Under same lightning, same distance, comparable budget, CCD image is better.

Dr. Onur TOKER14 CMUCam architecture CMOS sensor SX28 uC uC/DSP “User device” issues high level commands SX28 does the processing (Limited built-in functions) SX28 replies Serial I/O CMUCam

Dr. Onur TOKER15 What is wrong with CMUCam ? Serial I/O (Low bandwidth) Low frame rate (Max. 17fps) CMOS sensor Processing done by SX28 Limited to built in functions Not much flexibility Instead of FPGA, uses SX28 Very compact design

Dr. Onur TOKER16 Proposed cwCam architecture CCD camera Video ADC FPGA uC/DSP Co-operating windowing approach (Discussed later) Parallel processors Parallel application specific digital architectures in the FPGA ASIC CPU cores in FPGA cwCam

Dr. Onur TOKER17 Digitized video signal VSYNC One field One frame

Dr. Onur TOKER18 A single field Several HSYNCs VSYNC

Dr. Onur TOKER19 Video ADC speed ? VSYNC HSYNC ??? Conclusion: Use 10MHz ADC

Dr. Onur TOKER20 Machine Vision with an Analog Industrial camera NTSC/30fps or PAL/25fps Even/odd field interlacing: 60fips/50fips rate 31ms VSYNC, 4.7us HSYNC for NTSC Needs a high speed ADC (AD9048 is 35 MHz) Most 8-bit uCs are too slow for this task Scenix SX28AC/DP 13.3 ns instruction cycle FPGA for accurate and high resolution capture

Dr. Onur TOKER21 Scenix SX28AC/DP 13.3 ns instruction cycle (75MHz clock) 10MHz video sampling = 100 ns loop time 1 Branch=3 cycles 4 instruction loop OK, but int. RAM too small 8051 too slow ! PIC16F877 too slow ! USE AN FPGA !

Dr. Onur TOKER22 Our FPGAs (Prototyping boards) Spartan II FPGA 50 Kgate 8MB RAM 8051

Dr. Onur TOKER23 Our ADC (AD9048) 35MSPS, 8-bit Flash ADC, Bipolar, 550mW, DIP 28 available AD9203, 40MSPS, 10-bit,CMOS, 74mW, No DIP available Actual photo of AD9048 used in our video digitizer

Dr. Onur TOKER24 Cortex-I approach Bederson, 1992 Logarithmic structured space variant pixel geometry Based on human vision system For real-time machine vision, reduce data to < 1500 pixels

Dr. Onur TOKER25 Co-operating windowing (1) Nassif & Capson, Watch windows (200x20) 1 Peripheral window (40x40 … 200x200) 1 Foveal window (20x20) Object tracking at 113Hz

Dr. Onur TOKER26 Co-operating windowing (2)

Dr. Onur TOKER27 Where we are at cwCAM ? AD9048 Video ADC board design completed (PCB layout !) AD9048 interfaced to 8051 prototyping board and tested Logic design is being done by Xilinx ISE software Mixed VHDL and graphical logic designs Tedious and long task CCD camera Video ADC FPGA cwCam

Dr. Onur TOKER28 Human Vision ? HMD and Dual monitor support PUMA robot arm and dual camera set

Dr. Onur TOKER29 Conclusion Real time machine vision requires innovative use of software and hardware techniques. Cortex-I (Human Eye), Co-operating windowing, etc. Innovative use of FPGAs and uC/DSPs. High frame rate CCD sensors. Optimum designs likely to be an application specific one. cwCAM is based on co-operating windowing approach and innovative hardware/software techniques.

Dr. Onur TOKER30 QUESTIONS ?

Dr. Onur TOKER31 Other slides

Dr. Onur TOKER32 Prototyping / Final product Prototyping board Serial download, EEPROM based, 9V battery Final design EPROM based minimum size PCB

Dr. Onur TOKER33 A Student Project Line following robot Phototransistor based sensors

Dr. Onur TOKER34 Image sensor types 1.Charged coupled devices (CCD) 2.Charge injection devices (CID) 3.CMOS Active Pixel Sensors (CMOS) They all convert incident light (photons) into electronic charge (electrons) by a photo-conversion process. Color sensors can be made by coating each individual pixel with a filter color (e.g. red, green, and blue). Beyond that point, everything is different.