1. October 5, 2006. 1Memorial University of Newfoundland Faculty of Engineering & Applied Science Engineering 6806 Electrical & Computer Engineering Design Project INTRODUCTION TO MACHINE VISION II Prof. Nick Krouglicof

2. October 5, 2006. 2Memorial University of Newfoundland Presentation Outline 1.Machine vision systems for mechanical metrology A systematic approach to the calibration of machine vision systems for industrial metrology An automatic, machine vision based system for robot calibration 2.Industrial Applications of Machine Vision High speed, line scan camera-based inspection system for the food processing industry Vision based inspection of liquid crystal display (LCD) modules 3.CPLD based interfaces for sensors and actuators

3. October 5, 2006. 3Memorial University of Newfoundland A systematic approach to the calibration of machine vision systems for industrial metrology In the field of machine vision, camera calibration refers to the experimental determination of a set of parameters which describe the image formation process for a given analytical model of the machine vision system. Ideally, camera calibration is performed without specialized optical equipment, without modifications to the hardware, and without a priori knowledge of the vision system. Most calibration techniques are based on the observation of planar (2D) targets with a large number of control points.

4. October 5, 2006. 4Memorial University of Newfoundland A systematic approach to the calibration of machine vision systems for industrial metrology The machine vision parameters which must be identified include : a)The scale factor b)The frame buffer coordinates of the image center c)The effective focal length of the lens-camera assembly d)The radial lens distortion coefficient e)The pose (position and orientation) of the camera Parameters a) through d) are classified as intrinsic, e) as extrinsic.

5. October 5, 2006. 5Memorial University of Newfoundland A systematic approach to the calibration of machine vision systems for industrial metrology

6. October 5, 2006. 6Memorial University of Newfoundland A systematic approach to the calibration of machine vision systems for industrial metrology

7. October 5, 2006. 7Memorial University of Newfoundland A systematic approach to the calibration of machine vision systems for industrial metrology In general, most camera calibration techniques fail because: They fail to account for the mathematical dependency between the image center and the X-Y position of the camera. The determination of the intrinsic parameters assumes a priori knowledge of the extrinsic parameters (i.e. position and orientation of the camera). Therefore, the extrinsic parameters are generally determined first. In order to accurately evaluate the intrinsic parameters, the extrinsic parameters must be know to a degree of accuracy which is practically unattainable (6 significant digits).

8. October 5, 2006. 8Memorial University of Newfoundland A systematic approach to the calibration of machine vision systems for industrial metrology In order to compensate for the deficiencies of existing calibration techniques, a special fixture was designed to evaluate the critical parameters (i.e. image center and effective focal length) before the remaining parameters.

9. October 5, 2006. 9Memorial University of Newfoundland To locate the edge with sub-pixel accuracy, the gradient in the vicinity of the edge is reconstructed using a continuous Gaussian function. The coordinates of the pixel nearest to the edge are denoted by i and j. The location of the edge with sub-pixel accuracy is determined by calculating the offsets x and y which maximize the function s(x,y) below over a 5 by 5 kernel : A systematic approach to the calibration of machine vision systems for industrial metrology

10. October 5, 2006. 10Memorial University of Newfoundland An automatic, machine vision based system for robot calibration Objective: To develop a low cost, non-contact robot calibration system that can be applied on a routine basis without significantly affecting production schedules. Based on a CCD camera mounted on the robot’s wrist. The camera is used to determine the position and orientation of a fixed, passive target in six degrees-of- freedom. The system employs the Hayati and Mirmirani modeling convention for closed kinematic chains and non-linear least squares analysis to determine the real kinematic parameters specific to a particular robot.

11. October 5, 2006. 11Memorial University of Newfoundland An automatic, machine vision based system for robot calibration

12. October 5, 2006. 12Memorial University of Newfoundland An automatic, machine vision based system for robot calibration

13. October 5, 2006. 13Memorial University of Newfoundland An automatic, machine vision based system for robot calibration

14. October 5, 2006. 14Memorial University of Newfoundland Industrial Machine Vision Applications

15. October 5, 2006. 15Memorial University of Newfoundland High Speed, Line Scan Based Inspection System

16. October 5, 2006. 16Memorial University of Newfoundland Objective: To remove defects (I.e. visible, dark particle larger than 0.007”) from apple sauce System must be able to handle 12 metric tons per 8 hour shift System must remove 95% of visible defects Line Scan Based Inspection System: Specifications

17. October 5, 2006. 17Memorial University of Newfoundland 2 distinct challenges: Detection Removal High Speed, Line Scan Based Inspection System

18. October 5, 2006. 18Memorial University of Newfoundland High Speed, Line Scan Based Inspection System

19. October 5, 2006. 19Memorial University of Newfoundland High Speed, Line Scan Based Inspection System

20. October 5, 2006. 20Memorial University of Newfoundland Detection system based on a high performance line scan camera; 4096 pixels per line at 4800 lines per second. Image acquisition and processing functions implemented on a Complex Programmable Logic Device (CPLD) as opposed to a microprocessor or Digital Signal Processor (DSP). The objective is to implement image processing functions with dedicated logic gates (i.e. hardware) for real-time performance. Line Scan Based Detection System

21. October 5, 2006. 21Memorial University of Newfoundland What are CPLDs? Complex Programmable Logic Devices (CPLDs) are a class of programmable logic device that are commonly used to implement complex digital designs on a single integrated circuit. Applications of CPLDs in the field of computer engineering include the implementation of bus controllers, address decoders, priority encoder and state machines

22. October 5, 2006. 22Memorial University of Newfoundland Line Scan Camera-Based Detection System

23. October 5, 2006. 23Memorial University of Newfoundland Line Scan Camera-Based Detection System

24. October 5, 2006. 24Memorial University of Newfoundland Line Scan Camera-Based Detection System Typical Particle Typical section of apple sauce recorded with an area scan camera

25. October 5, 2006. 25Memorial University of Newfoundland Line Scan Camera-Based Detection System

26. October 5, 2006. 26Memorial University of Newfoundland Line Scan Camera-Based Detection System

27. October 5, 2006. 27Memorial University of Newfoundland Removal System

28. October 5, 2006. 28Memorial University of Newfoundland Removal System

29. October 5, 2006. 29Memorial University of Newfoundland Removal System: Flow Characterization Rheological nomenclature and associated velocity profiles for steady flow through tubes with circular cross section.

30. October 5, 2006. 30Memorial University of Newfoundland Viscosity Measurement

31. October 5, 2006. 31Memorial University of Newfoundland Viscosity Measurement Velocity profile can be characterized by power law!

32. October 5, 2006. 32Memorial University of Newfoundland Flow Profile of Apple Sauce

33. October 5, 2006. 33Memorial University of Newfoundland Particle Removal Window

34. October 5, 2006. 34Memorial University of Newfoundland System Timing Diagram

35. October 5, 2006. 35Memorial University of Newfoundland Aspiration Valve Characterization

36. October 5, 2006. 36Memorial University of Newfoundland Aspiration Valve Characterization

37. October 5, 2006. 37Memorial University of Newfoundland Particle Detection Rate Versus Flowrate

38. October 5, 2006. 38Memorial University of Newfoundland High Speed, Line Scan Based Inspection System: Current Status Industrial partner is currently developing the production version of the system. Packaging of the principle components (i.e., lenses, cameras, electronics, light sources) remains a major challenge given the environment. One possible solution is to integrate all the electronics in the camera enclosure. Partner is anxious to explore applications in the pulp and paper industry.

39. October 5, 2006. 39Memorial University of Newfoundland Vision Based Inspection of Liquid Crystal Display (LCD) modules Objective: to automate the inspection of LCD modules in order to improve quality control One step in the implementation of a Six-Sigma Program (“3.4 defects per million opportunities”) The inspection must be completed within 30 seconds for 10 predetermined LCD patterns System can “learn” new LCD modules without modifying software

40. October 5, 2006. 40Memorial University of Newfoundland Vision Based Inspection of LCD modules: System Components Pulnix camera with macro lens High frequency fluorescent light sources Coreco Bandit integrated image acquisition and VGA accelerator Software developed using with WiT graphical programming environment in combination with Microsoft VB

41. October 5, 2006. 41Memorial University of Newfoundland Vision Based Inspection of Liquid Crystal Display (LCD) modules

42. October 5, 2006. 42Memorial University of Newfoundland Original Image Showing Error in Alignment Vision Based Inspection of Liquid Crystal Display (LCD) modules

43. October 5, 2006. 43Memorial University of Newfoundland Thresholding Operation – Image Subtraction with respect to an image with no segments illuminated Vision Based Inspection of Liquid Crystal Display (LCD) modules

44. October 5, 2006. 44Memorial University of Newfoundland Blob Analysis – Reference Points are Identified Vision Based Inspection of Liquid Crystal Display (LCD) modules

45. October 5, 2006. 45Memorial University of Newfoundland Image Rotation and Translation Vision Based Inspection of Liquid Crystal Display (LCD) modules

46. October 5, 2006. 46Memorial University of Newfoundland Pixel by Pixel Image Subtraction from Reference Image – Thinning Operator Vision Based Inspection of Liquid Crystal Display (LCD) modules

47. October 5, 2006. 47Memorial University of Newfoundland Blob Analysis Vision Based Inspection of Liquid Crystal Display (LCD) modules

48. October 5, 2006. 48Memorial University of Newfoundland Blob Analysis Vision Based Inspection of Liquid Crystal Display (LCD) modules: Conclusions Inspection system was installed at BAE Systems Canada Ltd. where it was used to test between 200 and 600 LCD displays per day. Number of defective modules that passed inspection was basically reduced to zero. Occasional “false positives” proved to be technical problems with the devices that previously went unreported. Applications for this technology are numerous given the number of LCD displays produced annually.

49. October 5, 2006. 49Memorial University of Newfoundland CPLD based interfaces for smart sensors and actuators

50. October 5, 2006. 50Memorial University of Newfoundland Introduction Several novel “mechatronic” applications for CPLDs are presented. CPLD basically provides reconfigurable digital I/O that permits the implementation of interfaces for smart sensors and actuators. Applications that will be discussed include: Stepper motor controller Quadrature decoder/counter for optical encoder Pulse-Width Modulation (PWM) motor drives By incorporating a CPLD that supports In-System Programmability (ISP) the target device can be reprogrammed by the user for a variety of applications without removing it from the host system.

51. October 5, 2006. 51Memorial University of Newfoundland Mechatronic Applications of CPLDs – RSG System

52. October 5, 2006. 52Memorial University of Newfoundland Design Objectives of RSG System To develop a low cost, microcontroller-based system that allows the user to interface a variety of sensors and actuators to a laptop computer. In addition to being “user- friendly”, the system must be capable of functioning as either: 1.A portable data acquisition system, 2.An autonomous, stand-alone controller in support of mechatronic design projects, 3.A data logger that can acquire and store data at regular intervals over extended periods of time.

53. October 5, 2006. 53Memorial University of Newfoundland Design Philosophy Embedded Controller System incorporates an 80C52-BASIC Microcontroller Simplifies the management of the various onboard peripherals (Analog-to-Digital converter, LCD display, serial interface) Incorporates a powerful, imbedded BASIC interpreter Can also be programmed in assembler or C (Small Device C Compiler: SDCC) Assembled/compiled code can be “called” from Basic

54. October 5, 2006. 54Memorial University of Newfoundland Design Philosophy (continued) Programmable Logic System incorporates a user configurable Complex Programmable Logic Device (CPLD) CPLD supports in-system programmability by the user Mechatronic applications envisioned for the CPLD on the UC 2 system include quadrature decoder/counter interfaces for optical encoders, stepper motors controllers, PWM motor drives, sequential controllers, event counters/timers, Serial Peripheral Interface (SPI), etc.

55. October 5, 2006. 55Memorial University of Newfoundland Block Diagram of RSG System

56. October 5, 2006. 56Memorial University of Newfoundland Stepper Motor Controller: Synchronous Counter Design

57. October 5, 2006. 57Memorial University of Newfoundland Stepper Motor - Full-Stepping

58. October 5, 2006. 58Memorial University of Newfoundland Stepper Motor – Half Stepping

59. October 5, 2006. 59Memorial University of Newfoundland Stepper Motor Half-Step Switching Sequences STEP TRANSISTORS 1234 1ONOFFONOFF 2ONOFF 3ONOFF ON 4OFF ON 5OFFONOFFON 6OFFONOFF 7 ON OFF 8 ONOFF

60. October 5, 2006. 60Memorial University of Newfoundland Truth Table for Half-Stepping Switching Sequence

61. October 5, 2006. 61Memorial University of Newfoundland Stepper Motor: Synchronous Counter Design

62. October 5, 2006. 62Memorial University of Newfoundland Schematic of stepper motor controller Stepper Motor: Synchronous Counter Design

63. October 5, 2006. 63Memorial University of Newfoundland Quadrature Decoder/Counter Interface for Optical Encoder: State Machine

64. October 5, 2006. 64Memorial University of Newfoundland Optical Encoder

65. October 5, 2006. 65Memorial University of Newfoundland Optical Encoder - Decoding

66. October 5, 2006. 66Memorial University of Newfoundland Optical Encoder – State Diagram

67. October 5, 2006. 67Memorial University of Newfoundland Optical Encoder – Edge Detection Circuitry

68. October 5, 2006. 68Memorial University of Newfoundland Summary of Quadrature Decoding Logic State Transition Encoder Signals Flipflop outputs: enc_decX.q Past State Present State Count Chan A Chan B 0123 12UP1  1110 23  10111 34 0  0001 41  01000 14DOWN  00100 43 0  0010 32  11011 21 1  1101 Note:  and  indicate a rising and falling edge respectively.

69. October 5, 2006. 69Memorial University of Newfoundland Truth Table Implementation of Quadrature Decoder TABLE enc_dec[3..0].q=> dwn_cnt, up_cnt; B"0000"=>0,0; B"0001"=>0,1; B"0010"=>1,0; B"0011"=>0,0; B"0100"=>1,0; B"0101"=>0,0; B"0110"=>0,0; B"0111"=>0,1; B"1000"=>0,1; B"1001"=>0,0; B"1010"=>0,0; B"1011"=>1,0; B"1100"=>0,0; B"1101"=>1,0; B"1110"=>0,1; B"1111"=>0,0; END TABLE;

70. October 5, 2006. 70Memorial University of Newfoundland AHDL Implementation of Counter Module IF up_cnt THEN encoder[].d = encoder[].q + 1; ELSE IF dwn_cnt THEN encoder[].d = encoder[].q - 1; ELSE encoder[].d = encoder[].q; END IF;

71. October 5, 2006. 71Memorial University of Newfoundland Pulse-Width Modulation: VHDL/AHDL Programming

72. October 5, 2006. 72Memorial University of Newfoundland AHDL Implementation of PWM Controller totaltime[].clk = clk; totaltime[].clrn= VCC; totaltime[].prn = VCC; totaltime[].ena = io_space_2 & (add_latch[3..0].q == B"0110") & !write/; totaltime[].d= add_data[]; lowtime[].clk = clk; lowtime[].clrn= VCC; lowtime[].prn = VCC; lowtime[].ena = io_space_2 & (add_latch[3..0].q == B"0111") & !write/; lowtime[].d= add_data[];

73. October 5, 2006. 73Memorial University of Newfoundland cntr[].clk = clk; cntr[].clrn= !(cntr[].q == totaltime[].q); cntr[].prn = VCC; cntr[].ena= VCC; cntr[].d= cntr[].q + 1; switch.clk= clk; switch.clrn= VCC; switch.prn = VCC; switch.ena = VCC; pwm_out= switch.q; switch.d = (cntr[].q < lowtime[].q); AHDL Implementation of PWM Controller

74. October 5, 2006. 74Memorial University of Newfoundland Pedagogical Applications

75. October 5, 2006. 75Memorial University of Newfoundland Pedagogical Applications

76. October 5, 2006. 76Memorial University of Newfoundland Conclusions CPLDs are well suited to the implementation of interfaces for smart sensors and actuators. Incorporating a CPLD that supports In-System Programmability (ISP) into a data acquisition system provides reconfigurable digital I/O that allows the target system to be reprogrammed by the user for a variety of applications without hardware modifications. This is particularly useful for those applications that require hardware interfaces for several different sensors simultaneously.

77. October 5, 2006. 77Memorial University of Newfoundland QUESTIONS ?