1. Introduction to Computer Vision Sebastian van Delden USC Upstate svandelden@uscupstate.edu

2. What is Computer Vision? The goal of Computer Vision (or Machine Vision) is:  to make useful decisions about real physical objects and scenes based on sensed images.  to construct scene descriptions from images.

3. Issues Sensing: How do sensors obtain images from the world? Encoded Information: How do images yield information for understanding the 3D world, including the geometry, texture, motion, and identity of objects in it? Representations: How to represent objects, their parts, properties, and relationships, in a computer? Algorithms: How to process image information and construct descriptions of the objects and the world?

4. Digital Images A 2D image is a projection of a scene from a specific viewpoint; many 3D features are captured, some not. A 2D arrangement of pixels (picture elements) with a fixed number rows and columns. In grey scale images, each pixel is a single value of grey usually in the range [0…255] where 0 is black and 255 is white. In color images, each pixel has color information associated with it  RGB Color scheme – quantities of Red, Green and Blue  True color uses 1 byte for Red, 1 for Green, and 1 for blue  For many computer vision applications color is not needed.

5. Digital Images cont… Example digital image with 8 x 8 block of pixels from left eye.

6. Numerous Applications Image Database Querying Aerial Images and GIS Medical Imaging Processing Scanned Text Understanding a Scene of Parts Inspection applications Automated navigation Etc, etc, etc…

7. Understanding a Scene of Parts A robot needs to classify (or inspect) parts and act accordingly.

8. Combining Multiple Images Images with a constant background are subtracted to detect change over time. Pixel differences at boundary reveals moving object and its shape

9. Combining Multiple Images Images can also be added to blend them together.

10. Reality Check Success is hard won Some potential issues:  Lighting Fluctuations or inadequacies  Object positioning and occlusion  Background noise or other un-important image features. Good news  Industrial robotics usually provides a very controlled environment.

11. Imaging and Image Representation Sebastian van Delden USC Upstate svandelden@uscupstate.edu

12. Imaging Devices CCD Camera  Charge-coupled Device (CCD)  Instead of chemical film that reacts to light (like 35mm film), tiny solid cells convert light energy into electrical charge.

13. Problem with Digital Images Blooming Difficult to insulate adjacent sensing elements. Charge often leaks from hot cells to neighbors, making bright regions larger.

14. Problem with Digital Images Clipping or Wraparound Dark grid intersections at left were actually brightest of scene. In A/D conversion the bright values were clipped to lower values.

15. Problem with Digital Images Lens distortion distorts image “Barrel distortion” of rectangular grid is common for cheap lenses ($50) Precision lenses can cost $1000 or more. Zoom lenses often show severe distortion.

16. Problem with Digital Images CCD Variations  CCD sensors imperfections can cause different reading for the same light intensity. Chromatic Distortion  Different light wavelength bent differently. Quantization Effects  Mixing and Rounding problems.

17. Spatial Quantization Problems Both pixel size relative position make a difference.  Mixed pixel represents a mixture of intensity values in a real scene.  Small features can be lost or blended.

18. Portable Bit Map Image (PGM) P2 means ASCII gray Comments W=16; H=8 192 is max intensity Can be made with editor

19. Compression Lossless – if a decompression method exists to precisely recover the original image. Lossy – information is lost during compression and cannot to recovered during decompression (JPG, GIF)  GIF – (Graphics Interchange Format) only 8 bits used for color; can contain transparency and animation.  JPG – (Joint Photographic Experts Group) for high quality images; considers human vision systems and uses discrete cosine transform and Huffman coding.

20. Binary Image Analysis Sebastian van Delden USC Upstate svandelden@uscupstate.edu

21. Pixels and Neighborhoods A binary image consists of only two intensities – 0 and 1 (or 0 and 255). A binary image B can be obtained from a grayscale image I through an operation that selects a subset of image pixels as foreground pixels, the pixels of interest in an image. Everything else would be considered as background pixels. 00010010001000 00011110001000 00010010001000

22. Thresholding and Segmentation Gray level thresholding is the simplest segmentation process. Many objects or image regions are characterized by constant reflectivity or light absorption of their surface. Thresholding is computationally inexpensive and fast. Thresholding can easily be done in real time using specialized hardware

23. Thresholding Background is black Healthy cherry is bright Bruise is medium dark This Histogram shows two cherry regions (black background has been removed) gray-tone values pixel counts 0256

24. Thresholding - Example Original ImageThresholded Image (95)

25. Thresholding Example Over-segmentation (225) Under-segmentation (25)

26. Algorithm Thresholding algorithm  Search all the pixels f(i,j) of the image f.  An image element g(i,j) of the segmented image is an object pixel if f(i,j) >= T, and is a background pixel otherwise Correct threshold selection is crucial for successful threshold segmentation Threshold selection can be interactive or can be the result of some threshold detection method

27. Region Properties Once a binary image has been processed we could obtain properties about the regions in the processed image. Some of those properties are  Area, centroid  Measure of circularity and elongation

28. Area and Centroid

29. Connected Components Components are objects that share at least one common neighbor (in 4- or 8- neighborhood). Definition: A connected component labeling of binary image B is a labeled image LB in which the value of each pixel is the label of its connected component.

31. Recursive Labeling Algorithm Given a binary image B  Negate the image (make all 1-pixels to –1)  Search and find a –1 pixel, label it and find its (4- or 8-) neighboring pixels with –1 and assign the same label.  Recursively apply to resolve (merge or split) components. Increment label each time…

32. Robot/Camera Calibration Sebastian van Delden USC Upstate svandelden@uscupstate.edu

33. Vision Input Sensor A camera can be used to provide visual input data to the robot  Objects in the input images must be represented in the robot world coordinate system so that the robot can manipulate them Two Approaches:  Visual Servoing  Calibration

34. Visual Servoing A target image is compared against the current image An error vector is generated which indicates how the robot should be moved in order to minimized the error between the current and target images. The machine is incrementally moved.  PRO: No need for a mm/pixel ratio to be calculated. Very nice!

35. Visual Servoing Example: Camera mounted to end- effector and end- effector must move so that circular piece in the center of the image.

36. Camera Calibration Common in Industry CON: Usually must be manually done and can become un-calibrated over time. Steps:  Must calculate mm/pixel ratio  Must train a reference frame that can be seen in the input image

37. Example Setup: Frame {f} has been training in the robot work area. Parts can coordinates in this area. Consider Z to be fixed. Note: cannot recover depth this way…

38. Same example as before, but this is what the camera is seeing: