Objective of Computer Vision • The field of computer vision can be divided into two areas – Image enhancement – Image analysis Here we concentrate on fast methods typical for robot soccer and robot theatre applications

Binary image processing Image with two gray levels 0 and 1 It contains the basic terms and concepts used in machine vision. Its techniques are used in all aspects of a vision system. Small memory requirements Fast execution time

Gray level image vs. binary image

Image Enhancement

POINT OPERATIONS

Point Operations

Region and segmentation A subset of an image Segmentation Grouping of pixels into regions such that

Thresholding

Thresholding : Binary image Thresholding : A method to convert a gray scale image into a binary image for object-background separation : Thresholded gray image Obtained using a threshold T for the original gray image . : Binary image Equivalent to .

Three types of thresholding where Z is a set of which elements are integer-valued intervals.

Original image and its histogram

Thresholding

Histogram Equalization

Vision System of Soccer Robot Without going first to details we will discuss a set of techniques for robot soccer

Image notation for soccer Image : a two-dimensional array of pixels Pixel B[i, j] Pixel a[i, j]

Geometric properties In many cases, some simple features of regions are useful to: determine the locations of objects, and to recognize objects. Geometric properties: Size Position Orientation

Total size is number of black dots Size and position Given an m x n binary image, Size (area) A : zero-order moment Position : the center of area Pixel B[i, j] Total size is number of black dots Average in x Average in y

Example

Size filter for Noise removal It can effectively remove noise after component algorithm labeling. If objects of interest have sizes greater than T, all components below T are removed by changing the corresponding pixels’ values to 0. A noisy binary image and the resulting image after size filtering (T = 10)

How to get the position and angle of robot Get frame-grabber, color CCD camera and computer. Understand how you can read the image data from frame-grabber. Find the position of a colored object in 2-D image. Determine the robot by determining its two colored objects Calculate the position and angle of the robot from the positions of two colored objects.

Position of colored object 1. Setting of ranges for YUV [Ymin,Ymax], [Umin,Umax], [Vmin, Vmax] 2. Thresholding 3. Labeling (grouping) 4. Size filtering (noise elimination) 5. Finding the center of a colored object

Calculations for soccer field

Finding robots in the field Without special From size of soccer field and finding robot 1 with camera I can calculate its center in centimeters

Robot position and heading This cross sign is easy to recognize Using two colored objects

Robot color and team color This slide explains labeling robots and teams

Window tracking method Processing only the data within a small window Getting a fast vision processing

This is called line orientation Finding Orientation using the axis of elongation This is called line orientation Having the shape we want to find its axis of elongation

Having the shape we want to find its axis of elongation Line equation : : the minimum distance between the line and origin : the angle from x-axis to the line The distance, d, from any (x, y) within the object to the line : which satisfies Minimize Our task is to find values of angle theta and rho for which this formula is minimum. This provides best fit to line equation

How to calculate line orientation? Step 1: calculate a,b,c coefficients The center of object : Let By the least-squares fit of the line, Pixel B[i, j] Step 2: calculate angle, line orientation

Given is object. Find its line orientation The center of object : Let Calculate center Calculate a, b, and c.

By the least-squares fit of the line, Calculate center Calculate a, b, and c. Calculate theta

Binary algorithms to measure shapes and sizes from top camera Several definitions Neighbors 4-neighbors (4-connected) 8-neighbors (8-connected)

Main definitions Path Foreground : Connectivity A sequence of neighbors Foreground : The set of all unity valued pixels in an image Connectivity A pixel is said to be connected to if there is a path from to consisting entirely of pixels of . Connected components A set of pixels in which each pixel is connected to all other pixels.

Component labeling Component labeling algorithm It finds all connected components in an image and assigns a unique label to all the points in a component. One of the most common operations in machine vision Recursive connected components algorithm Sequential connected components algorithm The points in a connected component form a candidate region for an object.

An image and its connected component image

Recursive algorithm for connected component labeling Recursive connected components algorithm Scan the image to find an unlabeled unity valued pixel and assign it a new label L. 2. Recursively assign a label L to all its unity valued neighbors. 3. Stop if there are no more unlabeled unity valued pixels. 4. Go to step 1.

Pseudo code for the recursive algorithm Recursive call of procedure Label

Sequential algorithm for connected component labeling Sequential connected components algorithm using 4-connectivity Scan the image from left to right and top to bottom. 2. If the pixel is unity valued , then If only one of its upper or left neighbors has a label, then copy the label. (b) If both have the same label, then copy the same label. (c) If both have different labels, then copy the upper pixel’s label and enter the labels in an equivalence table as equivalent labels.

Sequential algorithm for connected component labeling continued (d) Otherwise assign a new label to this pixel and enter this label in the equivalence table. 3. If there are more pixels to consider, then go to step 2. 4. Find the lowest label for each equivalent set in the equivalence table. 5. Scan the picture. Replace each label by the lowest label in its equivalent set.

Pseudo code for step 2 in the sequential algorithm

An example for the sequential algorithm Initial labeling relabeling

Connected Component Labeling 1 1 2

Connected Component Labeling Algorithm 1 1. Image is A. Let A = -A; 2. Start in upper left and work L to R, Top to Bottom, looking for an unprocessed (-1) pixel. 3. When one is found, change its label to the next unused integer. Relabel all of that pixel’s unprocessed neighbors and their neighbors recursively. 4. When there are no more unprocessed neighbors, resume searching at step 2 -- but do so where you left off the last time. 1 2

Digital Geometry Neighborhood Connectedness Distance Metrics I(i,j) j (0,0) Picture Element or Pixel i We now turn our attention to the familiar concepts of geometry, such as distances between points, but from the point of view of a digital image. A digital image, as already noted, can be thought of as an integer-valued array whose entries are drawn from the finite set {0...K-1}. Each entry in this array is called a 'picture element' or pixel. Each pixel has a unique 'address' in the array given by it's coordinates (i,j). The value at location (i,j) is the quantized value of the continuous signal sampled at that spatial location. If K=2, the image is called a binary image, since all gray levels are either 0 or 1. For K>2, the image is called a grayscale image for the obvious reasons. When the value at a particular spatial location is a vector (implying multiple images), then the image is called a multispectral image. For example, a color image can be thought of as an image in which the value at a single pixel has three components (for each of the red, green, and blue components of the color signal) represented as a three element vector. 32 Pixel value I(I,j) = Neighborhood Connectedness Distance Metrics 0,1 Binary Image 0 - K-1 Gray Scale Image Vector: Multispectral Image

Connected Components Binary image with multiple 'objects' Separate 'objects' must be labeled individually Let's consider the concept of 'connectedness' in an image represented in discrete form on a rectangular grid. The simple binary image shown contains four 'objects': a central object that looks like a blob, a ring completely encircling it, and a second blob in the lower right hand corner, all represented by white pixels. The fourth object is the background, represented by black pixels. One would like to say that all the pixels in one of these objects are connected together and that the objects are not connected to each other or the background. In this image, there are six connected components: the three objects and the three pieces of the background. All of the pixels in the ring, for example, share several features: they are all white, taken together they form the ring, and, for every pixel in the ring, a path can be found to every other pixel in the ring without encountering any 'non-ring' pixels. This latter feature illustrates the concept of connectedness in a digital image; its definition can be made much more precise. This image has 6 Connected Components

Finding Connected Components Two points in an image are 'connected' if a path can be found for which the value of the image function is the same all along the path. P1 connected to P2 P3 connected to P4 By definition, two pixels in an image are connected if a path from the first point to the second point (or vice-versa) can be found such that the value of the image function is the same for all points in the path. It remains to be defined exactly what is meant by a 'path'. P1 not connected to P3 or P4 P2 not connected to P3 or P4 P3 not connected to P1 or P2 P4 not connected to P1 or P2

General Labeling Algorithm Pick any pixel in the image and assign it a label Assign same label to any neighbor pixel with the same value of the image function Continue labeling neighbors until no neighbors can be assigned this label Choose another label and another pixel not already labeled and continue If no more unlabeled image points, stop. Given a binary image, its connected components can be found with a very simple algorithm. Pick any point in the image and give it some label (say A). Examine the neighbors of the point labeled A. If any of these neighboring points have the same gray value* (here, only 0 or 1 are possible gray values), assign these points the label A also. Now consider the neighbors of the set of points just labelled; assign the label A to any which have the same gray value. Continue until no more points can be labelled A. The set of pixels labelled A constitute one connected component, which was 'grown' from the central point that was originally labelled. Now pick any point in the image which has not been labelled and assign it a different label (say B) and grow out from it. Repeat the process of picking an unlabelled point, assigning it a unique label, and growing out from this point until there are no unlabelled points left in the image. Eventually, every image pixel will have been assigned a label; the number of unique labels corresponds to the number of connected components in the image. Any pixel in a connected component is 'connected' to every other pixel in the same connected component but not to any pixel in a connected component with a different label. It is clear, therefore, that the notion of 'connectedness' in an image is intimately related to the definition of a pixel's neighbors. Let's explore the idea of the neighborhood of a pixel and the idea of the neighbors of a pixel in a little more detail. But first, let's illustrate this algorithm with an example. ___________________ * Or color or texture or whatever other feature we are interested in. Who's my neighbor?

Example of using this algorithm for labeling To illustrate the simple connected components algorithm, choose the top left pixel (arbitrary decision) and label it blue. This pixel has three neighbors, only two of which have the same gray level - the one to its immediate right and the one below it; label them both blue. In the next step, consider these two just-labeled points, starting with the right one. This point has 5 neighbors, one of which is our original point (already labeled) and one of which was labeled in the previous step. Of the three remaining pixels, only one has the same gray level, so label it blue. Now consider the second pixel labeled in the second step (just below the original start point). This pixel also has five neighbors, two of which are already labeled. Of the three remaining pixels, two have the same gray level and can therefore be labeled blue. This ends the third application of the growing process. Repeating this until no further pixels can be labeled results in the connected component shown in blue in the leftmost image of the middle row. Now pick any unlabeled point in the original image and start all over again. At the end of the process, the image has been decomposed into five connected components. Connected component labeling is a basic technique in computer vision; we will repeatedly use this and similar algorithms throughout the remainder of the course.

Who is my Neighbor? Consider the definition of the term 'neighbor' Two common definitions: In defining the notion of a path, and in the connected components algorithm, we used the term 'neighbor' in a very intuitive way. Let's examine this intuition a little more closely. There are two obvious definitions of neighbor and neighborhood commonly used in the vision literature. The first of these is called a 4-neighborhood; under this definition, the four pixels in gray surrounding the blue pixel are its neighbors and the blue pixel is said to have a neighborhood composed of the four gray pixels. These pixels are 'connected' to the blue pixel by the common edge between them. The second definition is called an 8-neighborhood and is shown on the right; the 8 neighbors of the blue pixel includes the four edge-connected neighbors and the four vertex connected neighbors. Neither of these definitions is entirely satisfactory for the definition of a neighborhood. Consider the digital counterpart of a closed curve in the plane, which partitions the background into two components........ Four Neighbor Eight Neighbor Consider what happens with a closed curve. One would expect a closed curve to partition the plane into two connected regions.

Alternate Neighborhood Definitions Suppose we have a closed curve in a plane. One of the properties of a closed curve is that it partitions the background into two components, one of which is inside the curve and one of which is outside the curve. The top image represents a digital version of such a curve. Let's apply the two definitions of 'neighborhood' to it and analyze what happens to the curve and the background in terms of their connectedness. Let's consider the 4-neighborhood definition first. In this case, the diagonally connected pixels that make up the 'rounded' corners of the O are not connected. Hence, the O is broken into four distinct components. The curve we said was connected is not connected using the 4-neighborhood definition of a neighborhood. On the other hand, the background pixels are not connected across the diagonally connected points either, so the background is broken into two separate connected components. We are now in the curious position of having the background partitioned into two parts by a curve that is not connected (or closed). Now consider the 8-neighborhood definition. In this case, the O is connected, since the vertex connected corners are part of the neighborhood. However, the inside and outside background components are also connected (for the same reason). We now have a closed curve which does not partition the background! Assuming 4-neighbor I have four different neighbors Assuming 8-neighbor I have just one neighbor

Possible Solutions to Neighbor problem Use 4-neighborhood for object and 8-neighborhood for background requires a-priori knowledge about which pixels are object and which are background Use a six-connected neighborhood: One possible solution to this dilemma is to use the 4-connected definition for objects and the 8-connected definition for the background. Of course, this implies that we must know which points correspond to the object and which parts correspond to the background. In general, the separation into object and background is not known a priori.

Digital Distances Alternate distance metrics for digital images In many cases, it is necessary to measure the distance between two points on the image plane. There are three metrics commonly used in vision. First, the standard Euclidean distance metric is simply the square root of the sum of the squares of the differences in the coordinates of the two points. The computation of this metric requires real arithmetic and the calculation of a square root; it is computationally expensive, but the most accurate. An approximation to the Euclidean distance is the city block distance, which is the sum of the absolute values of the differences in coordinates of the two points. The city block distance is always greater than or equal to the Euclidean distance. The third metric is called the chessboard distance, which is simply the maximum of the absolute values of the vertical and horizontal distances. These last two metrics have the advantage of being computationally simpler than the Euclidean distance. However, for precise mensuration problems, the Euclidean distance must be used. In other cases, the choice of the metric to use may not be as critical. Euclidean Distance City Block Distance Chessboard Distance = |i-n| + |j-m| = max[ |i-n|, |j-m| ] = (i-n) 2 + (j-m) 2 manhattan

Connected Components / Image Labeling Goal: To find clusters of pixels that are similar and connected to each other How it works: Assign a value to each pixel Define what similar values mean e.g., 10 +/- 2 Determine if like pixels are connected

Component Labeling 4- connected 8-connected Definitions Neighbors 4-neighbors (4-connected) 8-neighbors (8-connected) Connected components A set of pixels in which each pixel is connected to all other pixels.

An image and its connected component image

Connected Components / Image Labeling 1 2 A A A A A A A B B A A A A A A C A A A D D C C A A D D C C A Cross produces 4 segments

Connected Components / Image Labeling 1 2 A A A A A A A B B A A A A A A B A A A C C B B A A C C B B A rectangle produces 3 segments

Binary Image Processing After thresholding an image, we want to know something about the regions found ... How many objects are in the image? Where are the distinct “object” components? “Cleaning up” a binary image? Recognizing objects through their response to image masks Describing the shape/structure of 2d objects Pattern

Counting Objects external corners internal corners 1 1 1 1 1 1 1 1 You can characterize a shape by counting its internal and its external corners This gives you two parameters

Summary A simple procedure to get robot position and heading: Thresholding, labeling, size filtering, center of area calculation, window tracking method Trade-off between: real-time vision processing and robustness Specific frame grabber required

Problems for students Binary image processing and basic operations on binary images. List and explain briefly operations used in image enhancement What are point operations. Give examples. Explain contrast stretching operation. Write simple program. Definition of image segmentation. Segmentation based on thresholding Types of thresholding. Give examples of each with practical applications. The general idea of histogram equalization. Show examples. Geometric positions of objects in robot soccer. Size and position of objects in robot soccer.

Problems for students 11. How to calculate quickly the center of the mass of an object? 12. How to calculate line orientation? 13. Describe and program size filter for noise removal. Emphasize speed not quality. 14. How to calculate position and angle of a robot using the ceiling camera? 15. Use of colors in vision of robot soccer. 16. Window tracking methods in robot soccer. 17. Use of Line orientation in robot soccer.

Problems for students Binary image processing algorithms in robot soccer. Component labeling Recursive connected components algorithm Sequential connected components algorithm Algorithms to find connected components. Alternate definitions of neighbors in connected components algorithms. Applications in robot soccer. The concept of digital distance and its use. Connected component labeling in binary images. Counting objects. What are applications in robot soccer?