1. מערכות לבדיקת איכות חקלאיות: מערכת למיון פרחים מערכת לזיהוי ירקות הטכניון - מכון טכנולוגי לישראל : הפקולטה להנדסת חשמל - המעבדה לבקרה ורובוטיקה מגישים: מאיר לוין אייל בראונשטיין מנחה: דר' גבי דוידוב

2. Introduction Project goals and essence Manufacture systems for scenarios of Quality Assurance in agriculture: 1. A system that checks intactness of flowers. If leaves are missing, the flower is marked accordingly as not intact 2. A system that identifies vegetables from a given vegetables bank Both systems perform tracking of flowers / vegetables after identification Tracking by correlation (BW or RGB color correlation) and center-of- mass calculation. Prediction of states by Kalman Filter We implemented both systems in the MATLAB environment

3. Interlaced - Films Sequential frames in a movie from a digital camera are interlaced together. Interlacing smoothes the movement of objects for the human eye Example of interlacing in our movie. In this case the interlacing blurs the image objects, and causes undesirable effects. Therefore, we performed “De-interlacing” process to the input movie

4. Interlaced - Films

5. Correlation and Center of Mass Correlation is used for detecting the location of target in tracking Correlation coefficient (here Correlation matrix): Center of Mass in gray scale images – by referring to a black pixel as a mass point, and using the formula: - intensity level - coordinate (x or y)

6. Kalman Filter Kalman Filter is used for prediction of current state by weighing past states and measurements. Optimal prediction in sense of minimizing the error covariance The Kalman state – prediction equation: Where: F – State transition matrix K – Kalman Gain matrix Z – Measurement M – Measurement matrix (location, not velocity)

7. Tracking by correlation, center of mass and Kalman Filter In frame k, the previous frame (k-1) state is given, meaning the predicted coordinates (x,y) from frame (k-1) We create a spiral search path, starting from the previous frame coordinates, to measure the coordinates which fit best to the new location of the target The radius of the spiral search path is variable. Increasing it gives better coverage to find the target in the screen, with the price of longer processing time The measured coordinates are those that produce maximum correlation coefficient between the tracking window and reference window of target The previous state and current measurement are used in Kalman Filter, to produce the predicted state

8. Flower Classification System

11. General scheme: Scanning Phase Tracking Phase Testing Phase Input Output Scanning Phase – Scanning the screen for color [R, G, B], to identify flowers Tracking Phase – Tracking the flowers by correlation, center of mass and Kalman Filter Testing Phase – when the flower is most visible we test if it is intact or not

13. Color Reducer block – reduce the number of separate colors in frame Mark Color block – mark the color of flower by a binary mask The color is marked by proximity with tolerance to the specified [R, G, B] vector Flower Scanner block – scan frame for flower by color Flower ID

14. Color Reducer block 1. Converting to “Indexed-Image” with less colors 2. Converting back to RGB - Image (for better processing) Flower ID [R, G, B] image Indexed color image 1 2

15. Function “find color” [R, G, B] [255,63,0] [255,191,0] Flower ID 1. scan and calculate the difference between the current color vector and chosen color vector (reference color) 2. mark only the pixels that fall in the gap – Euclidian distance

16. - Getting a binary mask for current flower color - Labeling connected objects in the binary mask - Obtaining properties of objects – Centroid, Area and Bounding Box of the object - Identifying flowers by area threshold criterion - creating “ID Card” for each flower with its properties Flower ID

17. Tracking Phase Implementation: 1.Correlation - Spiral path around the previous coordinates to produce the measured coordinates. Correlation only established the measurement, NOT the updated state We give a schematic of the spiral path around the previous state 2. Inserting the measurement and the previous state into Kalman Filter, to produce the updated state

18. Testing Phase Adjustment of contrast inside tracking window. Helps in clear separation between flower and background Function color_linker to distinguish connected pixels which also have color proximity within a specified tolerance Function edge to extract the coordinates of the edge of the flower in the tracking window. The edge coordinates will be used to determine if the flower is intact or not Check intactness of flower by finding the edge coordinate which is closest to the centroid of the flower. If the distance between this coordinate and the centroid is below the threshold, we deduce that a leaf is missing, and the flower is not intact. Otherwise, the flower is presumed intact

19. Color linker We first give coordinates of pixels in the image The colors of these pixels with some proximity tolerance are marked to a binary mask (BW) From the binary mask we take only the objects which surround and are connected (8-connectivity) to the input coordinates

20. System Features The original frame is passed through various image filters in order to continue processing The system “waits” for the right moment to invoke the quality test The tracking process is essential here. we must track the object in order to find the best view point for testing intactness Criterion for Euclidian distance between reference color and current pixel color:

21. Vegetables Identification System

24. General scheme: Scanning Block Detection Block By color Input Output By shape Tracking Block The system scans the screen to perform acquisition for objects we recognize as vegetables by area and color criteria. For each new vegetable we create an “ID” card - centroid, area and bounding box Identification of vegetables by color and shape. First we acquire the color, to reduce the options of identity. From the sub-group of vegetables with appropriate color, we choose the one with maximum correlation (optimal shape matching) Output

25. Scanning Block - Reduce the number of colors in the RGB image. This reduces processing time and increases the tolerance for colors proximity - Converting to gray-scale image, and then to binary (BW) image - Labeling objects in BW image. Acquire region properties for the object – centroid, area and bounding box - Distinguishing vegetables by area threshold criterion - Inner and outer screens: When the object is centered in the outer screen, it is scanned (acquisition). Inner screen – tracking object

26. Detection Block Two - phase detection of vegetable in the window: 1. By color - create a sub-group of vegetables with same centroid pixel color. For example - for red - tomato and red pepper 2. By shape - Color correlation with reference window. The 3D correlation is performed in each color component (R, G, B) Coefficient = 0.56 Coefficient = 0.92 Coefficient = 0.78

27. Tracking Block Tracking phase is implemented by: - 3D color correlation - average of R, G, B correlation components - Center of mass for the tracking window - Kalman filter to weigh the measurement (correlation) against the prediction of previous state Reference Window Reference Window Reference Window

28. System Features When a vegetable is centered in the outer screen, it is acquired (center of mass, area etc.), since it is not yet entirely revealed in the screen When a vegetable is centered in the inner screen, we perform tracking on it When a vegetable exits the inner screen we stop tracking it (tracking flag is off) since it leaves the screen Vegetable identification is performed in 2 phases to reduce the processing time. Using only RGB color correlation would be extremely exhaustive and time-consuming. Therefore, we first reduce to a sub-group of vegetables by matching the color of the centroid pixels to the reference vegetables centroid pixels

29. weaknesses Vegetables Identification System – different orientation of objects between bank and film Flower Classification System– two flowers near each other

30. weaknesses

31. And to sum-up… The systems can be implemented in real-time environment to supply automatic Quality Assurance capabilities in the agricultural industry The flowers system can be further developed for more uses, such as checking the perfection of a flower by the nuances and shades of its color. Also, it can also be augmented to deal with flowers of different shapes, as seen below The vegetables systems can be further developed, for example, by adding a feature that checks quality and ripeness of the vegetable. The color features can be developed further to be more adaptive and robust (such as averaging of colors)

32. Thank you !