Combining Human and Machine Capabilities for Improved Accuracy and Speed in Visual Recognition Tasks Team 1 Amir Schir (Team Leader), Fritz Gabriel, Jean Chu, Shahid Zaheer, Wen Cao DCS860A Emerging IT1-2011

Content Problem Statement Flower Recognition Task Segmentation Process Flower Recognition System – Human only Segmentation – Machine only segmentation – Combination methods Flower Feature Extraction Methods – Color based features – Contour based features – Texture based features Methodology Expect Results

Problem Statement Investigate human-computer interaction in applications of pattern recognition where higher accuracy is required than is currently achievable by automated systems, but where there is enough time for a limited amount of human interaction, and measure the degree of accuracy and speed improvement attributable to the human interaction. This topic has so far received only limited attention from the research community. Dr. George Nagy, Rensselaer Polytechnic Institute (RPI)

Flower Recognition Task Within flower recognition, there are three main tasks: a.Object segmentation b.feature extraction: numerous tasks c.Matching/classifying Each task can be done by human only, automated only or a combination of human and computers

Segmentation Process Investigate various machine only segmentation methods: thresholding, clustering, compression-based, histogram- based, edge-detection, region-growing, split-and merge, partial difference equation methods, graph-partitioning, watershed transformation, model-based segmentation, multi- scale segmentation, image segmentation and primal sketch, and neural-network segmentation method. Hypothesis 1: Automated methods cannot achieve a high level of accuracy reliably It is not possible to reliably achieve a high level of accuracy by Automated Methods (this is a re-write)

Flower Recognition System Machine only segmentation below:

Human only Segmentation Human can draw using a mouse or digital-pen around the flower foreground. Hypothesis 2: Human can perform segmentation task reliably and accurately in a timely manner [2]

Combination methods A machine can perform initial segmentation and human can make corrections. Example is in CAVIAR (Computer Assisted Visual Interactive System) Hypothesis 3: human machine combination is faster than human only segmentation and the result is comparable. This is the recommended approach until such a time that an automated only segmentation is reliable and accurate

Flower Feature Extraction Methods Flower Features Texture Based: Gabor, Gist Contour Based: Distances vs Angles Distances Projection Number of Petals Min/Max Area Ratio Color Features: RGB, HSB, LAB, YCbCr Figure 1: The Feature class and its children organized in different categories [4]

Flower Feature Extraction Methods Color based features: Color is the most important feature of a flower There are four common color spaces: RGB (red/green/blue), HSV (Hue, Saturation, Value), LAB (color-opponent space), and YCrCb (a method of encoding RGB more efficiently) Two common basic methods to extract color features from a digital image: Color Histogram and Color Moments Figure 2: Example of Color Histogram modeling [6]

Flower Feature Extraction Methods Contour based features: a. Distances vs. Angles Going through the contour for all angles, and then distances from the contour point and the center gravity are computed b. Projection of distance Projecting distance from the center gravity to the contour point c. Number of petals: can be calculated by capturing the mean of the distances and then going around the flower; the number of petals is based on the number of times this mean distance is crossed d. Min/Max: is taken by comparing the minimum distance and maximum distance of points from the center of the flower e. Area Ratio: takes the maximum distance and then computes the area of the flower region over the area of the circumscribed circle

Flower Feature Extraction Methods Texture based features: Gabor filter, named after Dennis Gabor, is a linear filter used for edge detection. Frequency and orientation representations of Gabor filters are similar to those of the human visual system, and they have been found to be particularly appropriate for texture representation and discrimination. In the spatial domain, a 2D Gabor filter is a Gaussian kernel function modulated by a sinusoidal plane wave. The Gabor filters are self-similar: all filters can be generated from one mother wavelet by dilation and rotation. Gist Descriptor is to develop a low dimensional representation of a scene, which does not require any form of segmentation. It proposes a set of perceptual dimensions (naturalness, openness, roughness, expansion, ruggedness) that represent the dominant spatial structure of a scene. They show that these dimensions may be reliably estimated using spectral and coarsely localized information. The image is divided into a 4-by-4 grid for which orientation histograms are extracted. The GIST descriptor has recently shown good results for image search.

Methodology For each subtasks above, measurement of speed and accuracy will be taken from human and computer Best possible combination will be presented Client and server architecture used for automated system. Client is in a cellphone or desktop, used to capture image, perform any manual steps by human and display results Server will perform any automated tests, store results and store a database of flowers for matching

Expected Results Possible combination of human and computer will produce a tool that is capable of recognition rate higher than 90% within acceptable time

References [1] Artificial Intelligence: Hype or Reality? IEEE Computer, vol. 36, no. 5, pp , May, 2003 – from Jies thesis p.2 [2] Interactive Visual System/IVS Arthur Evans, John Sikorski, Patricia Thomas, Jie Zou, George Nagy, Sung-Hyuk Cha, Charles Tappert, Interactive Visual System [3] Jie Zou: CAVIAR thesis [4] Semester Project Final Report June 2010, Vincent Vuarnoz, Supervisor: Péter Vajda, Professor: Touradj Ebrahimi ( n.pdf ) n.pdf [5] [6] Four Eyes Lab, Department of Computer Science University of California Santa Barbara [7] [8] [9] [10] Cancelable Biometrics: A Case Study in Fingerprints Ratha, N.; Connell, J.; Bolle, R.M.; Chikkerur, S.; IBM T. J. Watson Res. Center, Hawthorne, NY [11] Evaluation of GIST descriptors for web-scale image search