Combining Human and Machine Capabilities for Improved Accuracy and Speed in Visual Recognition Tasks Research Experiment Design Sprint: IVS Flower Recognition System Amir Schur

Problem Statement Investigate human-computer interaction in applications of pattern recognition where – higher accuracy is required than is currently achievable by automated systems – and there is enough time for a limited amount of human interaction Measure the degree of accuracy and speed improvement attributable to the human interaction

Background Most existing automated visual recognition tasks yield far from production-level results And manual task is too cumbersome and time consuming We need to find a place in between where speed is highly increased and accuracy is still acceptable

Interactive Visual System (IVS) for Flower Identification Originally developed at RPI for desktop PC, system called “Caviar” Later developed into a handheld application in an RPI/Pace project, called IVS

Overview of Interactive Tasks IVS Flower Identification System 1.Object segmentation 2.Feature extraction (numerous tasks) 3.Matching/classifying Each task can be done by human only, automated only, or combination of human and computer

Detail of IVS Interactive Tasks IVS for flower identification has six system activities: 1.Determining the dominant color of the flower petal 2.Determining secondary (less dominant) color of the flower petal 3.Determining color of the stamen or center portion of the flower 4.Counting the number of petals 5.Getting the horizontal and vertical bounds of the target flower 6.Getting the horizontal and vertical bounds of a flower petal Original software developed for Desktop and Palm Pilot, Java code Uses k-Nearest Neighbor (thus accurate training data is required) Color determination utilizes RGB color schema

Design Sprint: Overview Three separate experiments: 1.human only 2.machine only (no human subjects necessary) 3.human and machine combined Currently IVS has – 75 training photos (three photos each of 25 flowers) – 20 test photos Half the subjects will start with human-only identification, followed by machine + human, using existing 20 test images Other half vice versa: human + machine then human only. This group will also collect new flower images. (Balanced experimental design: no subjects get an unfair advantage)

Design Sprint: Human Only Scenario Capture time and accuracy Ideas: – Use IVS test photos – Use good flower guide to identify photos

Design Sprint: Machine Only Scenario Capture time and accuracy Ideas: – First iteration: use existing 20 test photos – Record top 10 choices – More digital images must be acquired with correct identification to enlarge training data

Design Sprint: Human + Machine Capture time and accuracy of combination of human + machine activity Ideas: – Segregate each available automated task. Run all automated except for one, where this part is done by human input. – Segregate group of tasks (color determination, background segmentation). Perform one task with computer and another with human.

Anticipated Experimental Outcomes Accuracy Time Human only Machine only Machine + human Time vs accuracy in Visual Recognition Tasks

Analysis of Results Time required by human will dictate the need of machine assistance. How much time is saved by using human + machine tool? Accuracy level of human + machine will dictate the need of such tool. Can it achieve the same level of accuracy as compared to human only? What is the maximum capability of machine only in terms of time and accuracy?

Possible Extensions Many functions can be extended: – Utilize different automated methods for color recognition (HSB, LAB, YCbCr, etc). – Utilize automated texture based methods (gabor and gist) – Utilize automated contour based pattern recognitions (distance vs angles, distances projection, min/max ratio, area ratio, automated number of petals counting) – More seamless integration of human and machine input. Currently it’s one or the other: cannot update machine’s cropping and outlining result, cannot update machine’s color determination.