Veggie Vision: A Produce Recognition System R.M. Bolle J.H. Connell N. Haas R. Mohan G. Taubin IBM T.J. Watson Resarch Center Presented by Chris McClendon Presented by Chris McClendon

What is it? / Veggie vision in an automated produce ID system

Hardware / A scale / A polarized light / A camera / A PIII 200 MHz / A method

Challenges / The segmentation problem / Foregroud/background differentiation / Packaging / Background variation / The segmentation problem / Foregroud/background differentiation / Packaging / Background variation

Challenges / Color Constancy / One element of recognition is based on color profiles. / The lighting in a grocery store is subject to large variation / Color Constancy / One element of recognition is based on color profiles. / The lighting in a grocery store is subject to large variation

Challenges / Speed of Recognition / The system should integrate with the time scale for other checkout operations / The agreed time parameter should be around 1 second / Speed of Recognition / The system should integrate with the time scale for other checkout operations / The agreed time parameter should be around 1 second

Challenges / Performance / Ideally equal to barcode scanning (100%) / Realistic expectations of performance should be at least as good as that of the average checker (~80%) / Performance / Ideally equal to barcode scanning (100%) / Realistic expectations of performance should be at least as good as that of the average checker (~80%)

Challenges / Ease of Use / Integrated into existing barcode reader housing / Minimal operator training / Ease of Use / Integrated into existing barcode reader housing / Minimal operator training

Challenges / System Training / Gradual adaption to variations in season, supplier, and ripeness/freshness / System Training / Gradual adaption to variations in season, supplier, and ripeness/freshness

Challenges / Database size / Seasonal / Varies by store, region, and harvest / Database size / Seasonal / Varies by store, region, and harvest

Solutions / Parallel vs. perpendicular polarization for filtering out glare from the light source

Solutions / 2 images used for segmentation, one light and one dark / Brightness variation greater than the threshold (T ∆ ) are considered foreground / 2 images used for segmentation, one light and one dark / Brightness variation greater than the threshold (T ∆ ) are considered foreground

Solutions / Notice the plastic bag is also illuminated / Another threshold (T dark ) is used to identify the transparent bags / Notice the plastic bag is also illuminated / Another threshold (T dark ) is used to identify the transparent bags

Question Session #1 / Any other illumination-related problems in this scenario?

Question Session #1 / Any other illumination-related problems in this scenario? / Dark produce / Waxed or shiny produce / Any other illumination-related problems in this scenario? / Dark produce / Waxed or shiny produce

Segmentation Results / Necessary Conditions / Stationary / Scale assistance ensures stable produce / Necessary Conditions / Stationary / Scale assistance ensures stable produce

Feature Selection / Histograms used extensively for feature representation / Much smaller than actual images / Well researched method for representation of visual cues / Training issues / Histograms used extensively for feature representation / Much smaller than actual images / Well researched method for representation of visual cues / Training issues

Color Features / HSI (HSL) color space

Apples vs Oranges

Texture Features / More difficult to describe computationally / Textel--artichokes or pineapples / Random variation--parsley / More difficult to describe computationally / Textel--artichokes or pineapples / Random variation--parsley

Texture Detection Methods / Measure A / Convolution of crossed bar masks [ ] [ ] / Measure A / Convolution of crossed bar masks [ ] [ ]

Texture Detection Methods / Method B / Deviation of image intensity from its nearest neighbors / Performed on reduced images for speed / Method B / Deviation of image intensity from its nearest neighbors / Performed on reduced images for speed

Question Session #2 / Which method would you use?

Question Session #2 / Results

Classification & Training / Symbols / P i, i = 1,2,….,N prototype histograms / Q, a produce histogram to be identified / Each histogram has 4 components / F = {hue, saturation, intensity, texture} / Normalized / Symbols / P i, i = 1,2,….,N prototype histograms / Q, a produce histogram to be identified / Each histogram has 4 components / F = {hue, saturation, intensity, texture} / Normalized

Classification & Training / Symbols / Each prototype histogram P i is associated with an identifier I(P i ) / How to compute distance Manhattan style / Symbols / Each prototype histogram P i is associated with an identifier I(P i ) / How to compute distance Manhattan style

Classification & Training / Distance equation

Classification & Training / Now that the distance is calculated / Decision Rule 1 / Now that the distance is calculated / Decision Rule 1

Classification & Training / For acceptable threshold but multiple identifiers / Decision Rule 2 / For acceptable threshold but multiple identifiers / Decision Rule 2

Classification & Training / Out of bounds / Decision Rule 3 / Out of bounds / Decision Rule 3

Acting on Results / Sure / Directly accepted by the register / Okay & Uncertain / Sorted nearest matches displayed / Sure / Directly accepted by the register / Okay & Uncertain / Sorted nearest matches displayed

Acting on Results / If none of the options are chosen, a new prototype for P chosen is added / The correctly identified Q’s prototype is judged for accuracy / All non-chosen prototypes of the P class are ‘aged’ / If none of the options are chosen, a new prototype for P chosen is added / The correctly identified Q’s prototype is judged for accuracy / All non-chosen prototypes of the P class are ‘aged’

Question Session #3 How do you think it did?

Overall Results / Feature Success trial 1

Overall Results / Feature Success trial 2

Overall Results / Training technique analysis

Overall Results

Super Straigtforward Results