1. Veggie Vision: A Produce Recognition System R.M. Bolle J.H. Connell N. Haas R. Mohan G. Taubin IBM T.J. Watson Resarch Center Presented by Chris McClendon Presented by Chris McClendon

2. What is it? / Veggie vision in an automated produce ID system

3. Hardware / A scale / A polarized light / A camera / A PIII 200 MHz / A method

4. Challenges / The segmentation problem / Foregroud/background differentiation / Packaging / Background variation / The segmentation problem / Foregroud/background differentiation / Packaging / Background variation

5. Challenges / Color Constancy / One element of recognition is based on color profiles. / The lighting in a grocery store is subject to large variation / Color Constancy / One element of recognition is based on color profiles. / The lighting in a grocery store is subject to large variation

6. Challenges / Speed of Recognition / The system should integrate with the time scale for other checkout operations / The agreed time parameter should be around 1 second / Speed of Recognition / The system should integrate with the time scale for other checkout operations / The agreed time parameter should be around 1 second

7. Challenges / Performance / Ideally equal to barcode scanning (100%) / Realistic expectations of performance should be at least as good as that of the average checker (~80%) / Performance / Ideally equal to barcode scanning (100%) / Realistic expectations of performance should be at least as good as that of the average checker (~80%)

8. Challenges / Ease of Use / Integrated into existing barcode reader housing / Minimal operator training / Ease of Use / Integrated into existing barcode reader housing / Minimal operator training

9. Challenges / System Training / Gradual adaption to variations in season, supplier, and ripeness/freshness / System Training / Gradual adaption to variations in season, supplier, and ripeness/freshness

10. Challenges / Database size / Seasonal / Varies by store, region, and harvest / Database size / Seasonal / Varies by store, region, and harvest

11. Solutions / Parallel vs. perpendicular polarization for filtering out glare from the light source

12. Solutions / 2 images used for segmentation, one light and one dark / Brightness variation greater than the threshold (T ∆ ) are considered foreground / 2 images used for segmentation, one light and one dark / Brightness variation greater than the threshold (T ∆ ) are considered foreground

13. Solutions / Notice the plastic bag is also illuminated / Another threshold (T dark ) is used to identify the transparent bags / Notice the plastic bag is also illuminated / Another threshold (T dark ) is used to identify the transparent bags

14. Question Session #1 / Any other illumination-related problems in this scenario?

15. Question Session #1 / Any other illumination-related problems in this scenario? / Dark produce / Waxed or shiny produce / Any other illumination-related problems in this scenario? / Dark produce / Waxed or shiny produce

16. Segmentation Results / Necessary Conditions / Stationary / Scale assistance ensures stable produce / Necessary Conditions / Stationary / Scale assistance ensures stable produce

17. Feature Selection / Histograms used extensively for feature representation / Much smaller than actual images / Well researched method for representation of visual cues / Training issues / Histograms used extensively for feature representation / Much smaller than actual images / Well researched method for representation of visual cues / Training issues

18. Color Features / HSI (HSL) color space

19. Apples vs Oranges

20. Texture Features / More difficult to describe computationally / Textel--artichokes or pineapples / Random variation--parsley / More difficult to describe computationally / Textel--artichokes or pineapples / Random variation--parsley

21. Texture Detection Methods / Measure A / Convolution of crossed bar masks [ -1 2 -1] [-1 -1 2 2 -1 -1] / Measure A / Convolution of crossed bar masks [ -1 2 -1] [-1 -1 2 2 -1 -1]

22. Texture Detection Methods / Method B / Deviation of image intensity from its nearest neighbors / Performed on reduced images for speed / Method B / Deviation of image intensity from its nearest neighbors / Performed on reduced images for speed

23. Question Session #2 / Which method would you use?

24. Question Session #2 / Results

25. Classification & Training / Symbols / P i, i = 1,2,….,N prototype histograms / Q, a produce histogram to be identified / Each histogram has 4 components / F = {hue, saturation, intensity, texture} / Normalized / Symbols / P i, i = 1,2,….,N prototype histograms / Q, a produce histogram to be identified / Each histogram has 4 components / F = {hue, saturation, intensity, texture} / Normalized

26. Classification & Training / Symbols / Each prototype histogram P i is associated with an identifier I(P i ) / How to compute distance Manhattan style / Symbols / Each prototype histogram P i is associated with an identifier I(P i ) / How to compute distance Manhattan style

27. Classification & Training / Distance equation

28. Classification & Training / Now that the distance is calculated / Decision Rule 1 / Now that the distance is calculated / Decision Rule 1

29. Classification & Training / For acceptable threshold but multiple identifiers / Decision Rule 2 / For acceptable threshold but multiple identifiers / Decision Rule 2

30. Classification & Training / Out of bounds / Decision Rule 3 / Out of bounds / Decision Rule 3

31. Acting on Results / Sure / Directly accepted by the register / Okay & Uncertain / Sorted nearest matches displayed / Sure / Directly accepted by the register / Okay & Uncertain / Sorted nearest matches displayed

32. Acting on Results / If none of the options are chosen, a new prototype for P chosen is added / The correctly identified Q’s prototype is judged for accuracy / All non-chosen prototypes of the P class are ‘aged’ / If none of the options are chosen, a new prototype for P chosen is added / The correctly identified Q’s prototype is judged for accuracy / All non-chosen prototypes of the P class are ‘aged’

33. Question Session #3 How do you think it did?

34. Overall Results / Feature Success trial 1

35. Overall Results / Feature Success trial 2

36. Overall Results / Training technique analysis

37. Overall Results

38. Super Straigtforward Results