1. © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon University

2. © Devi Parikh 2008 Motivation UPC Barcode QR CodeDatamatrix

3. © Devi Parikh 2008 HCCB Microsoft’s High Capacity Color Barcode

4. © Devi Parikh 2008 Application Uniquely identifying commercial audiovisual works such as motion pictures, video games, broadcasts, digital video recordings and other media

5. © Devi Parikh 2008 Goal Locate and Segment the barcode from consumer images

6. © Devi Parikh 2008 Overview  Design specifications of Microsoft’s HCCB  Approach  Localization  Segmentation  Progressive Strategy  Results  Conclusions

7. © Devi Parikh 2008 Microsoft’s HCCB 4 or 8 colors Triangles String of colors palette

8. © Devi Parikh 2008 Microsoft’s HCCB

9. © Devi Parikh 2008 Microsoft’s HCCB

10. © Devi Parikh 2008 Microsoft’s HCCB

11. © Devi Parikh 2008 Microsoft’s HCCB R rows S symbols per row S = (r+1)*R Aspect ratio: r

12. © Devi Parikh 2008 Approach Thresholding Orientation prediction Corner localization Row localization Symbol localization Color assignments Barcode localization Barcode segmentation point inside the barcode is known

13. © Devi Parikh 2008 Localization: Thresholding  Identify thick white band and row separators  Normalization  Adaptive

14. © Devi Parikh 2008 Localization: Orientation orientation distance -90900 summation

15. © Devi Parikh 2008 Localization: Corners  Rough estimates whiteness masknon-texture maskcombined mask

16. © Devi Parikh 2008 Localization: Corners  Gradient based refinement

17. © Devi Parikh 2008 Localization: Corners  Line based refinement

18. © Devi Parikh 2008 Segmentation: Rows Summation Flip?

19. © Devi Parikh 2008 Segmentation: Symbols S E Local search Number of symbols per row q(S,E) =  q(samples|S,E)

20. © Devi Parikh 2008 Segmentation: Colors Palette

21. © Devi Parikh 2008  Segmentation results given accurate localization  Satisfactory  Corner localization  Unsatisfactory  No one strategy works well on all images  However (1) Errors of different strategies are complementary (2) Results are verifiable with decoder in the loop! Observations

22. © Devi Parikh 2008 Progressive strategy  Hence – progressive strategy!  Similar to ensemble of weak classifiers  Or hypothesize-and-test  Multiple strategies:  Rough + gradient + line, or rough + line, or rough + gradient, or rough alone  Different values of threshold during rough corner detection  Total 12  Order of strategies

23. © Devi Parikh 2008 Results  Dataset of 500 images  Performance metric: % barcodes successfully decoded  Decoder model: Barcode successfully decoded if 80% of symbols are correctly identified

24. © Devi Parikh 2008 Results Allows for explicit trade-off between accuracy and computational time

25. © Devi Parikh 2008 Results

26. © Devi Parikh 2008 Results

27. © Devi Parikh 2008 Results

28. © Devi Parikh 2008 Results

29. © Devi Parikh 2008 Results

30. © Devi Parikh 2008 Results

31. © Devi Parikh 2008 Results

32. © Devi Parikh 2008 Results

33. © Devi Parikh 2008 Results

34. © Devi Parikh 2008 Results

35. © Devi Parikh 2008 Conclusions  2D High Capacity Color Barcode (HCCB)  Successful localization and segmentation of HCCB from consumer images  Varying densities, aspect ratios, lighting, color balance, image quality, etc.  Simple computer vision and image processing techniques  Progressive strategy

36. © Devi Parikh 2008 Acknowledgements Microsoft Research  Larry Zitnick  Andy Wilson  Zhengyou Zhang Carnegie Mellon University  Advisor: Tsuhan Chen

37. © Devi Parikh 2008 Thank you!