© Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon University

© Devi Parikh 2008 Motivation UPC Barcode QR CodeDatamatrix

© Devi Parikh 2008 HCCB Microsoft’s High Capacity Color Barcode

© Devi Parikh 2008 Application Uniquely identifying commercial audiovisual works such as motion pictures, video games, broadcasts, digital video recordings and other media

© Devi Parikh 2008 Goal Locate and Segment the barcode from consumer images

© Devi Parikh 2008 Overview  Design specifications of Microsoft’s HCCB  Approach  Localization  Segmentation  Progressive Strategy  Results  Conclusions

© Devi Parikh 2008 Microsoft’s HCCB 4 or 8 colors Triangles String of colors palette

© Devi Parikh 2008 Microsoft’s HCCB

© Devi Parikh 2008 Microsoft’s HCCB

© Devi Parikh 2008 Microsoft’s HCCB

© Devi Parikh 2008 Microsoft’s HCCB R rows S symbols per row S = (r+1)*R Aspect ratio: r

© Devi Parikh 2008 Approach Thresholding Orientation prediction Corner localization Row localization Symbol localization Color assignments Barcode localization Barcode segmentation point inside the barcode is known

© Devi Parikh 2008 Localization: Thresholding  Identify thick white band and row separators  Normalization  Adaptive

© Devi Parikh 2008 Localization: Orientation orientation distance summation

© Devi Parikh 2008 Localization: Corners  Rough estimates whiteness masknon-texture maskcombined mask

© Devi Parikh 2008 Localization: Corners  Gradient based refinement

© Devi Parikh 2008 Localization: Corners  Line based refinement

© Devi Parikh 2008 Segmentation: Rows Summation Flip?

© Devi Parikh 2008 Segmentation: Symbols S E Local search Number of symbols per row q(S,E) =  q(samples|S,E)

© Devi Parikh 2008 Segmentation: Colors Palette

© Devi Parikh 2008  Segmentation results given accurate localization  Satisfactory  Corner localization  Unsatisfactory  No one strategy works well on all images  However (1) Errors of different strategies are complementary (2) Results are verifiable with decoder in the loop! Observations

© Devi Parikh 2008 Progressive strategy  Hence – progressive strategy!  Similar to ensemble of weak classifiers  Or hypothesize-and-test  Multiple strategies:  Rough + gradient + line, or rough + line, or rough + gradient, or rough alone  Different values of threshold during rough corner detection  Total 12  Order of strategies

© Devi Parikh 2008 Results  Dataset of 500 images  Performance metric: % barcodes successfully decoded  Decoder model: Barcode successfully decoded if 80% of symbols are correctly identified

© Devi Parikh 2008 Results Allows for explicit trade-off between accuracy and computational time

© Devi Parikh 2008 Results

© Devi Parikh 2008 Results

© Devi Parikh 2008 Results

© Devi Parikh 2008 Results

© Devi Parikh 2008 Results

© Devi Parikh 2008 Results

© Devi Parikh 2008 Results

© Devi Parikh 2008 Results

© Devi Parikh 2008 Results

© Devi Parikh 2008 Results

© Devi Parikh 2008 Conclusions  2D High Capacity Color Barcode (HCCB)  Successful localization and segmentation of HCCB from consumer images  Varying densities, aspect ratios, lighting, color balance, image quality, etc.  Simple computer vision and image processing techniques  Progressive strategy

© Devi Parikh 2008 Acknowledgements Microsoft Research  Larry Zitnick  Andy Wilson  Zhengyou Zhang Carnegie Mellon University  Advisor: Tsuhan Chen

© Devi Parikh 2008 Thank you!