ARTcode: Preserve Art and Code In Any Image Zhe Yang, Yuting Bao, Chuhao Luo, Xingya Zhao, Siyu Zhu, Chunyi Peng, Yunxin Liu, Xinbing Wang (Shanghai Jiao Tong University, The Ohio State University, Microsoft Research)

Outlines Introduction ARTcode Architecture How to Preserve Art in ARTcode Reliable ARTcode Detection and Decoding ARTcode Detection ARTcode Decoding Evaluation 2

Outlines Introduction ARTcode Architecture How to Preserve Art in ARTcode Reliable ARTcode Detection and Decoding ARTcode Detection ARTcode Decoding Evaluation 3

Introduction 4  In scenarios like electronic payment, state-of-the-art technologies apply barcodes such as QR code, shown on left.  What if we scan the picture of an article directly, as shown in the left image?

Introduction 5  These three barcodes encode the same information: URL to Wikipedia page of Lenna. Which one could help identify the contained information?

Introduction 6  The introduction of ARTcode has two advantages:  Traditional barcodes demand more space to display both human-readable and machine-recognizable information while ARTcode is a two-in-one visualization code.  Traditional barcodes impair user experience while ARTcode takes the form of input image.

Introduction 7  Goals of ARTcode:  Carry the human-friendly information (mostly, artistic images and layouts) and machine-friendly information (coded data bits) together in the same source over the visual channel.  Preserve pleasant userviewing experience (as if the device-oriented code would not exist).

Outlines Introduction ARTcode Architecture How to Preserve Art in ARTcode Reliable ARTcode Detection and Decoding ARTcode Detection ARTcode Decoding Evaluation 8

ARTcode Artchitecture 9

Outlines Introduction ARTcode Architecture How to Preserve Art in ARTcode Reliable ARTcode Detection and Decoding ARTcode Detection ARTcode Decoding Evaluation 10

ARTcode Encoding 11  To address this tension between both communications for humans and devices, we tackle the following three technical questions:  How to make changes to an image to largely keep its form and enable data embedment?  Colored dot matrix and Shuffling  How to reduce visual distortion in data embedment?  Embedding Data with Minor Modifications  How to manipulate colors to encode data?  Encoding Color to Bit

ARTcode Encoding 12  Colored dot matrix and Shuffling  ARTCODE uses a pointillism-like blending technique to generate a structure, maintaining original images’ general appearance, called colored dot matrix.

ARTcode Encoding 13  Colored dot matrix generation consists of two phases: Color palette selection by clustering. We select the colors that represent a majority of the original image to form a color palette. Revised error diffusion dithering. Dithering is a technique that converts complicated colored or gray-scale images to images with a palette of much fewer colors, retaining original images’ appearance.

ARTcode Encoding 14  Shuffling Algorithm finds embeddable modules: Shuffling algorithm spreads encoding modules over the dot matrix according to a shuffling table. The values of shuffling table are randomly generated. Shuffling algorithm can achieve multiple message transmission.

ARTcode Encoding 15  Embedding Data with Minor Modifications.  In ARTcode, the embedment of machine-readable information in the dot matrix is regarded as noises for human perception.  In order to trade off the capacity and visual quality demands of ARTcode, we integrate a data hiding algorithm and adapt it to the present work.  The key idea is, in a -length vector, by changing only 1 bit, it can embed bits data, for there are in total positions in this vector, plus 1 original unchanged vector.

ARTcode Encoding 16  Encoding Color to Bit.  Two considerations: To guarantee decoding accuracy, encoding colors should be “sufficiently distant” from each other. Visual effect preservation demands that selected encoding colors take large proportion in embedding positions.

ARTcode Encoding 17  Encoding Color to Bit.  Four steps: Step1. In color palette, calculate all n colors’ proportion they occupy in L encoding modules. Step2. Find all feasible color sets. Colors in feasible sets are “sufficiently distant” from each other. Step3. Calculate the sum of colors’ proportion in each feasible color set. Select the color set with the largest proportion. Step4. If, by any chance, the algorithm fails to find a feasible color set, the algorithm decreases the threshold until feasible color sets are found.

Outlines Introduction ARTcode Architecture How to Preserve Art in ARTcode Reliable ARTcode Detection and Decoding ARTcode Detection ARTcode Decoding Evaluation 18

ARTcode Detection 19  Pre-processing  Local thresholding binarization  Erosion-dilation to eliminate moiré pattern

ARTcode Detection 20  Module Localization  As huge finder patterns affect visual, we do not insert locators in the code area and place an alignment pattern on borders of the barcode for module localization.

ARTcode Detection 21  Module Localization  Module localization detects the following entities in sequence:  The localization of code area modules considers distortion introduced by camera. ARTcode corners ARTcode double deck alignment Each module in code area

Outlines Introduction ARTcode Architecture How to Preserve Art in ARTcode Reliable ARTcode Detection and Decoding ARTcode Detection ARTcode Decoding Evaluation 22

ARTcode Decoding 23  Color recognition  Color recognition assigns nearest RGB values in encoding color set obtained by double deck alignment pattern.  Comprehensive Error Correction  Both Reed Solomon error correction and Cyclic Redundancy Code error detection.  If CRC algorithm detects an error, another frame will be sent in for detection and decoding.

Outlines Introduction ARTcode Architecture How to Preserve Art in ARTcode Reliable ARTcode Detection and Decoding ARTcode Detection ARTcode Decoding Evaluation 24

Evaluation 25  Image set.  197 images from Google Image, under keywords “Logo”, “Gray-scale”, “Portrait”, “Landscape”, “Animal”, and also classical images like “Lenna”.  25 gray-scale images, 22 logos and 150 colored images.  48 low contrast, 64 intermediate contrast and 38 high contrast images.

Evaluation 26  Default setting.  50 bytes, 16 clustered colors, embedding block size 15.  Six examples: (a) Gray(b) Logo1(c) Logo2(d) Elephant (e) Butterfly (f) Lenna

Evaluation 27  Overall Performance  User study with 50 participants. Each participant gives a subjective distortion score δ from 0 to distortion not perceptible, noticeable but acceptable, heavily corrupted.

Evaluation 28  Overall Performance.  Information-carrying accuracy. PC screen as the sender and each code is 8.4 × 8.4. Camera-screen distance 18.

Evaluation 29  Assessment on Image Visual Quality.

Evaluation 30  Information-Carrying Capability.

Thanks ！