Copyright Protection of Images Based on Large-Scale Image Recognition Koichi Kise, Satoshi Yokota, Akira Shiozaki Osaka Prefecture University

Outline  Demo  Image Recognition with Local Descriptors  Our Research Topics  Copyright Protection and Digital Watermarking  Proposed Method  Experiments  Conclusion 2

DEMO Large-Scale Image Recognition

Outline  Demo  Image Recognition with Local Descriptors  Our Research Topics  Copyright Protection and Digital Watermarking  Proposed Method  Experiments  Conclusion 4

Image Recognition with Local Descriptors Database Images are represented by local descriptors (feature vectors) Typically a few hundreds to a few thousands are extracted Image 1 Image 2 Image 3 Image 4 Query 5

Database Image 1 Image 2 Image 3 Image 4 Query Image Recognition with Local Descriptors Nearest Neighbor Search no. of votes max. votes 6 Nearest vector in the DB Nearest vector in the DB

Pros and Cons  Extraction of descriptors a few seconds / query  Matching By brute-force NN search with the DB of 1,000 images 315 seconds / query PROS CONS Robust to occlusion and variations Locality and stability of descriptors Large number of expensive descriptors 7

Outline  Demo  Image Recognition with Local Descriptors  Our Research Topics  Copyright Protection and Digital Watermarking  Proposed Method  Experiments  Conclusion 8

Our Research Topics Extraction of Descriptors Matching (with 10,000 images) BaselineSoftware2.3 sec.Brute-force3,000 sec. Existing Technologies ANN (Approximate Nearest Neighbor) 50 ms 60,000 times faster Our MethodsGPU & CUDA0.57 sec.Hash & Cascade0.78 ms 4 times faster60 times faster 9 Application Copyright Protection

Outline  Demo  Image Recognition with Local Descriptors  Our Research Topics  Copyright Protection and Digital Watermarking  Proposed Method  Experiments  Conclusion 10

Copyright Protection and Digital Watermarking  Digital Watermarking is a technology to embed information (copyright notice) to original digital data.  Embedded information (watermark) is often unnoticeable.  We focus on digital images as original data.  Methods:  Non-blind  Blind 11 Copyright: A Purchaser: B Copyright: A Purchaser: B original watermarked

Non-Blind and Blind Watermarking  Non-blind watermarking  Blind watermarking  History  From non-blind to blind watermarking  Non-blind watermarking needs matching of images  spoils the scalability  Blind watermarking has been the focus of interest 12 watermarked image & originalimage Copyright: A Purchaser: B Copyright: A Purchaser: B watermarked image Copyright: A Purchaser: B Copyright: A Purchaser: B

Limitations of Blind Watermarking  StirMark  distorts images imperceptibly  severely damages the wartermark 13 OriginalStirMark

StirMark: Some Image Transformations 14 corner shift cropping punch / pinch original fraction of pixel displacement

Outline  Demo  Image Recognition with Local Descriptors  Our Research Topics  Copyright Protection and Digital Watermarking  Proposed Method  Experiments  Conclusion 15

Proposed Method: Ideas  How do we solve the problem of blind watermarking?  Cancel the effect of StirMark, especially geometric transformations  Normalization could be done by comparing with the original image  Blind to non-blind 16 StirMark’ed OriginalStirMark’ed normalization normalized StirMark’ed

Proposed Method: Ideas  How to solve the problem of non-blind watermarking?  Matching problem  solved by the efficient and robust image recognition  How about the normalization?  Image recognition with local descriptors  provides matching of descriptors as a side effect  helpful for normalization 17

Proposed Method: Ideas  To solve the problem of StirMark  Image normalization is required to detect the watermark  Normalization could be done by comparing with the original image  Non-blind watermarking  Efficient and robust image recognition  overcome the difficulty of matching in non-blind watermarking  precise matching of local descriptors (feature points) is obtained as a side effect of recognition  This helps us to normalize the StirMark’ed image to detect the watermark 18

Proposed Method: Processing Steps 19 STEP1: Recognition Image Database query image point matching STEP2: Normalization STEP3: Detection Embedded Watermark

Proposed Method: Processing Steps 20 STEP2: Normalization Embedded Watermark STEP3: Detection Local Descriptors: PCA-SIFT Matching: ANN Local Descriptors: PCA-SIFT Matching: ANN STEP1: Recognition Image Database query image point matching

Outline  Demo  Image Recognition with Local Descriptors  Our Research Topics  Copyright Protection and Digital Watermarking  Proposed Method  Experiments  Conclusion 21

Experiments: Overview  Experiment 1  Robustness against StirMark with default setting (imperceptible change)  Experiment 2  Robustness against perceptible changes  Experiment 3  Robustness against watermarking 22

Experiments: Overview  Experiment 1  Robustness against StirMark with default setting (imperceptible change)  Experiment 2  Robustness against perceptible changes  Experiment 3  Robustness against watermarking 23

Experiments: Dataset & Task 24 10,000 images 100 Base set WatermarkStirMark query images Input : query image Task: find its original from 10,000 images Input : query image Task: find its original from 10,000 images

Experiment 1 25 imperceptible 100 random application with default setting 10,000 images 100 StirMark query images 10,000 images 100 query images watermarkingwithoutwith accuracy100%99.49% time (per query)217ms216ms Watermark with StirMark without 10,000

Experiment2  4 image transformations  cropping, fraction of pixel displacement, punch/pinch, corner shift  perceptible 26 perceptible 4 trans. 43 application in total 10,000 images 100 StirMark query images Watermark 10,000 images 100 StirMark query images with without 4,300

97% Cropping: 100 ; Accuracy: 97%

Cropping: Original Image

97% Cropping: 70; Accuracy 97% 29

95% Cropping: 50; Accuracy: 95% 30

92% Cropping: 30; Accuracy: 92% 31

Cropping 32 With watermarking Without watermarking Accuracy [%] Cropping parameter

Punch / Pinch Original Image 33

99% Punch / Pinch: 1; Accuracy: 99% 34

99% Punch / Pinch: 25; Accuracy: 99% 35

97% Punch / Pinch: 50; Accuracy: 97% 36

50% Punch / Pinch: 100; Accuracy: 50% 37

Punch / Pinch 38 Accuracy [%] Punch / Pinch Parameter With watermarking Without watermarking

Conclusion  A non-blind framework for copyright protection of images has been proposed  Efficient image recognition technology with local descriptors  Fundamental experiments on image recognition  Image recognition is robust to StirMark attacks and watermarking  Future Work  Completion of overall method 39

Copyright Protection of Images Based on Large-Scale Image Recognition Thank you for your attention

Image Recognition: Local Descriptors  PCA-SIFT (PCA Scale-Invariant Feature Transform)  36 dimensions  Invariant to scale and rotation changes 41

Image Recognition: Matching  ANN(Approximate Nearest Neighbor)  Tree structure Feature Space Feature Vector

ANN ： Feature Vector of a query

ANN Nearest Neighbor Search for

ANN NN Search for Approximate Result This cell is not searched

ANN more approximation with a smaller circle Wrong Correct Errors caused by approximation NN Search for Approximate

NNS accuracy vs. Image recognition rate Image recognition rate[%] NNS accuracy[%] ANN dist E2LSH

Corner Shift Original Image 48

99% Corner Shift: 1; Accuracy: 99% 49

98% Corner Shift: 100; Accuracy: 98% 50

96% Corner Shift: 150; Accuracy: 96% 51

74% Corner Shift: 225; Accuracy: 74% 52

Corner Shift 53 Accuracy [%] Corner Shift Parameter With watermarking Without watermarking

Fraction of Pixel Displacement Original Image 54

99% Fraction of Pixel Displacement: 1; Accuracy 99% 55

94% Fraction of Pixel Displacement: 3 Accuracy: 94% 56

81% Fraction of Pixel Displacement: 5; Accuracy: 81% 57

58% Fraction of Pixel Displacement: 8 Accuracy: 58% 58

24% Fraction of Pixel Displacement: 10 Accuracy: 24% 59

Fraction of Pixel Displacement 60 Accuracy [%] Fraction Parameter With watermarking Without watermarking