1. ENEE631 Digital Image Processing (Spring'09) Image Forensics Spring ’09 Instructor: Min Wu Electrical and Computer Engineering Department, University of Maryland, College Park   bb.eng.umd.edu (select ENEE631 S’09)   minwu@eng.umd.edu ENEE631 Spring’09 Lecture 22 (4/22/2009) UMCP ENEE631 Slides (created by M.Wu © 2004)

2. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [2] Overview and Logistics Last Time: –Fragile data embedding for image forgery detection –Improving watermark robustness via quantization –Robust watermark via spread spectrum embedding Today: –Continue on spread spectrum watermark –Digital forensic fingerprinting for traitor tracing –Non-intrusive image forensics UMCP ENEE631 Slides (created by M.Wu © 2004)

3. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [3] From Fragile/Semi-Fragile to Robust Watermark Applications of fragile/semi-fragile watermark –Tampering detection –Secret communications => “Steganography” (covert writing) –Convey side info. in a seamless way: lyric, director’s notes Situations demanding higher robustness –Protect ownership (copyright label), prevent leak (digital fingerprint) –Desired robustness against compression, filtering, etc. How to make it robust? –Use “quantization” from signal processing –Use error correcting coding –Borrow theories from signal detection & telecommunications u “Spread Spectrum Watermark”: use “noise” as watermark and add it to the host signal for improved invisibility and robustness UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

4. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [4] Background: Robust Watermark / Data-Embedding u Embedding domain tailored to media characteristics & application requirement 10011010 … © Copyright …

5. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [5] Spread Spectrum Watermark: Cox et al (NECI) What to use as watermark? Where to put it? –Place wmk in perceptually significant spectrum (for robustness) u Modify by a small amount below Just-noticeable-difference (JND) –Use long random noise-like vector as watermark u for robustness/security against jamming+removal & imperceptibility Embedding v’ i = v i +  v i w i = v i (1+  w i ) –Perform DCT on entire image and embed wmk in DCT coeff. –Choose N=1000 largest AC coeff. and scale {v i } by a random factor 2D DCTsort v’=v (1+  w) IDCT & normalize Original image N largest coeff. other coeff. marked image random vector generator wmk seed UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

6. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [6] Watermarking Example by Cox et al. Original Cox Difference between whole image DCTmarked and orig. Embed in 1000 largest coeff. UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

7. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [7] Cox et al’s Scheme (cont’d): Detection  Subtract original image from the test one before feeding to detector (“non-blind detection”)  Correlation-based detection u a correlator normalized by |Y| in Cox et al. paper DCT compute similarity threshold test image decision wmk DCTselect N largest original unmarked image select N largest preprocess – – orig X test X’ X’=X+W+N ? X’=X+N ? To think UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

8. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [8] Performance of Cox et al’s Scheme Robustness –(claimed) scaling, JPEG, dithering, cropping, “printing-xeroxing- scanning”, multiple watermarking –No big surprise with high robustness u equiv. to sending just 1-bit {0,1} with O(10 3 ) samples Comment –Must store orig. unmarked image  “private wmk”/“non-blind” detection –Perform image registration if necessary –Adjustable parameters: N and  UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

9. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [9] Comments on Cox et al’s Schemes “1000 largest coeff.” before and after embedding –May not be identical (and order may also changes) –Solutions: use orig. as ref; “embeddable” mask to maintain synch. Detection without using original/host image –Treat host image as part of the noise/interference ~ Blind detection u need long wmk signal to combat severe host interference [Zeng-Liu] –Can do better than blind detection, as embedder knows the host signal => “Embedding with Side Info.” H0: H1: = vs. H0: = H1: =

10. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [10] Improve Invisibility and Robustness on Cox scheme Apply better Human Perceptual Model –Global scaling factor is not suitable for all coefficients –More explicitly compute just-noticeable-difference (JND) u JND ~ max amount each coefficient can be modified invisibly u Employ human visual model: freq. sensitivity, masking, … –Use more localized transform => fine tune wmk for each region u block-based DCT; wavelet transform Improve robustness: detection performance depends on ||s|| /  d u Add a watermark as strong as JND allows u Embed in as many “embeddable” coeff. => improve robustness Block-DCT schemes: Podichuk-Zeng; Swanson-Zhu-Tewfik ’97 –Leverage existing visual model for block DCT from JPEG UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

11. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [11] Perceptual Comparison: Cox vs. Podilchuk OriginalCox Podilchuk whole image DCT block-DCT Embed in 1000 largest coeff. Embed to all “embeddables” UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

12. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [12] Compare Cox & Podilchuk Schemes (cont’d) CoxPodilchuk Amplified pixel-wise difference between marked and original (gray~0) UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

13. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [13] Comments on Cox & Podilchuk’s SS Wmk Robustness –Very robust against additive noise (seen from detection theory) –Sensitive to synchronization errors, esp. under blind detection u jitters (line dropping/addition) u geometric distortion (rotation, scale, translation) Question: How to improving synchronization resilience? => add registration pattern; embed in RST-invariant domain; …

14. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [14] Localized Embedding: Double-Edge Sword “Innocent Tools” exploited by attackers: block concealment Recovery of lost blocks –for resilient multimedia transmission of JPEG/MPEG –good quality by edge-directed interpolation: Jung et al; Zeng-Liu Remove robust watermark by block replacement edge estimation edge-directed interpolation UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

15. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [15] Block Replacement Attack

16. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [16] Attack effective on block-DCT based spread-spectrum watermark marked original (no distortion)JPEG 10%after proposed attack 512x512 lenna Threshold: 3 ~ 6 Recall: claimed high robustness&quality by fine tuning wmk strength for each region

17. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [17]

18. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [18] Watermark Attacks: What and Why? Attacks: intentionally obliterate watermarks –remove a robust watermark –make watermark undetectable (e.g., miss synchronization) –uncertainty in detection (e.g., multiple ownership claims) –forge a valid (fragile) watermark –bypass watermark detector Why study attacks? –identify weaknesses –propose improvement –understand pros and limitation of tech. solution To win each campaign, a general should know both his troop and the opponent’s as well as possible. -- Sun Tzu, The Art of War, 500 B.C. UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

19. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [19] Summary: Spread Spectrum Embedding Main ideas –Place wmk in perceptually significant spectrum (for robustness) u Modify by a small amount below Just-noticeable-difference (JND) –Use long random vector of low power as watermark to avoid artifacts (for imperceptibility, robustness, and security) Cox’s approach –Perform DCT on entire image & embed wmk in large DCT AC coeff. –Embedding: v’ i = v i +  v i w i = v i (1+  w i ) –Detection: subtract original and perform correlation w/ wmk Podilchuk’s improvement –Embed in many “embeddable” coeff. in block-DCT domain –Adjust watermark strength by explicitly computing JND

20. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [20] Suggested Readings 1.I. Cox, J. Kilian, T. Leighton, T. Shamoon: “Secure Spread Spectrum Watermarking for Multimedia'', IEEE Trans. on Image Proc., vol.6, no.12, pp.1673-1687, 1997. 2.M. M. Yeung, F. Mintzer: “An Invisible Watermarking Technique for Image Verification", Proc. of the IEEE Int’l Conf. on Image Processing (ICIP), Oct. 1997. 3.M. Wu and B. Liu: "Data Hiding in Image and Video: Part-I -- Fundamental Issues and Solutions", IEEE Trans. on Image Proc., vol.12, no.6, pp.685- 695, June 2003. 4.M. Wu, W. Trappe, Z.J. Wang, and K.J.R. Liu: “Collusion-resistant fingerprinting for Multimedia,” IEEE Signal Proc Magazine, March 2004. 5.M. Wu and B. Liu: Multimedia Data Hiding, Springer-Verlag, 2003. 6.I. Cox, M. Miller, and J. Bloom: Digital Watermarking, Morgan Kauffman, 2002. And the related references cited by these publications. UMCP ENEE631 Slides (created by M.Wu © 2004)

21. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [21] Summary: Type-I Additive Embedding Add secondary signal in host media Representative: spread spectrum embedding –Add a noise-like signal and detection via correlation –Good tradeoff between security, imperceptibility & robustness –Limited capacity: host signal often appears as major interferer modulation data to be hidden  X original source X’ = X +  marked copy 10110100... = UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

22. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [22] Type-II Relationship Enforcement Embedding Deterministically enforcing relationship –Secondary information carried solely in watermarked signal –Typical relationship: parity/modulo in quantized features Representative: odd-even (quantized) embedding –Alternative view: switching between two quantizers w/ step size 2Q u “Quantization Index Modulation” –Robustness achieved by quantization or tolerance zone –High capacity but limited robustness even “0” odd “1” feature value 2kQ (2k+1)Q (2k+2)Q (2k+3)Q odd-even mapping lookup table mapping 0 1 0 1 … 0 1 1 0 … UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

23. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [23] Robustness vs. Payload Blind/non-coherent detection ~ original copy unavailable Robustness and payload tradeoff Advanced embedding: quantization w/ distortion-compensation –Combining the two types with techniques suggested by info. theoryRobustnessPayload Imperceptibility noise stronger noise weaker UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

24. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [24] Distortion Compensated Quantization Embedding Distortion compensation technique –Increase quantization step by a factor  for higher robustness –Compensate the extra distortion by dragging the enforced feature toward the original feature value Overall embedding distortion unchanged UMCP ENEE631 Slides (created by M.Wu © based on Talks ’03, updated’07-’09) odd/even mapping 0 1 0 X 0 Choose alpha to maximize a distortion-compensation SNR

25. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [25] Improved Robustness by Distortion Compensation –ICS (ideal Costa’s scheme) –SS (spread spectrum additive embedding) –binary DM (odd-even quantized embedding) –binary SCS (odd-even quantized embedding with distortion compensation)

26. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [26] Issues to Address Tradeoff among conflicting requirements –Imperceptibility –Robustness & security –Capacity Key elements of data hiding –Perceptual model –Embedding one bit –Multiple bits –Uneven embedding capacity –Robustness and security –What data to embed Upper Layers Uneven capacity equalization Error correction Security …… Lower Layers Imperceptible embedding of one bit Multiple-bit embedding Coding of embedded data Robustness Capacity Imperceptibility UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

27. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [27] Robust Wmk Application for Tracing Traitors Leak of information as well as alteration and repackaging poses serious threats to government operations and commercial markets –e.g., pirated content or classified document Promising countermeasure: robustly embed digital fingerprints –Insert ID or “fingerprint” (often through conventional watermarking) to identify each user –Purpose: deter information leakage; digital rights management (DRM) –Challenge: imperceptibility, robustness, tracing capability studio The Lord of the Ring Alice Bob Carl w1 w2 w3Sell UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

28. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [28] Embedded Fingerprint for Tracing Traitors Insert special signals to identify recipients –Deter leak of proprietary documents –Complementary protection to encryption –Consider imperceptibility, robustness, traceability –Attacks mounted by single and multiple users Multi-user Attacks Traitor Tracing President Satellite Image Alice Bob Carl w1 w2 w3 Leak

29. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [29] Potential civilian use for digital rights management (DRM) u u Copyright industry – $500+ Billion business ~ 5% U.S. GDP Alleged Movie Pirate Arrested (23 January 2004) – –A real case of a successful deployment of 'traitor-tracing' mechanism in the digital realm – –Use invisible fingerprints to protect screener copies of pre- release movies Carmine CaridiRussellfriends … Internet w1 Last Samurai Hollywood studio traced pirated version http://www.msnbc.msn.com/id/4037016/ Case Study: Tracing Movie Screening Copies UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

30. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [30] Collusion Attacks by Multiple Users Collusion: A cost-effective attack against multimedia fingerprints – –Users with same content but different fingerprints come together to produce a new copy with diminished or attenuated fingerprints – –Fairness: Each colluder contributes equal share through averaging, interleaving, and nonlinear combining Collusion by averaging 1/3 AliceChris Bob Colluded copy Originally fingerprinted copies AliceChris Collage

31. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [31] Key Issues How to construct fingerprints? –Identify individual users –Resist multi-user collusion How to embed fingerprints in media data? –Tailor to media characteristics for robustness & imperceptibility Interaction between choices of fingerprint construction, embedding, and detection –esp. to combat collusion attacks –Analogous to “cross-layer” methods in communications

32. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [32] Road Map on Media Fingerprinting Research Robust Embedding Orthogonal Fingerprints Represent how many users? Resist how many colluders? “most effective” collusions? Amount of resources used? Group-based FP to exploit Attacker Behavior Coded FP Joint Coding-Embedding Framework overcome prior work’s problems of long code length, low resilience, and limited scalability adapt to media characteristics Combinatorial codes + CDM Error correcting codes + TDM Correlated Fingerprints

33. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [33] Example: Orthogonal Fingerprint for Curves/Graphics Use (approx.) orthogonal sequences as FPs for different users –Detection by looking for high correlation result Embed in parametric modeling domain of curve –Perturb B-spline parameters according to spread spectrum sequences Detection Statistics Typical threshold is 3~6 for false alarm of 10 -3 ~ 10 -9 Original Curve (captured by TabletPC) Fingerprinted Curve (100 control points)

34. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [34] Fingerprinting Topographic Map –Traditional protection: intentionally alter geospatial content –Embed much less intrusive digital fingerprint for a modern protection 9 long curves are marked; 1331 control points used to carry the fingerprint 1100x1100 Original MapFingerprinted Map

35. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [35] Collusion-Resistant Fingerprinting of Maps 2-User Interleaving Attack5-User Averaging Attack...   Can survive combined attacks of collusion + print + scan   Can extend to 3-D Digital Elevation Map

36. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [36] Explore unique issues associated with multimedia – –Consider interaction b/w fingerprint encoding, embedding & detection – –Use appropriate embedding to prevent arbitrary manipulation on code Build correlated fingerprints in two steps – –Use antipodal coded modulation to embed fingerprint codes u u via orthogonal spread spectrum sequences u u shared bits get sustained and used to identify colluders – –Binary Anti-collusion fingerprint codes resist up to K colluders any subset of up to K users take same value @ unique set of positions 1st bit 2nd bit... Spreading + Combinatorial Coded Fingerprint

37. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [37] 16-bit Codes for Detecting  3 Colluders Out of 20 User-1 ( -1,-1, -1, -1, 1, 1, 1, 1, …, 1 ) ( -1, 1, 1, 1, 1, 1, …, -1, 1, 1, 1 ) User-4 Extracted fingerprint code ( -1, 0, 0, 0, 1, …, 0, 0, 0, 1, 1, 1 ) Collude by Averaging Uniquely Identify User 1 & 4 Embed fingerprint via HVS-based spread spectrum embedding in block-DCT domain

38. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [38] Many Forensic Questions … Many Forensic Questions … arise from military, intelligence, law enforcement, and commercial applications What type of sensor was used? Which camera brand took this picture? What model? What processing has been done? –Has it been tampered? manipulated? What technologies were employed? –Given two images, are they acquired by devices with similar imaging technologies?

39. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [39] Break down the info. processing chain into individual components Identify algorithms and parameters employed in major components of a digital device or processing system Exploit Intrinsic Fingerprints via Component Forensics Ref: Swaminathan/Wu/Liu in ICASSP’06 and IEEE Trans. Info Forensics & Security (’07) Color Filter Array (CFA) Color Interpolation White Balancing Real world scene Digital image Camera Components … Sensors …

40. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [40] App 1: Image Source Authentication “Acquisition forensics”: –Use component forensics to obtain good features to construct a robust camera identifier Use acquisition knowledge to detect tampering –E.g. various parts of a forged picture are obtained from different cameras –Provide ground-truth model on non-tampered picture directly out of camera Color Filter ArrayInterpolation Fujifilm S3000 Canon powershot A75 Camera 1 Camera 2

41. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [41] App 2: Technology Infringement, Licensing, Evolution Between brands –High similarity suggests either licensing or potential infringement => Improve efficiency + efficacy from existing practice with soft/hardware documentation Evolution Forensics –Different models over time/price tier –What components were modified? What remain the same? => Facilitate companies to understand competitors’ technologies and develop alliance strategies for future innovations Quantitative assessment on similarity & differences of major components Canon A75 Canon A85 Canon A95 Powershot S410 Canon Powershot S400 Year 2003 2004 2005

42. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [42] Types of Component Forensics Intrusive forensics –Devices in hand –Break it apart and identify every component Semi non-intrusive forensics –Devices in hand but not to break it apart –Design test conditions and inputs to improve estimation accuracy Completely non-intrusive forensics –Products /devices not in hand –Sample outputs from devices available

43. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [43] Forensic Estimation and Identification Establish a processing model and estimate parameters –Small # possibilities => exhaustive search or by classifier design –More continuous valued parameters => analyze based on estimation theory Example: color interpolation in digital camera –Approximate by texture classification and linear filter (one set of interpolation coeff. for smooth, horizontal & vertical) –Find best linear estimate of filter coeff. in each class (least-square type of method for robustness) –Find CFA pattern in a search space that minimizes fitting errors CFA Interpolation R ? ?? R ? ?? R ? ?? R ? ?? ? Candidate CFA pattern A x = b Interp. equation set A ~ non-interp pixels; b ~ interpolated pixels Interp. coeff. x and fitting error for each region type and color

44. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [44] Experiments with Images from Digital Cameras Camera Model 1 2 3 4 5 6 7 8 9 10 Canon Powershot A75 Canon Powershot S400 Canon Powershot S410 Canon Powershot S1 IS Canon Powershot G6 Canon EOS Digital REBEL Nikon E4300 Nikon E5400 Sony Cybershot DSC P7 Sony Cybershot DSC P72 11 12 13 14 15 16 17 18 19 Olympus C3100Z/C3020Z Olympus C765UZ Minolta DiMage S304 Minolta DiMage F100 Casio QV-UX2000 FujiFilm Finepix S3000 FujiFilm Finepix A500 Kodak CX6330 Epson PhotoPC 650 19 cameras and 200 image blocks per camera model – –512 x 512 regions with maximum gradients chosen for analysis (s.t. have substantial revealing evidence on color interpolation)

45. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [45] Detecting Which Camera Brand Took the Image CanonNikonSonyOlympusMinoltaCasioFujiKodakEpson Canon96%******** Nikon*83%5%****** Sony**90%****** Olympus***93%***** Minolta8%***81%**** Casio***6%*89%*** Fuji****7%*87%** Kodak*******89%* Epson********100% Interpolation coefficients as features for classifier Average accuracy: 90% for 9 camera brands on uncontrolled scenes Best related work under controlled, uncompressed setting on input scenes 84% for 3 brands, uncompressed [Kharrazi et al’ 05]; 96% for 3 brands [Bayram et al’ 06] (* denotes values below 4%)

46. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [46] 01020304050607080910111213141516171819 01*0.06 0.170.140.350.360.680.220.760.250.310.180.090.310.180.540.830.25 020.06*0.050.070.050.19 0.460.110.550.120.140.110.030.150.170.350.570.16 Canon030.060.05*0.150.060.180.220.470.220.510.230.270.200.100.290.330.310.570.27 040.170.070.15*0.100.220.230.500.140.710.040.080.100.070.100.190.490.580.23 050.140.050.060.10*0.070.140.360.140.460.190.140.160.090.160.320.250.390.26 060.350.190.180.220.07*0.150.360.180.420.320.210.300.220.230.540.230.350.37 Nikon070.360.190.220.230.140.15*0.210.190.210.240.160.120.180.170.470.120.190.34 080.680.460.470.500.36 0.21*0.530.160.470.39 0.480.410.890.310.100.92 Sony090.220.110.220.14 0.180.190.53*0.090.170.080.110.07 0.130.430.610.14 100.760.550.510.710.460.420.210.160.09*0.660.610.520.560.591.020.180.230.82 Olympus110.250.120.230.040.190.320.240.470.170.66*0.110.100.080.110.190.560.610.25 120.310.140.270.080.140.210.160.390.080.610.11*0.080.120.010.200.42 0.26 Minolta130.180.110.200.100.160.300.120.390.110.520.100.08*0.060.080.170.390.450.25 140.090.030.100.070.090.220.180.480.070.560.080.120.06*0.11 0.420.610.13 Casio150.310.150.290.100.160.230.170.410.070.590.110.010.080.11*0.180.440.450.24 Fujifilm160.180.170.330.190.320.540.470.890.131.020.190.200.170.110.18*0.821.050.17 170.540.350.310.490.250.230.120.310.430.180.560.420.390.420.440.82*0.230.51 Kodak180.830.57 0.580.390.350.190.100.610.230.610.420.450.610.451.050.23*0.98 Epson190.250.160.270.230.260.370.340.920.140.820.250.260.250.130.240.170.510.98* Divergence Scores between Camera Models Training set: 20 representative images, interpolated with 6 common methods (bilinear, bicubic, gradient-based, etc.). Total: 120 images. Test set: 19 different camera models, 200 images each. Total: 3800 images Some cameras from same brand have Low Divergence Scores Some cameras of different brands have Low Scores.  Possible clues for infringement or licensing CanonNikonSonyOlympusMinoltaCasio

47. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [47] Tampering Detection Explore intrinsic fingerprints left by various processing modules –To infer the algorithms and parameters employed in various components of the digital device and processing systems –New traces or vanished old traces suggests potential post-camera operations Estimated coeff. From direct camera output After post- camera filtering Color Interpolation Color Sensors Scene Optical Lens System CAMERA Other Software Processing A Tampering / Stego B Black: Sony P72; White: Canon Powershot S410 Grey: Classified as other cameras with low confidence

48. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [48] Overview of Scanner Model Tri-linear CCD sensors Hardcopy graphic data Lamp & Mirrors Lens Tri-linear color filter array Scanner head Motion system Shift Register Amplifier A/D Converter Shift Register Amplifier A/D Converter Post-processing Interpolation, Color transformation White balancing, Exposure control Noise reduction,… Digital image Software operation Scanning noise Statistical noise features Scanner model Identification

49. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [49] Scanner Identification using Noise Features Digital Photographs ScanningPrinting SourceData Scanned Digital Image Statistical noise features determine which scanner brand/model EPSONOneTouch

50. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [50] E.g.: Noise Features from Wavelet Analysis Scanned Image I One stage wavelet decomposition HH HL LH Subband STD & goodness of Gaussian fitting Statistical features f (3) (I), f (4) (I) Digital photograph Scanner model 1 Scanner model 2 Histogram  Mean and STD  Gaussian distribution  Goodness of Gaussian fitting Fitting error HH,HL,LH sub-bands RGB components 2x3x3 = 18 features

51. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [51] Noise Features from Neighborhood Prediction Smooth regions Scanned Image I Neighborhood prediction Absolute prediction error Mean & STD b Statistical features f (5) (I), f (6) (I) a i,1 a i,4 a i,6 a i,2 b i a i,7 a i,3 a i,5 a i,8 Non-Negative Least Squares Dark & bright smooth regions RGB components 2x2x3 = 12 features

52. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [52] Acquisition Forensics w/ Noise + Interp. Features Q1) What type of device was used to capture the image?  94% accuracy in identifying device type Q2) What brand/model of the device captured the image?  Cellphone cameras: 98% accuracy over 5 brands  Standalone cameras: 90% over 19 camera models from 9 camera brands  Scanners: 93% accuracy over 9 scanner brands Cell Phone Camera   Standalone Camera   Scanner Computer Generated Input Image Brand/Model Identification Acquisition Device Type Identification Further Forensic Analysis Sony Samsung Nokia Audiovox Motorola Canon FujiFilm Casio Minolta Epson Microtek AcerScan Canon Step 1Step 2

53. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [53] Suggested Readings 1.I. Cox, J. Kilian, T. Leighton, T. Shamoon: “Secure Spread Spectrum Watermarking for Multimedia'', IEEE Trans. on Image Proc., vol.6, no.12, pp.1673-1687, 1997. 2.M. M. Yeung, F. Mintzer: “An Invisible Watermarking Technique for Image Verification", Proc. of the IEEE Int’l Conf. on Image Processing (ICIP), Oct. 1997. 3.M. Wu and B. Liu: "Data Hiding in Image and Video: Part-I -- Fundamental Issues and Solutions", IEEE Trans. on Image Proc., vol.12, no.6, pp.685-695, June 2003. 4.M. Wu, W. Trappe, Z.J. Wang, and K.J.R. Liu: “Collusion-resistant fingerprinting for Multimedia,” IEEE Signal Proc Magazine, March 2004. 5.Special Issue on Digital Forensics, IEEE Signal Processing Magazine, March 2009. => Several articles on (non-intrusive) image and video forensics 6.M. Wu and B. Liu: Multimedia Data Hiding, Springer-Verlag, 2003. 7.I. Cox, M. Miller, and J. Bloom: Digital Watermarking, Morgan Kauffman, 2002. And the related references cited by these publications. UMCP ENEE631 Slides (created by M.Wu © 2004)

54. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [54] Data Hiding in Binary Image UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

55. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [55] Binary Image: A Simple yet Important Class –Scanned documents, drawings, signatures Social Security E-Files From Princeton EE201 lab material E-PAD (InterLink Electronics)

56. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [56] Copyright Protection for E-Publishing Change horizontal and vertical spacing to embed data –Eyes can not easily identify such changes –“Make it difficult and not worthwhile rather than impossible” u for cheap, high-volume content ~ newspaper, magazine, E-books u possible to remove watermark, but why not just pay a bulk –Embedding may be through additive or enforcement methods from http://www.acm.org/~hlb/publications/dig_wtr/dig_watr.html N.F. Maxemchuk, S. Low: “Marking Text Documents”, ICIP, 1997.

57. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [57] Authentic Signatures? Digitized signatures become popular in everyday life –At least a good interim solution to carry a long tradition to digital world Forgery and mis-use of signatures Clinton electronically signed Electronic Signatures Act - Yahoo News 6/30/00 http://www.whitehouse.gov/ media/gif/bil.gif as of 7/00 E-PAD (InterLink Electronics)

58. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [58] Challenges on Hiding Data in Binary Images Only two levels are available –Black-white flipping –Minor tuning on the color is not available Little room for “invisible changes” –What places can be changed and what cannot Uneven distribution of changeable pixels Related to authentication –Extract hidden data without the use of original copy

59. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [59] Identify Flippable Pixels Flippability score –Take the human perception into account u Based on smoothness and connectivity –0~1, with 0 indicating the pixels that should not be flipped flip-score0.6250.3750.250.1250.10.050.01 # of pixels2503238638266271 (a) (b)

60. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [60] Pixels with high flippability score are shown in the images. Unevenness in Data Hiding (Binary Image Example) Uneven distribution of flippable pixels –most are on rugged boundary Embedding rate (per block) –variable: often need side info. u worthwhile if such overhead is relatively small –constant: require larger block Random shuffling equalizes distribution –embed more bits –enhance security u a key to generate shuffle table –con: sensitive to jitter and mis-alignment 05101520253035404550 0 0.05 0.1 0.15 0.2 0.25 embeddble coeff. # per block (signature img) portion of blocks before shuffle after shuffle Important ! image size 288x48, red block size 16x16 UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

61. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [61] Uneven Distribution of Flippable Pixels Most on rugged boundary Multi-bit embedding via spatial division –Partition the image into non-overlapping blocks Embedding rate (per block) –variable: need side info. –constant: require larger blocks Two advanced mechanisms to equalize the distribution –Random shuffling –Recent generalized approach: Wet paper codes UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04) image size: 288x48 red block size: 16x16

62. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [62] Shuffling and Block-based Embedding Shuffling to equalize distribution of flippables (54 blocks) Divide the image into blocks and hide one bit in each block –Manipulating pixels with the highest flippability scores in the block –Odd-even embedding u To embed a “0”: even number of black pixels u To embed a “1”: odd number of black pixels

63. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [63] Shuffling-based Embedding and Extraction Data embedding Data Extraction Marked binary image Shuffling Block- based embedding Inverse shuffling Original binary image Data to be embedded Key Key Computeflippability Shuffling Shuffling Block- based extraction Test binary image Key Key Extracted data Enhance security Simple

64. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [64] before shuffle std after shuffle mean after shuffle Compare Analysis with Simulation for Shuffling Simulation: 1000 indep. random shuff. q = 16 x 16 S = 288 x 48 N = S/q = 18 x 3 p = 5.45% before shuffle mean after shufflestd after shuffle analysissimulationanalysissimulation m 0 /N (0 th bin) 20.37%5.16x10 -5 %0 %9.78x10 -5 0 m 1 /N (1 st bin) 1.85%7.77x10 -4 %0 %3.79x10 -4 0 m 2 /N (2 nd bin) 5.56%5.81x10 -3 %5.56x10 -3 %0.0010

65. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [65] Application: “Signature in Signature” –Annotating digitized signature with content info. of the signed document Each block is 320- pixel large, 1bit / blk.

66. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [66] Application: Annotating Binary Line Drawings u 10 characters (~ 70bits) are embedded original marked w/ “01/01/2000” pixel-wise difference

67. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [67] Fragile Watermark for Tamper Detection of Document u Embed pre-determined pattern or content features beforehand u Verify hidden data’s integrity to decide on authenticity (f) alter (a) (b) (g) after alteration (e) (c) (d) UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04)

68. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [68]

69. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [69] Quick Review on Signal Detection Hypothesis testing –Error-free case: for observation values where only one hypothesis having non-zero probability (or prob. density)

70. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [70] Quick Review on Signal Detection (cont’d) Minimize probability of detection error –Equal probable prior: maximum likelihood (ML) detection decision rule:arg j max P j (y) –General prior probability: scale by prior probability decision rule: arg j max P(H j ) P j (y) = arg j max P(H j | y) u Maximum A Posteriori (MAP) detection u Likelihood ratio test with proper threshold

71. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [71] Review on Detection of Signal Sequence in Noise Formulation u Known signals; noise values unknown but statistics known Under equal prior probability and AWGN noise –Equivalent to minimum distance detection –For equal-power signals, equiv. to maximum correlation detection (1) Antipodal signaling: s 1 =  s 0 = s (2) On-Off signaling: s 0 = 0; s 1 = s u detection performance proportional to the square root of ratio between signal energy and noise power

72. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [72] Conveying One-bit Info. Through Noisy Channel [ Known signal s; noise values unknown but statistics known ] Optimal detection ~ minimize prob. of error MAP ~ maximize posterior probability …P(H ? |y) => ML ~ maximum likelihood detector [for equal prior] … P(y|H ? ) => Minimum distance detector [for i.i.d. Gaussian noise] … ||y – bs|| => Maximum correlation detector [for equal-energy sig.] Detection statistic (under i.i.d. Gaussian noise) [correlator]  i y i s i  normalize variance prob. distribution under each hypothesis ~ N(  ||s|| 2, ||s|| 2  d 2 ) T b=1 b=  1

73. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [73] Illustration: Anti-podal Signaling & Detection Project onto the deterministic signal and check correlation –Positive high correlation => (s1, s2) being sent with high confidence; –Negative high correlation => (-s1, -s2) being sent Use long signal to boost SNR –Each component may be subject to low power constraints, but overall signal energy can be large by increasing sig. length => two sig. point (red dots) can be widely separately, thus reliably detected We can analyze on-off keying similarly.

74. ENEE631 Digital Image Processing (Spring'09) Lec 22 – Image Forensics [74] Hypothesis Testing for Ownership Watermark Optimal detection for On-Off Keying (OOK) –under i.i.d. Gaussian noise {d i } u use a correlator-type detector u b  {0,1} represents absence vs. presence of ownership mark –Need to determine how to choose watermark signal {s i } Neyman-Pearson Detection [Poor’s book Sec.2.4] –False-alarm: claiming wmk existence when nothing embedded –Given max. allowed false-alarm, maximize detection probability u Use likelihood ratio as detection statistic u Determine threshold based on false-alarm prob. => Q(3)=10 -3 ; Q(6)=10 -5 –Pd vs. Pfp tradeoff ~ ROC curve UMCP ENEE631 Slides (created by M.Wu © based on Research Talks ’98-’04) T presence absence