1. Detecting Digital Image Forgeries Using Sensor Pattern Noise presented by: Lior Paz Jan Lukas, jessica Fridrich and Miroslav Goljan

2. Abstract Use the camera’s sensor pattern noise in order to link pictures to the camera by which they were taken. Use the camera’s sensor pattern noise in order to link pictures to the camera by which they were taken. Use this noise pattern in order to identify areas in the image that were manipulated. Use this noise pattern in order to identify areas in the image that were manipulated. photomontage photomontage cropping cropping

3. Other possible solutions Watermark Watermark Needs a world wide standard Needs a world wide standard Needs cooperation from the image taker Needs cooperation from the image taker Effective only in a controlled environment. Effective only in a controlled environment. Embed information in image file Embed information in image file Needs cooperation from the image taker Needs cooperation from the image taker Easy to manipulate Easy to manipulate

4. Where other methods fail Shadow analyzing – poor lighting conditions Shadow analyzing – poor lighting conditions Copy Move Forge – photomontage from another image Copy Move Forge – photomontage from another image Statistical methods – Results are not accurate Statistical methods – Results are not accurate

5. Noise The pattern noise has 2 main components The pattern noise has 2 main components 1. Fixed Pattern Noise (FPN) pixel to pixel difference when not exposed to light pixel to pixel difference when not exposed to light Additive noise Additive noise 2. Photo-Response Non-Uniformity (PRNU) the dominant part of the pattern noise the dominant part of the pattern noise multiplicative noise multiplicative noise

6. 1. Fixed Pattern Noise The FPN is the pixel to pixel difference when the sensor is not exposed to light The FPN is the pixel to pixel difference when the sensor is not exposed to light In most digital cameras this difference is equalized by subtracting a dark frame (mask) from the picture. In most digital cameras this difference is equalized by subtracting a dark frame (mask) from the picture. - = 0 0 -2 0 3 -5

7. 2. Photo-Response Non-Uniformity The PRNU is a multiplicative noise that derives from: The PRNU is a multiplicative noise that derives from: optics optics dust particles on the lens dust particles on the lens PNU – Pixel Non Uniformity PNU – Pixel Non Uniformity

8. Pixel Non Uniformity The PNU is cause by the different sensibility of pixels to light. The PNU is cause by the different sensibility of pixels to light. not dependent on temperature not dependent on temperature not dependent on time not dependent on time Thus, if we capture this noise pattern, we can make a distinctive link between a camera and its photos Thus, if we capture this noise pattern, we can make a distinctive link between a camera and its photos

9. Notations x = x ij clean image without any noise x = x ij clean image without any noise n = n ij random shot noise n = n ij random shot noise  =  ij additive noise - read-out noise  =  ij additive noise - read-out noise c = c ij additive noise - dark current ( FPN ) c = c ij additive noise - dark current ( FPN ) y = y ij digitized output of the sensor y = y ij digitized output of the sensor f ij multiplicative noise ( PRNU ) f ij multiplicative noise ( PRNU )

10. The Sensor’s Output y ij = f ij ( x ij + n ij ) + c ij +  ij x = x ij clean image without any noise n = n ij random shot noise  =  ij additive noise - read-out noise c = c ij additive noise - dark current ( FPN ) y = y ij digitized output of the sensor f ij multiplicative noise ( PRNU )

11. The signal Post process p ij = T(y ij,N(y ij ), i, j) T = Transformations on the sensor’s outputsignal T = Transformations on the sensor’s output signal Demosaicking Demosaicking Color correction Color correction Gama correction Gama correction White balance White balance Etc. Etc. N = Neighborhood of pixel y ij (e.g. of size 5x5)

12. Flat Fielding An algorithm that suppresses the pattern noise An algorithm that suppresses the pattern noise Correct FPN with dark frame Correct FPN with dark frame Divide by a flat field frame obtained by averaging images of a uniformly lit scene. Divide by a flat field frame obtained by averaging images of a uniformly lit scene. The algorithm must work on y ij (the raw sensor data) and will not work on p ij The algorithm must work on y ij (the raw sensor data) and will not work on p ij Common for astronomical imaging Common for astronomical imaging Very difficult (thus not done) to achieve in consumer digital cameras Very difficult (thus not done) to achieve in consumer digital cameras

13. Other Sensors All imaging sensors are made of semiconductors in quite similar techniques All imaging sensors are made of semiconductors in quite similar techniques CCD, CMOS, JFET and other sensors have similar noise pattern properties CCD, CMOS, JFET and other sensors have similar noise pattern properties

14. Obtain Camera’s PNU p c = the PNU of camera C p c = the PNU of camera C p (k), (k = 1..n) set of images taken by C p (k), (k = 1..n) set of images taken by C F = denoising filter F = denoising filter n (k) = p (k) – F(p (k) ) the noise residue of image k n (k) = p (k) – F(p (k) ) the noise residue of image k p c = average( n (k) ), k = 1..n) p c = average( n (k) ), k = 1..n)

15. Image & Camera Link How to decide whether image Y was taken by camera C ? How to decide whether image Y was taken by camera C ? calculate the correlation between: calculate the correlation between: Image noise: Y – F  (Y) Image noise: Y – F  (Y) camera pattern P c camera pattern P c  c (Y)=corr(Y-F  (Y),P c ) =  c (Y)=corr(Y-F  (Y),P c ) = || || is the l2 norm: || || is the l2 norm:

16. Result with TIFF Images Correlations of noise from Olympus C765 images in TIFF format with 9 reference patterns

17. Result with JPEG Images Correlations under JPEG

18. Result with Resized Images & Gamma Correction Correlations of noise from Canon G2 JPEG images resized to 1600x1200 pixels with gamma correction

19. Conclusion This method of linking images to the camera from which they were taken overcomes: This method of linking images to the camera from which they were taken overcomes: Low resolution images Low resolution images JPEG compression JPEG compression Linear & non-linear actions : Linear & non-linear actions : Brightness/Contrast Brightness/Contrast Histogram operations Histogram operations Gamma correction Gamma correction

20. Forgery Detection How can we use the sensor pattern noise in order to detect regions that have been tampered? How can we use the sensor pattern noise in order to detect regions that have been tampered??

21. Forgery Identification Obtain camera’s PNU (fingerprint) Obtain camera’s PNU (fingerprint) Obtain photo’s PNU Obtain photo’s PNU Find regions lacking the pattern noise Find regions lacking the pattern noise

22. Region Of Interest Let R be our Region Of Interest (ROI), a suspected region in an image. Let R be our Region Of Interest (ROI), a suspected region in an image.

23. Forgery Detection In order to check the ROI authenticity we will correlate between the ROI’s noise pattern from the image, and the same region in the reference pattern. In order to check the ROI authenticity we will correlate between the ROI’s noise pattern from the image, and the same region in the reference pattern.

24. Forgery Detection – cont. Now we will calculate the correlations of other regions of the same shape and size from the same images and others with P c (R). Now we will calculate the correlations of other regions of the same shape and size from the same images and others with P c (R).  (n(Q k ), P c (R)) for regions Q k, k= 1...N R  (n(Q k ), P c (R)) for regions Q k, k= 1...N R Q k is a sample of regions that don’t contain the reference pattern P c (R). Q k is a sample of regions that don’t contain the reference pattern P c (R). R Q2Q2 Q1Q1 Q3Q3 R

25. Forgery Detection – cont. Model the distribution of the correlations with the generalized Gaussian distribution with the cumulative distribution function G(x) Model the distribution of the correlations with the generalized Gaussian distribution with the cumulative distribution function G(x) #Q k  (n(Qk), Pc(R)) Q2Q2 Q1Q1 Q3Q3

26. Forgery Detection – cont. p=1-G(  (n(R), P c (R)) p=1-G(  (n(R), P c (R)) R was tampered if p>  =10 -3 and not otherwise R was tampered if p>  =10 -3 and not otherwise #Q k  (n(Qk), Pc(R))

27. Forgery Detection – cont. If the ROI was detected as tempered If the ROI was detected as tempered Apply the same algorithm on other Regions from the same image which are disjoint from R and have the same number of pixels and similar histogram. Apply the same algorithm on other Regions from the same image which are disjoint from R and have the same number of pixels and similar histogram. If all of those regions are decided as non-tempered, R was tempered. If all of those regions are decided as non-tempered, R was tempered. Q2Q2 Q1Q1 Q3Q3

28. Automatic ROI Identification A region that was tempered will have a lower correlation with it’s appropriate region in the Sensor Pattern A region that was tempered will have a lower correlation with it’s appropriate region in the Sensor Pattern Method: One way to find a suspicious region to be inspected is to use sliding square blocks and compute their correlation. Method: One way to find a suspicious region to be inspected is to use sliding square blocks and compute their correlation.

29. Automatic ROI Identification Problem: This will only work if the tempered area is significantly larger than the sliding block. Problem: This will only work if the tempered area is significantly larger than the sliding block. Solution: Sliding blocks of different sizes and shapes. Solution: Sliding blocks of different sizes and shapes.

30. Auto. Identification Algorithm Prepare N blocks of different types Prepare N blocks of different types For each block type correlate the image noise residual with the camera reference pattern in overlapping blocks.  (, ) For each block type correlate the image noise residual with the camera reference pattern in overlapping blocks.  (, ) For each block type select m blocks with the smallest correlation. For each block type select m blocks with the smallest correlation.

31. Auto. Identification Algorithm Construct the mask B from these blocks Construct the mask B from these blocks Choose the threshold t: Choose the threshold t: For each pixel p let t(p)=# of blocks covering p (step 3) For each pixel p let t(p)=# of blocks covering p (step 3) threshold t = median value of t(p) for all pixels p threshold t = median value of t(p) for all pixels p ROI R = {p|t(p)>t} ROI R = {p|t(p)>t} apply the ROI verification apply the ROI verificationalgorithm.

32. To sum up... Uses of the Sensor Pattern: Uses of the Sensor Pattern: Link between an image and the camera that took it. Link between an image and the camera that took it. Distinguish if a region in an image was distorted or manipulated. Distinguish if a region in an image was distorted or manipulated. Distinguish automatically if an image Distinguish automatically if an image is authentic or not.