Detecting False Captioning Using Common Sense Reasoning James Byrd

Picture editing is popular Medical, Journalism, Science all at risk how do we detect what is real from what is not? We shall introduce a nearly flawless mathematical technique to tell the true from the false Abstract

Pictures are a compelling way to communicate information Why modify? o To make the picture tell a different story o “Yellow Journalism” Intro

4 ways a. Deletion of Details - deleting details b. Insertion of Details - inserting additional details c. Photomontage - combining multiple images d. False Captioning - misrepresenting image content Photo Manipulation

Photomontage False Captioning This Paper’s Focus

1.Segmentation 2.Classification 3.Common Sense Reasoning Steps for detection

1.Segment the image into regions 2.Make an “importance map” to compare across images in a given corpus Segmentation

Perhaps the most important part Segment based color scheme compare segments among images Classification

2 approaches o resolve local classification ambiguities within images; we will query a knowledge base to resolve proper relations o reason across a larger corpa of images to find unique or missing elements during an investigation Common Sense Reasoning

Image segmentation Importance Map Calculating ROIs Classification Reasoning Methods

Segment the image o mean-shift image segmentation to decompose an image into homogeneous regions o choosing parameter values is often difficult o therefore we adds tons of colors and do many segmetns Image Segmentation

to truly understand the importance of an image requires a thorough understanding of what the image contains and what the viewer needs recognize objects (faces) and regions are important Importance Map

2 step process o Identify Candidate ROI  minimal image that identifies key important parts of the image o Grow the ROI  combine the ROI by using a clustering algorithm recursively Calculating ROI

Supervised Classification o Training Sites - predetermined objects of interest in an image o Software uses these to create a “signature analysis” Unsupervised Classification o Takes a large number of ROI and divides them into classes based on grouping in them o becoming increasingly popular Classification

Resolve local ambiguities within an image Answer questions the program has o i.e. water versus sky Then the software determines if the anomalies are consistent across images or show signs of tampering o More pictures to examine are better for consistency Reasoning

methods are limited by the performance of the components of image segmentation and important object identification Conclusion