Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 1 “OBJECTIVE AND SUBJECTIVE IDENTIFICATION OF INTERESTING AREAS IN VIDEO SEQUENCES” Danko Tomašić DIPLOMA PROJECT Erasmus Student, University of Trieste, Italy Responsible Assistant: Elisa Drelie Gelasca Professor: Touradj Ebrahimi

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 2 Outline  Introduction – Context – Application – State of the Art – Problem Statement  Subjective Experiments for Segmentation Quality Evaluation - Generation of Synthetic Segmentation Errors - Experimental Method - Data Analysis - Work in progress  Conclusions and Future Work

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 3 Motivation  REAL WORLD SCENES - presents a large quantity of information to the Human... - high visual acuity – fovea - eye movements – not random ! VISUAL ATTENTION Automatic method for.. REGIONS OF INTEREST (ROI) In video sequences SELECTIVITY

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 4 APPLICATIONS  COMPRESSION - preserve the quality in ROIs  SEGMENTATION QUALITY EVALUATION - overall and individual object  IMAGE AND VIDEO QUALITY ANALYSIS –Perceptual PSNR..  IMAGE AND VIDEO DATABASES - extract ROIs in each scene ‘a priori’  DIGITAL WATERMARKING - perceptual marking  MACHINE VISION - real-time robot navigation

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 5 FACTORS WHICH INFLUENCE VISUAL ATTENTION  low-level or bottom-up - motion - position - size - color and brightness - contrast - shape - orientation  high-level or top-down - presence of people - background / foreground rapid saliency driven task independent slower volition controlled Relevance.. task dependent

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 6 SALIENCE AND RELEVANCE  Salience - bottom-up task-independent factors - connected with observal-external objects or properties - unexpectedness or unusualness of an object  Relevance - top-down volition-controlled and task-dependent factors - observal-internal factors (goals and motivations) - connected to the specific situation = Conspicuity

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 7 PREVIOUS ATTENTION MODELS  Correia and Perreira - estimation of video object’s relevance - both individually and in a given context  Osberger - perceptual vision model for image quality assessment and compression applications - early vision model and higher level attention processes - Importance Maps

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 8 PREVIOUS MODELS  Osberger and Rohaly - automatic detection of ROIs in video sequences - improvement of previous model - adding temporal and movement features  Pardo - extraction of semantic objects from still images - perceptual metric and Importance Maps - improvement of Osberger’s models

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 9 COLOR – STATE OF THE ART  Osberger - red attracts attention more than other colors - red induces higher amount of masking  Correia and Perreira - bright and colored objects are more noticed - red seems to be preferred PROBLEM – NO OTHER QUANTIFIED SOLUTIONS !!!

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 10 COLOR – SIGNALLING AND MARKETING  American Standards Association and French Standard - red - dinamic, warm - fireproof protection - red - salient ‘par excellence’ - orange and yellow - salient colors – danger - green - calming color - materials of first aid - blue - cold and calming - caution PROBLEM – NO OTHER QUANTIFIED SOLUTIONS !!! “psychologically active”

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 11 Subjective Experiment on Color..  COLOR Color Test II Color Test III Color Test I

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 12 COLOR – COLOR TEST I First cycle Second cycle Tested colors in CIELab color space

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 13 COLOR TEST I - RESULTS

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 14 COLOR – COLOR TEST II Colored objects with semantical meaning Annoyance and Salience of a color interconnected ? Adding gaussian noise (sigma =...)to channel L in HSL color space Matlab - RANDN (M,N) - normal distribution mean = 0 variance = Rating the annoyance level

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 15 COLOR – COLOR TEST II Tested colors in CIELab color space Test image

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 16 COLOR TEST II - RESULTS

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 17 COLOR TEST II - RESULTS Cyan Green Pink Magenta Red Light blue Blue Orange Yellow = 241

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 18 COLOR – COLOR TEST III Colored objects with semantical meaning Annoyance and Salience of a color interconnected ? Blurred images - 3 levels of blurriness Rating the annoyance level

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 19 COLOR – COLOR TEST III Original imageTest image

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 20 COLOR TEST III - RESULTS COLOR TEST I vs. COLOR TEST III Second cycle highest level of blurriness 5 CASES Most important Most annoying 3 CASES 1st or 2nd most 2nd or 1st most important annoying 4 CASES RED 3rd most annoying !!! 4 CASES CYAN 1st most annoying !!!

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 21 COLOR TEST III - RESULTS COLOR TEST I vs. COLOR TEST III  Light blue  Violet  Dark Green  Maroon  Red  Yellow  Green  Blue Similar importance Similar annoyance

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 22 COLOR TEST III - RESULTS Average level lines 2 nd level of blurriness 3 rd level of blurriness

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 23 COLOR TEST III - RESULTS HSL color spaceCIELab color space

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 24 OBJECTIVE IDENTIFICATION OF INTERESTING AREAS Proposed method

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 25 SEGMENTATION  WATERSHED SEGMENTATION WATERSHED SEGMENTATION RESULT OF MERGE Sequence #1 - Akiyo Frame #15 ORIGINAL FRAME #15

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 26 FACTORS WHICH INFLUENCE ATTENTION  COLOR

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 27 FACTORS WHICH INFLUENCE ATTENTION  CONTRAST L, a, b – CIELab coordinates B i,j – border pixels in R i,j with R j k border = 10 R i,j – considered region R j – neighboring region

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 28 FACTORS WHICH INFLUENCE ATTENTION  SIZE

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 29 FACTORS WHICH INFLUENCE ATTENTION  POSITION

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 30 FACTORS WHICH INFLUENCE ATTENTION  SKIN HSV color space

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 31 FACTORS WHICH INFLUENCE ATTENTION  MOTION - OPTICAL FLOW Original video Movement mask

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 32 FACTORS WHICH INFLUENCE ATTENTION  FINAL COMBINATION OF FEATURES

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 33 SUBJECTIVE EXPERIMENTS  SEGMENTATION QUALITY DEFECT IN THE MOST IMPORTANT REGION DEFECT IN THE LEAST IMPORTANT REGION Sequence #1 - Akiyo Frame #15

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 34 SUBJECTIVE EXPERIMENTS  SEGMENTATION QUALITY DEFECT IN THE MOST IMPORTANT REGION DEFECT IN THE LEAST IMPORTANT REGION Sequence #2 - Highway Frame #28

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 35 SUBJECTIVE EXPERIMENTS  SEGMENTATION QUALITY DEFECT IN THE MOST IMPORTANT REGION DEFECT IN THE LEAST IMPORTANT REGION Sequence #3 - VideoEPFL Frame #39

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 36 SUBJECTIVE EXPERIMENTS  RESULTS Sequence #1 - Akiyo

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 37 SUBJECTIVE EXPERIMENTS  RESULTS Sequence #2 - Highway

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 38 SUBJECTIVE EXPERIMENTS  RESULTS Sequence #3 - VideoEPFL

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 39 VISUAL PERCEPTION - extracting information from the light  Acquiring knowledge - HVS = video camera  Objects and events in the environment  Light emitted or reflected by objects

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 40 VISUAL ATTENTION - processing different information within the visual field  Overt eye movements - determine available optic information  Covert selective attention - determines what gets full processing CAPACITY SELECTIVITY amount of available what gets processed perceptual resources and what does not

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 41 COLOR TEST I - RESULTS

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 42 COLOR TEST II - RESULTS Cyan Green Pink Magenta Red Light blue Blue Orange Yellow >>10

Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 43 FACTORS WHICH INFLUENCE ATTENTION  COLOR Color Experiment 2 Color Experiment 3 Color Experiment 1