1. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 1 “OBJECTIVE AND SUBJECTIVE IDENTIFICATION OF INTERESTING AREAS IN VIDEO SEQUENCES” Danko Tomašić DIPLOMA PROJECT Erasmus Student, University of Trieste, Italy Responsible Assistant: Elisa Drelie Gelasca Professor: Touradj Ebrahimi

2. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 2 Outline  Introduction – Context – Application – State of the Art – Problem Statement  Subjective Experiments for Segmentation Quality Evaluation - Generation of Synthetic Segmentation Errors - Experimental Method - Data Analysis - Work in progress  Conclusions and Future Work

3. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 3 Motivation  REAL WORLD SCENES - presents a large quantity of information to the Human... - high visual acuity – fovea - eye movements – not random ! VISUAL ATTENTION Automatic method for.. REGIONS OF INTEREST (ROI) In video sequences SELECTIVITY

4. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 4 APPLICATIONS  COMPRESSION - preserve the quality in ROIs  SEGMENTATION QUALITY EVALUATION - overall and individual object  IMAGE AND VIDEO QUALITY ANALYSIS –Perceptual PSNR..  IMAGE AND VIDEO DATABASES - extract ROIs in each scene ‘a priori’  DIGITAL WATERMARKING - perceptual marking  MACHINE VISION - real-time robot navigation

5. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 5 FACTORS WHICH INFLUENCE VISUAL ATTENTION  low-level or bottom-up - motion - position - size - color and brightness - contrast - shape - orientation  high-level or top-down - presence of people - background / foreground rapid saliency driven task independent slower volition controlled Relevance.. task dependent

6. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 6 SALIENCE AND RELEVANCE  Salience - bottom-up task-independent factors - connected with observal-external objects or properties - unexpectedness or unusualness of an object  Relevance - top-down volition-controlled and task-dependent factors - observal-internal factors (goals and motivations) - connected to the specific situation = Conspicuity

7. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 7 PREVIOUS ATTENTION MODELS  Correia and Perreira - estimation of video object’s relevance - both individually and in a given context  Osberger - perceptual vision model for image quality assessment and compression applications - early vision model and higher level attention processes - Importance Maps

8. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 8 PREVIOUS MODELS  Osberger and Rohaly - automatic detection of ROIs in video sequences - improvement of previous model - adding temporal and movement features  Pardo - extraction of semantic objects from still images - perceptual metric and Importance Maps - improvement of Osberger’s models

9. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 9 COLOR – STATE OF THE ART  Osberger - red attracts attention more than other colors - red induces higher amount of masking  Correia and Perreira - bright and colored objects are more noticed - red seems to be preferred PROBLEM – NO OTHER QUANTIFIED SOLUTIONS !!!

10. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 10 COLOR – SIGNALLING AND MARKETING  American Standards Association and French Standard - red - dinamic, warm - fireproof protection - red - salient ‘par excellence’ - orange and yellow - salient colors – danger - green - calming color - materials of first aid - blue - cold and calming - caution PROBLEM – NO OTHER QUANTIFIED SOLUTIONS !!! “psychologically active”

11. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 11 Subjective Experiment on Color..  COLOR Color Test II Color Test III Color Test I

12. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 12 COLOR – COLOR TEST I First cycle Second cycle Tested colors in CIELab color space

13. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 13 COLOR TEST I - RESULTS

14. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 14 COLOR – COLOR TEST II Colored objects with semantical meaning Annoyance and Salience of a color interconnected ? Adding gaussian noise (sigma =...)to channel L in HSL color space Matlab - RANDN (M,N) - normal distribution mean = 0 variance = 0.005 Rating the annoyance level

15. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 15 COLOR – COLOR TEST II Tested colors in CIELab color space Test image

16. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 16 COLOR TEST II - RESULTS

17. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 17 COLOR TEST II - RESULTS Cyan Green Pink Magenta Red Light blue Blue Orange Yellow = 241

18. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 18 COLOR – COLOR TEST III Colored objects with semantical meaning Annoyance and Salience of a color interconnected ? Blurred images - 3 levels of blurriness Rating the annoyance level

19. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 19 COLOR – COLOR TEST III Original imageTest image

20. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 20 COLOR TEST III - RESULTS COLOR TEST I vs. COLOR TEST III Second cycle highest level of blurriness 5 CASES Most important Most annoying 3 CASES 1st or 2nd most 2nd or 1st most important annoying 4 CASES RED 3rd most annoying !!! 4 CASES CYAN 1st most annoying !!!

21. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 21 COLOR TEST III - RESULTS COLOR TEST I vs. COLOR TEST III  Light blue  Violet  Dark Green  Maroon  Red  Yellow  Green  Blue Similar importance Similar annoyance

22. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 22 COLOR TEST III - RESULTS Average level lines 2 nd level of blurriness 3 rd level of blurriness

23. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 23 COLOR TEST III - RESULTS HSL color spaceCIELab color space

24. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 24 OBJECTIVE IDENTIFICATION OF INTERESTING AREAS Proposed method

25. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 25 SEGMENTATION  WATERSHED SEGMENTATION WATERSHED SEGMENTATION RESULT OF MERGE Sequence #1 - Akiyo Frame #15 ORIGINAL FRAME #15

26. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 26 FACTORS WHICH INFLUENCE ATTENTION  COLOR

27. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 27 FACTORS WHICH INFLUENCE ATTENTION  CONTRAST L, a, b – CIELab coordinates B i,j – border pixels in R i,j with R j k border = 10 R i,j – considered region R j – neighboring region

28. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 28 FACTORS WHICH INFLUENCE ATTENTION  SIZE

29. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 29 FACTORS WHICH INFLUENCE ATTENTION  POSITION

30. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 30 FACTORS WHICH INFLUENCE ATTENTION  SKIN HSV color space

31. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 31 FACTORS WHICH INFLUENCE ATTENTION  MOTION - OPTICAL FLOW Original video Movement mask

32. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 32 FACTORS WHICH INFLUENCE ATTENTION  FINAL COMBINATION OF FEATURES

33. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 33 SUBJECTIVE EXPERIMENTS  SEGMENTATION QUALITY DEFECT IN THE MOST IMPORTANT REGION DEFECT IN THE LEAST IMPORTANT REGION Sequence #1 - Akiyo Frame #15

34. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 34 SUBJECTIVE EXPERIMENTS  SEGMENTATION QUALITY DEFECT IN THE MOST IMPORTANT REGION DEFECT IN THE LEAST IMPORTANT REGION Sequence #2 - Highway Frame #28

35. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 35 SUBJECTIVE EXPERIMENTS  SEGMENTATION QUALITY DEFECT IN THE MOST IMPORTANT REGION DEFECT IN THE LEAST IMPORTANT REGION Sequence #3 - VideoEPFL Frame #39

36. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 36 SUBJECTIVE EXPERIMENTS  RESULTS Sequence #1 - Akiyo

37. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 37 SUBJECTIVE EXPERIMENTS  RESULTS Sequence #2 - Highway

38. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 38 SUBJECTIVE EXPERIMENTS  RESULTS Sequence #3 - VideoEPFL

39. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 39 VISUAL PERCEPTION - extracting information from the light  Acquiring knowledge - HVS = video camera  Objects and events in the environment  Light emitted or reflected by objects

40. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 40 VISUAL ATTENTION - processing different information within the visual field  Overt eye movements - determine available optic information  Covert selective attention - determines what gets full processing CAPACITY SELECTIVITY amount of available what gets processed perceptual resources and what does not

41. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 41 COLOR TEST I - RESULTS

42. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 42 COLOR TEST II - RESULTS Cyan Green Pink Magenta Red Light blue Blue Orange Yellow >>10

43. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 43 FACTORS WHICH INFLUENCE ATTENTION  COLOR Color Experiment 2 Color Experiment 3 Color Experiment 1