Download presentation
Presentation is loading. Please wait.
Published bySarah Garrison Modified over 9 years ago
1
Spatio-temporal saliency model to predict eye movements in video free viewing Gipsa-lab, Grenoble Département Images et Signal CNRS, UMR 5216 S. Marat, T. Ho Phuoc, L. Granjon, N. Guyader, D. Pellerin, A. Guérin-Dugué GDR-vision 12/06/2008
2
Plan Introduction Model Experiment and results Conclusion 2/24
3
Salient region attracts attention and so the eyes Saliency depends mainly on two factors: Bottom-up : task-independent, depending on intrinsic features of the stimuli Top-down : task-dependant, integrating high-level processes (cognitive state,...) 3/24 Introduction
4
Spatio-temporal saliency model Achromatic stimuli Simulates some parts of the human visual system: retina, primary visual cortex (V1) Two pathways : static and dynamic 4/24 Model MsMs MdMd
5
5/24 Model MsMs MdMd Spatio-temporal saliency model Achromatic stimuli Simulates some parts of the human visual system: retina, primary visual cortex (V1) Two pathways : static and dynamic
6
6/24 Model MsMs MdMd Spatio-temporal saliency model Achromatic stimuli Simulates some parts of the human visual system: retina, primary visual cortex (V1) Two pathways : static and dynamic
7
7/24 Model Spatio-temporal saliency model Achromatic stimuli Simulates some parts of the human visual system: retina, primary visual cortex (V1) Two pathways : static and dynamic
8
Two outputs : Magnocellular-like: Low spatial frequencies, band pass filter, whitens spectrum, provides global information Parvocellular-like: High spatial frequencies, high pass filter, whitens spectrum, enhances frame contrast 8/24 Retina model Model _ retina model
9
9/24 Retina model Model _ retina model « Parvocellular-like » « Magnocellular-like » MsMs MdMd
10
Visual stimuli are processed in different frequency bands and orientation in V1 Static: 6 orientations, 4 frequency bands Dynamic: 6 orientations, 3 frequency bands (lower) 10/24 Cortical-like filters Model _ cortical-like filters MsMs MdMd
11
11/24 Static pathway Model _ static pathway Static pathway: Interactions: strengthens the contours Short: between cells of overlapping receptive field Long: between collinear cells Normalization Summation in all orientation and frequency bands: static saliency MsMs MdMd
12
12/24 Dynamic pathway Model _ dynamic pathway Dynamic pathway: 2 motion estimation steps: Dominant motion compensation Local motion estimation using the same bank of cortical filters as static pathway Temporal filtering Dynamic saliency: module of motion vector MsMs MdMd
13
13/24 Dynamic pathway Model _ dynamic pathway Dynamic pathway: 2 motion estimation steps: Dominant motion compensation Local motion estimation using the same bank of cortical filters as static pathway Temporal filtering Dynamic saliency: module of motion vector MsMs MdMd
14
Multiplicative fusion 14/24 Fusion and example of saliency maps Model _ fusion and example of saliency maps Original video MdMd M and MsMs
15
Purpose : compare model results with human eye positions Free viewing, eye positions recorded by Eyetracker Eyelink II 15 subjects 20 clips of 30s composed of different snippets strung together Stimulus size = 720x576 pixels, 40°x30° field of view 15/24 Experiment and results Experiment and result Snippet 1Snippet 2 Snippet k-1Snippet kSnippet k-2 [Itti] : R. Carmi and L. Itti, « Visual causes versus correlates of attentional selection in dynamic scenes », Vision Research, vol.46, 2008 MhMh
16
Criterion : Normalized Scanpath Saliency (NSS) [Itti] 16/24 Global analysis Experiment _ global analysis 0.540.440.54Real eye movements M dn M snSD M snH Naives saliency maps M s : static M d : dynamic M and : fusion M snH : entropy M snSD : standard-deviation M dn : absolute difference [Itti] : R. J. peters and L. Itti, « Applying computational tools to predict gaze direction in interactive visual environments », ACM Trans. On Applied Perception, vol.5, 2008 Saliency mapsMsMs MdMd M and Real eye movements0.680.870.96
17
NSS as a function of frame 17/24 Temporal analysis Experiment _ temporal analysis Snippet 1Snippet 2 Snippet N Average on the k th frame of each snippet 1…k … Frame rate = 25 fps
18
NSS as a function of frame 18/24 Temporal analysis Experiment _ temporal analysis Frame rate = 25 fps Snippet 1Snippet 2 Snippet N Average on the k th frame of each snippet 1…k …
19
NSS as a function of frame 19/24 Temporal analysis Experiment _ temporal analysis Frame rate = 25 fps Snippet 1Snippet 2 Snippet N Average on the k th frame of each snippet 1…k …
20
NSS as a function of frame 20/24 Temporal analysis Experiment _ temporal analysis Frame rate = 25 fps Snippet 1Snippet 2 Snippet N Average on the k th frame of each snippet 1…k …
21
NSS as a function of frame 21/24 Temporal analysis Experiment _ temporal analysis Dispersion of eye positions as a function of frame Frame rate = 25 fps
22
NSS as a function of frame 22/24 Temporal analysis Experiment _ temporal analysis Dispersion of eye positions as a function of frame Frame rate = 25 fps 10-13 th frame ≈ 400-520 ms
23
NSS as a function of frame 23/24 Temporal analysis Experiment _ temporal analysis Dispersion of eye positions as a function of frame Frame rate = 25 fps
24
New model of spatio-temporal saliency, biologically inspired Retina filter with two outputs Interactions Same bank of cortical-like filters for static and dynamic pathways This model is reliable to predict the first fixations references : S. Marat, T. Ho Phuoc, L. Granjon, N. Guyader, D. Pellerin, A. Guérin-Dugué, « Spatio-temporal saliency model to predict eye movements in video free viewing », Proc. Eusipco 2008 S. Marat, T. Ho Phuoc, L. Granjon, N. Guyader, D. Pellerin, A. Guérin-Dugué, « Modelling spatio-temporal saliency to predict gaze direction for short videos », submitted in International Journal of Computer vision 24/24 Conclusion
25
Thanks for your attention !
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.