1. Video quality assessment accounting for temporal visual masking of local flicker
Lark Kwon Choi, Alan Conrad Bovik
Signal Processing: Image Communication 2018

2. Outline
Motivation
Background
FS-MOVIE Model
Experiments
Conclusion

3. Motivation
Effect of temporal visual masking on the visibility of temporal distortions, e.g., local flicker can be strongly reduced by motion
computational models of temporal visual masking
motion silencing illusion
the temporal flicker masking model augments the MOVIE Index
Local flicker denotes the temporal fluctuation of spatially local luminance or chrominance in videos.
Local flicker occurs mainly due to coarse quantization, mismatching of inter-frame blocks, improper deinterlacing, and dynamic rate changes in adaptive rate control.
Local flicker distortions, which are not well explained by current VQA models, frequently appear near moving edges and textures in compressed videos as well as in interlaced videos, producing annoying visual artifacts such as line crawling, interline flicker, and edge flicker.
‘‘motion silencing’’ illusion, in the form of a powerful temporal visual masking phenomenon called change silencing, wherein the salient temporal changes of objects in luminance, color, size, and shape appear to cease in the presence of large, coherent object motions.
commonly occurring temporal distortions, such as local flicker, can be dramatically suppressed by the presence of motion

4. Background Visual Masking Motion silencing of flicker distortions
Motion perception
From MOVIE to FS-MOVIE

5. Flicker sensitive motion tuned VQA model (FS-MOVIE)

6. Gabor decomposition

7. Spatial FS-MOVIE index
7x7, N=49

8. Spatial FS-MOVIE index
7x7, N=49
larger values of \sigma_{Q_S} indicate a broader spread of both high
and low quality regions, yielding lower overall perceptual quality [24].

9. Motion tuned video integrity
Temporal MOVIE penalizes the shifted spectrum of the distorted video lying along a different orientation than the reference video, by computing a weighted sum of the Gabor filter outputs.
The weight assigned to each individual Gabor filter is determined by its distance from the spectral plane of the reference video. The motion- tuned error of a distorted video relative to the reference video serves to evaluate temporal video integrity [24].

10. Motion tuned video integrity
Temporal MOVIE penalizes the shifted spectrum of the distorted video lying along a different orientation than the reference video, by computing a weighted sum of the Gabor filter outputs.
The weight assigned to each individual Gabor filter is determined by its distance from the spectral plane of the reference video. The motion- tuned error of a distorted video relative to the reference video serves to evaluate temporal video integrity [24].

11. Motion tuned video integrity
Temporal MOVIE penalizes the shifted spectrum of the distorted video lying along a different orientation than the reference video, by computing a weighted sum of the Gabor filter outputs.
The weight assigned to each individual Gabor filter is determined by its distance from the spectral plane of the reference video. The motion- tuned error of a distorted video relative to the reference video serves to evaluate temporal video integrity [24].

12. Modeling V1 cortical neuron responses
The responses of Area V1 neurons were modeled using the spatiotemporal energy model [53] with divisive normalization [54]
R = 4 and σ= 0.2 in agreement with recorded physiological data [54]

13. Temporal flicker masking --- The nature of flicker
the weights are defined only in terms of the distance from the motion tuning plane without considering the speed of object motion (i.e., slope of the motion plane)
simulations by controlling the velocity of motion, and the levels of flicker distortions caused by video compression in the form of quantization flicker.
Database downloading …

14. Temporal flicker masking --- The nature of flicker
the weights are defined only in terms of the distance from the motion tuning plane without considering the speed of object motion (i.e., slope of the motion plane)
simulations by controlling the velocity of motion, and the levels of flicker distortions caused by video compression in the form of quantization flicker.
Database downloading …

15. the weights are defined only in terms of the distance from the motion tuning plane without considering the speed of object motion (i.e., slope of the motion plane)
simulations by controlling the velocity of motion, and the levels of flicker distortions caused by video compression in the form of quantization flicker.
Database downloading …

17. Temporal flicker masking --- Perceptual flicker visibility index
temporally masked perceptual flicker visibility index
flicker sensitive index
the way flicker changes the local spectral signatures of videos and how motion influences the resulting V1 neuron responses
Shifted or separated spectral signatures not present in a reference video might be associated with flicker distortions. Therefore, we devised an approach to capture temporally masked, perceptual flicker visibility by measuring locally shifted energy deviations relative to those on the reference video at each spatiotemporal frequency.

19. Temporal FS-MOVIE index
since the range of Temporal FS- MOVIE scores is smaller than that of the Spatial FS-MOVIE scores, due to the divisive normalization in (10) and (11).

20. FS-MOVIE index

21. Temporal pooling
temporal hysteresis (TH) model of temporal pooling

22. Experiments: bitrate & motion
Low bitrate, large motion

23. Experiments: temporal pooling

24. Experiments: parameter variation of temporal pooling

25. Experiments: computational complexity

26. Conclusion
A new VQA model called FS-MOVIE that accounts for temporal visual masking of local flicker in distorted videos by augmenting the MOVIE framework.
It is also important to note that motion on the retina, not in space, is responsible for the motion silencing phenomena, as shown in [40].