IEEE TCSVT 2011 Wonjun Kim Chanho Jung Changick Kim Spatiotemporal Saliency Detection and Its Applications in Static and Dynamic Scenes IEEE TCSVT 2011 Wonjun Kim Chanho Jung Changick Kim

Outline Introduction Proposed Method Experiment Result Application Conclusion

Introduction Problem occurs when background is highly textured

Feature Representation Proposed Method Feature Representation Edge orientation histogram (EOH) Color orientation histogram (COH) Temporal Feature Self-ordinal Measure Saliency Map Scale-invariant Saliency Map

Edge Orientation Histogram (EOH) Compute the edge orientation of every pixel in the local region center at the 𝑖 𝑡ℎ pixel Quantized into K angle in the range of [ 0 ° , 180 ° ] Compute the histogram of edge orientation local region 𝑖 m(x,y,n):edge magnitude 𝜽(x,y,n):quantized orientation

Color Orientation Histogram (COH) Quantize the angle in HSV color space in the range of [ 0 ° , 360 ° ] into H angles Compute the histogram of color orientation Add an additional bin with q=0 for both EOH and COH to handle pixels with zero edge magnitudes and color attributes s(x,y,n):saturation value 𝒗(x,y,n):quantized hue value

Temporal Feature Compute the intensity differences between frames Feature at the 𝑖 𝑡ℎ pixel of 𝑛 𝑡ℎ frame P :total number of pixels in local region j :index of those pixels in P 𝝉 :user-defined latency

Self-ordinal Measure Define a 1×(K+1) rank matrix by ordering the elements of EOH(COH) ex:

Self-ordinal Measure

Saliency Map of Edge and Color Compute the distance from the rank matrix of center region to surrounding regions Saliency Map of Edge Saliency Map of Color N :total number of local regions in a center-surround window 𝒁 𝑬 , 𝒁 𝑪 :maximum distance between two rank matrices

Spatial Saliency Map Combine the edge and color saliency

Combining with Temporal Saliency Compute the SAD of temporal gradients between center and the surrounding regions Combine the spatial and temporal saliency

Scale-invariant Saliency Map Combine 3 different scales of saliency Map (32×32, 64×64, 128×128) 32×32 64×64 128×128

Algorithm

Experiment Result Static Images Video Sequences

Experiment Result Static Image Video Sequence Local region = 5×5 center-surround window = 7×7 K = 8, H= 6 𝑍 𝐸 = 40, 𝑍 𝐶 = 24 Video Sequence 𝑍 𝑇 = 49 Speed: 23ms per frame (43 fps)

Static Images

Static Images

Video Sequences

Video Sequences

Moving Object Extraction Application Image Retargeting Moving Object Extraction

Image Retargeting

Image Retargeting

Moving Object Detection G:the set of salient pixels in the ground truth image P:salient pixels in the binarized object map Card(A):the size of the set A When the scene is cluttered or the background is complex some old methods might lead to severe distortions

Moving Object Detection

Conclusion Ordinal signature can tolerate more local feature distribution than sample values. The proposed scheme performs in real-time and can be extended in both static and dynamic scenes.