1. Saliency detection
Donghun Yeo
CV Lab.

2. Contents Definition of Saliency Detection Conventional Approaches
Saliency Detection via Absorbing Markov Chain

3. Definition of Saliency Detection
Identify image regions which are important to human vision.
Applications
Original Image GT
[Yan et al.]
Prime object proposals
[Manen et. al]
Image categorization
[Sharma et al.]
Context-aware image resizing
[Achanta and Susstrunk]

4. Conventional approaches
Underlying idea
High feature contrast between salient regions and non-salient regions
Features
Color contrast
Texture contrast
Conventional approaches
Background Modeling (Non-salient region modeling)
Global Contrast

5. Conventional approaches (1) Non-salient region modeling
Assume that
image boundary region is non-salient region.
salient regions have high contrasts from non-salient regions.
Weakness
Salient objects can be exist at image boundaries.
[Jiang et al. CVPR13]
Non-salient region
Original image
Original image
Non-salient region
[Jiang et al. CVPR13]
Original image
[Jiang et al. CVPR13]
Original image
[Jiang et al. CVPR13] 5

6. Conventional approaches (2) Global contrast
Assume that
most parts of an image are non-salient.
Saliency of a selected superpixel is contrast from non-selected regions.
Weakness
Find smaller consistently colored region.
[Cheng et al. CVPR11]
Original image
[Cheng et al. CVPR11]
Original image
[Cheng et al. CVPR11]
Original image
[Cheng et al. CVPR11]
Original image
[Cheng et al. CVPR11]
Original image
[Cheng et al. CVPR11]
Original image 6

7. non-salient superpixels
Our Proposal
Our goal
Identify non-salient regions!
Our assumption
Most parts of image boundary are non-salient!
Measure rareness and saliency via absorbing Markov chain
Our method
non-salient superpixels
Result
[Jiang et al. CVPR13]
Salient region at image boundary is rare!

8. Principles of Markov Chain
Markov chain : A chain moves from a state to a state.
A set of states : 𝑆={ 𝑠 1 , 𝑠 2 , …, 𝑠 𝑚 }
Transition matrix : P 𝑚×𝑚
𝑝 𝑖𝑗 : the probability of moving from state 𝑠 𝑖 to state 𝑠 𝑗
𝑠 1
𝑠 2
0.6
0.7
0.4
0.3
𝑆={ 𝑠 1 , 𝑠 2 }
𝑃= 𝟎.𝟔 𝟎.𝟒 𝟎.𝟕 𝟎.𝟑

9. Absorbing Markov Chain
Absorbing state
A state is absorbing when which has no outgoing edge.
Absorbing Markov chain
A Markov chain is absorbing if it has at least one absorbing state.
Every Markov chain starting from any state falls into absorbing state.
𝑠 1
𝑠 2
0.6
0.5
0.4
0.3
𝑠 3
0.2
1.0

10. Principles of Markov Chain
Absorbing Markov chain
Absorbing state 𝑠 𝑖 : 𝑝 𝑖𝑖 =1
Transient state : not absorbing
A Markov chain with 𝑟 absorbing states and 𝑡 transient states
By renumbering the states so that the transient states come first,
Expected number of times that a chain spends in the transient states 𝑗 given that the chain starts in the transient state 𝑖 : 𝒏 𝒊𝒋
Expected number of times that a chain starting in transient state 𝑖 spends before absorption :
𝑄∈ 0,1 𝑡×𝑡
𝑅∈ 0,1 𝑡×𝑟
0 :𝑟×𝑡 zero matrix
𝐼 :r×𝑟 identity matrix
𝑃 → 𝑄 𝑅 0 𝐼
𝑁= 𝐼 𝑡×𝑡 −𝑄 −1
y 𝒊 = 𝒋 𝒏 𝒊𝒋

11. Graph Construction
Saliency detection via absorbing Markov chain [Jiang et al. CVPR13]
Transient states - All superpixels of images
Absorbing states – All duplicated boundary superpixels
Transition matrix – Proportional to mean color similarity of superpixels between neighborhoods

12. Graph Construction
Saliency detection via absorbing Markov chain [Jiang et al. CVPR13]
Transient states - All superpixels of images
Absorbing states – All duplicated boundary superpixels
Transition matrix – Proportional to mean color similarity of superpixels between neighborhoods

13. Graph Construction
Saliency detection via absorbing Markov chain [Jiang et al. CVPR13]
Transient states - All superpixels of images
Absorbing states – All duplicated boundary superpixels
Transition matrix – Proportional to mean color similarity of superpixels between neighborhoods

14. Saliency via absorbing markov chain [jiang et al. cvPR13]
Expected number of times that a chain starting in transient state 𝑖 spends before absorption :
y 𝒊 = 𝒋 𝒏 𝒊𝒋

15. non-salient superpixels
Weakness & complement
Salient objects can be exist at image boundaries.
We propose an algorithm to identify superpixels belonging to salient region
Original image
[Jiang et al. CVPR13]
Our method
non-salient superpixels
Result

16. Principles of Markov Chain
Absorbing Markov chain
Absorbing state 𝑠 𝑖 : 𝑝 𝑖𝑖 =1
Transient state : not absorbing
A Markov chain with 𝑟 absorbing states and 𝑡 transient states
By renumbering the states so that the transient states come first,
Expected number of times that a chain spends in the transient states 𝑗 given that the chain starts in the transient state 𝑖 : 𝒏 𝒊𝒋
Probability of that the chain starting in the transient state 𝑖 is absorbed into the absorbing state 𝒋 : {𝑏 𝑖𝑗 } 𝑗 - Absorbing probability distribution of transient state 𝑖
𝑄∈ 0,1 𝑡×𝑡
𝑅∈ 0,1 𝑡×𝑟
0 :𝑟×𝑡 zero matrix
𝐼 :r×𝑟 identity matrix
𝑃 → 𝑄 𝑅 0 𝐼
𝑁= 𝐼 𝑡×𝑡 −𝑄 −1
𝐵=𝑁𝑅

17. Absorbing Probability distribution
We connect all pairs of boundary superpixels directly in Markov chain graph.
Non-salient superpixels are similar to each other.
They fall into non-salient absorbing states.
Index of absorbing states
𝑏 𝑖𝑗
The absorbing probability distribution of a non-salient boundary superpixel.

18. Absorbing Probability distribution
We connect all pairs of boundary superpixels directly in Markov chain graph.
Salient superpixels are similar to each other.
They fall into salient absorbing state.
The number of salient absorbing states is smaller than the number of non-salient states.
Index of absorbing states
𝑏 𝑖𝑗
The absorbing probability distribution of a salient boundary superpixel.

19. Entropy of the probability distribution
Entropy of the absorbing probability distribution {𝑏 𝑖𝑗 } 𝑗 of transient state 𝑖
− 𝑗 𝑏 𝑖𝑗 ln 𝑏 𝑖𝑗
𝑗 : index of absorbing state
Superpixels which have entropies below a threshold are salient. …
Entropy
Absorbing probability distribution

20. Saliency via absorbing markov chain
Flow Chart of our algorithm
Expected absorbing time

21. Experiments – 2 Datasets
MSRA image segmentation dataset
Cut-MSRA dataset
Evaluation Metric
Precision-Recall curve
F-measure,
Mean Precision , Mean Recall, Mean F-measure

22. Experiments – MSRA Dataset
ours
[jiang et al. cvpr13]
[cheng et al. cvpr11]
Ground truth

23. Precision-Recall Curve
Results
Precision-Recall Curve

24. Results – Cut-MSRA DATASET Images
Cut images to make salient region be at image boundary
ours
[jiang et al. cvpr13]
[cheng et al. cvpr11]
ground truth

25. Precision-Recall Curve
Results
Precision-Recall Curve

26. Limits of bottom-up methods
ground truth
ours
[jiang et al. cvpr13]