Structure-measure: A New Way to Evaluate Foreground Maps 范登平 南开大学计控学院 媒体计算实验室 dengpingfan@mail.nankai.edu.cn I’m Deng-Ping Fan ICCV 2017 (Spotlight) 2017/10/11

Content Current Measures Limitation Motivation Our new S-measure

What is the foreground map? models Foreground map (FM)

What is the foreground map? models Foreground map (FM)

Goal Similarity? Foreground map (FM) Our goal is to evaluate the similarity between the foreground map and the ground truth.

Pixel-wise measures (AP, AUC,Fbw[1]) %The problem is that the current popular measures including AP and AUC are pixel-wise based. They ignore the structure similarity, %So they rank the two different foreground map in the same order. This is contradictory to our common sense. Existing methods rely on pixel-wise measure and ignore important structure similarity, improperly resulting in same scores for these two foreground maps. (a) GT (b) FM1 (c) FM2 [1] How to Evaluate Foreground Maps, IEEE CVPR, 2014, Margolin R, et. al.

Motivation Region structure consistency of object-parts; We propose to evaluate structure similarity in both region level and object level. Our measure prefers structure consistency of object parts and uniformly distributed objects contrast sharply to background.

Motivation Region structure consistency of object-parts; We propose to evaluate structure similarity in both region level and object level. Our measure prefers structure consistency of object parts and uniformly distributed objects contrast sharply to background.

Motivation Region Object structure consistency uniformly distributed; of object-parts; Object uniformly distributed; contrast sharply; We propose to evaluate structure similarity in both region level and object level. Our measure prefers structure consistency of object parts and uniformly distributed objects contrast sharply to background.

Motivation Region Object structure consistency uniformly distributed; of object-parts; Object uniformly distributed; contrast sharply; We propose to evaluate structure similarity in both region level and object level. Our measure prefers structure consistency of object parts and uniformly distributed objects contrast sharply to background.

Motivation Region Object structure consistency uniformly distributed; of object-parts; Object uniformly distributed; contrast sharply; We propose to evaluate structure similarity in both region level and object level. Our measure prefers structure consistency of object parts and uniformly distributed objects contrast sharply to background.

Region-Level We divide the image into parts and use famous ssim metric to evaluate the structure similarity for each region.

Region-Level We divide the image into parts and use famous ssim metric to evaluate the structure similarity for each region.

Region-Level We divide the image into parts and use famous ssim metric to evaluate the structure similarity for each region.

Region-Level We divide the image into parts and use famous ssim metric to evaluate the structure similarity for each region.

Region-Level We divide the image into parts and use famous ssim metric to evaluate the structure similarity for each region.

Region-Level 𝑆 𝑟𝑒𝑔𝑖𝑜𝑛 = 𝑗=1 4 𝑤 𝑗 ∗𝑠𝑠𝑖𝑚 𝑗 𝑆 𝑟𝑒𝑔𝑖𝑜𝑛 = 𝑗=1 4 𝑤 𝑗 ∗𝑠𝑠𝑖𝑚 𝑗 We divide the image into parts and use famous ssim metric to evaluate the structure similarity for each region. Image quality assessment: from error visibility to structural similarity , IEEE TIP 2004, Z Wang, AC Bovik et. al.

Object-Level: foreground FM For Object-Level, we evaluate the foreground and background similarity, respectively. Foreground parts of ground truth and corresponding predicted map are compared in a holistic way, considering both contrast and uniform term.

Object-Level: foreground GT foreground FM For Object-Level, we evaluate the foreground and background similarity, respectively. Foreground parts of ground truth and corresponding predicted map are compared in a holistic way, considering both contrast and uniform term.

Object-Level: foreground GT foreground FM 𝐷 𝐹𝐺 = 𝑥 𝐹𝐺 2 + 𝑦 𝐹𝐺 2 2 𝑥 𝐹𝐺 𝑦 𝐹𝐺 +𝝀∗ 𝝈 𝑥 𝐹𝐺 𝑥 𝐹𝐺 For Object-Level, we evaluate the foreground and background similarity, respectively. Foreground parts of ground truth and corresponding predicted map are compared in a holistic way, considering both contrast and uniform term. contrast uniform

Object-Level: foreground GT foreground FM 𝐷 𝐹𝐺 = 𝑥 𝐹𝐺 2 + 𝑦 𝐹𝐺 2 2 𝑥 𝐹𝐺 𝑦 𝐹𝐺 +𝝀∗ 𝝈 𝑥 𝐹𝐺 𝑥 𝐹𝐺 For Object-Level, we evaluate the foreground and background similarity, respectively. Foreground parts of ground truth and corresponding predicted map are compared in a holistic way, considering both contrast and uniform term. contrast

Object-Level: foreground GT foreground FM 𝐷 𝐹𝐺 = 𝑥 𝐹𝐺 2 + 𝑦 𝐹𝐺 2 2 𝑥 𝐹𝐺 𝑦 𝐹𝐺 +𝝀∗ 𝝈 𝑥 𝐹𝐺 𝑥 𝐹𝐺 For Object-Level, we evaluate the foreground and background similarity, respectively. Foreground parts of ground truth and corresponding predicted map are compared in a holistic way, considering both contrast and uniform term. contrast uniform

Object-Level: foreground GT foreground FM 𝐷 𝐹𝐺 = 𝑥 𝐹𝐺 2 + 𝑦 𝐹𝐺 2 2 𝑥 𝐹𝐺 𝑦 𝐹𝐺 +𝝀∗ 𝝈 𝑥 𝐹𝐺 𝑥 𝐹𝐺 For Object-Level, we evaluate the foreground and background similarity, respectively. Foreground parts of ground truth and corresponding predicted map are compared in a holistic way, considering both contrast and uniform term. contrast uniform 𝑂 𝐹𝐺 = 1 𝐷 𝐹𝐺

Framework Here is our proposed framework including region and object similarity.

Framework Here is our proposed framework including region and object similarity.

Framework Here is our proposed framework including region and object similarity.

Framework 𝑆 𝑟𝑒𝑔𝑖𝑜𝑛 Here is our proposed framework including region and object similarity.

Framework 𝑆 𝑟𝑒𝑔𝑖𝑜𝑛 Here is our proposed framework including region and object similarity.

Framework 𝑆 𝑟𝑒𝑔𝑖𝑜𝑛 Here is our proposed framework including region and object similarity.

Framework 𝑆 𝑟𝑒𝑔𝑖𝑜𝑛 𝑆 𝑜𝑏𝑗𝑒𝑐𝑡 =𝑢∗ 𝑂 𝐹𝐺 +𝑣∗ 𝑂 𝐵𝐺 Here is our proposed framework including region and object similarity. 𝑆 𝑜𝑏𝑗𝑒𝑐𝑡 =𝑢∗ 𝑂 𝐹𝐺 +𝑣∗ 𝑂 𝐵𝐺

+ = Framework 𝑆 𝑟𝑒𝑔𝑖𝑜𝑛 𝑆 𝑜𝑏𝑗𝑒𝑐𝑡 =𝑢∗ 𝑂 𝐹𝐺 +𝑣∗ 𝑂 𝐵𝐺 Here is our proposed framework including region and object similarity. 𝑆 𝑜𝑏𝑗𝑒𝑐𝑡 =𝑢∗ 𝑂 𝐹𝐺 +𝑣∗ 𝑂 𝐵𝐺 =

+ = Framework 𝑆 𝑟𝑒𝑔𝑖𝑜𝑛 𝑆 𝑜𝑏𝑗𝑒𝑐𝑡 =𝑢∗ 𝑂 𝐹𝐺 +𝑣∗ 𝑂 𝐵𝐺 Here is our proposed framework including region and object similarity. 𝑆 𝑜𝑏𝑗𝑒𝑐𝑡 =𝑢∗ 𝑂 𝐹𝐺 +𝑣∗ 𝑂 𝐵𝐺 = 𝑆= 𝝰∗𝑆 𝑟𝑒𝑔𝑖𝑜𝑛 +(1−𝝰)∗ 𝑆 𝑜𝑏𝑗𝑒𝑐𝑡

Ranking example Here is a realistic example, each row shows ranking results of AP measure, AUC measure and our results. Our results consider important structure similarity, resulting in preferable results in real applications.

Ranking example Here is a realistic example, each row shows ranking results of AP measure, AUC measure and our results. Our results consider important structure similarity, resulting in preferable results in real applications.

Ranking example Here is a realistic example, each row shows ranking results of AP measure, AUC measure and our results. Our results consider important structure similarity, resulting in preferable results in real applications.

Ranking example Here is a realistic example, each row shows ranking results of AP measure, AUC measure and our results. Our results consider important structure similarity, resulting in preferable results in real applications.

Ranking example Here is a realistic example, each row shows ranking results of AP measure, AUC measure and our results. Our results consider important structure similarity, resulting in preferable results in real applications.

Ranking example Here is a realistic example, each row shows ranking results of AP measure, AUC measure and our results. Our results consider important structure similarity, resulting in preferable results in real applications.

Ranking example Our Here is a realistic example, each row shows ranking results of AP measure, AUC measure and our results. Our results consider important structure similarity, resulting in preferable results in real applications.

Meta-Measure1 Agree with the application: Saliency Cut To test the quality of our measure, we utilized 4 meta-measures which are used to evaluate the quality of measure. A good evaluation measure rank result should consist with the application rank result.

Meta-Measure-2 Prefer a good result over an Generic result (a)Image (b)GT (c)FM1 (d)Generic The second meta-measure is that the measure should prefer a good result over an generic result. The third is that the evaluation measure should decrease the score when using the wrong ground-truth.

Meta-Measure-2 Meta-Measure-3 Prefer a good result over an Generic result (a)Image (b)GT (c)FM1 (d)Generic Meta-Measure-3 WRONG ground-truth decrease score The second meta-measure is that the measure should prefer a good result over an generic result. The third is that the evaluation measure should decrease the score when using the wrong ground-truth. (a)Image (b)FM (c)GT (d)WRONG GT

Results Results in ASD dataset. (a)Meta-measure1 (b)Meta-measure2 (c)Meta-measure3 We do the experiment in ASD dataset and other 4 datasets, the results shown that our measure is better than others with a large gap.

Results Results in ASD dataset. (a)Meta-measure1 (b)Meta-measure2 (c)Meta-measure3 Results in other popular datasets. We do the experiment in ASD dataset and other 4 datasets, the results shown that our measure is better than others with a large gap.

Our measure is better than current measures. Results Results in ASD dataset. Our measure is better than current measures. (a)Meta-measure1 (b)Meta-measure2 (c)Meta-measure3 Results in other popular datasets. We do the experiment in ASD dataset and other 4 datasets, the results shown that our measure is better than others with a large gap.

Meta-Measure 4 Agree with the human ranking. The final meta-measure is that the evaluation measure rank result should agree with the human ranking.

Meta-Measure 4 Agree with the human ranking. The final meta-measure is that the evaluation measure rank result should agree with the human ranking.

Meta-Measure 4 Agree with the human ranking. The final meta-measure is that the evaluation measure rank result should agree with the human ranking.

Meta-Measure 4 Agree with the human ranking. Our VS. Fbw: 63.69% The final meta-measure is that the evaluation measure rank result should agree with the human ranking. Our VS. Fbw: 63.69% Our VS. AP : 72.11% Our VS. AUC: 73.56%

Ranking Results

Conclusion Simple（5.3ms） 5.4ms

Conclusion Simple（5.3ms） New insights (structure-wise)

Conclusion Simple（5.3ms） New insights (structure-wise) Unified evaluation to both binary and non-binary maps.

Thanks! http://dpfan.net/smeasure/ Free: C++/Matlab code Thank you for your attention.