SFNet: Learning Object-aware Semantic Correspondence Junghyup Lee1, Dohyung Kim1, Jean Ponce2,3, Bumshub Ham1 1Yonsei University 2DI ENS 3INRIA CVPR 2019(Oral) 1

Background Dense correspondence: Focus on different views of same scene/object. Stereo matching、optical flow of adjacent frames、 Image warp/alignment… Establishing dense correspondences across images is one of the fundamental tasks in computer vision Fig.1 SIFT correspondences

Outlier cleansing(RANSAC) Background Dense correspondence: Focus on different views of same scene/object. Stereo matching、optical flow of adjacent frames、 Image warp/alignment… Keypoint detection Keypoint description Matching Outlier cleansing(RANSAC) Establishing dense correspondences across images is one of the fundamental tasks in computer vision Fig.1 SIFT correspondences

Background Semantic correspondence: Focus on different instances of same scene/object category Hand crafted features do not capture high-level semantics Semantic correspondence algorithms (e.g., SIFT Flow [30]) go one step further, finding a dense flow field between images depicting different instances of the same object or scene category. This is very challenging especially in the presence of large changes in appearance/scene layout and background clutter. Reseaches remain stagnant util the revolution of deep learning hand-crafted features do not capture high-level semantics (e.g., appearance and shape variations), and are not robust to image-specific details (e.g., texture, background clutter, occlusion). Fig.2 semantic correspondences Jonathan L Long, Ning Zhang, and Trevor Darrell. Do convnets learn correspondence? In NIPS, 2014.

Related Methods Semantic Correspondence(Deep Learning Methods) 1. Pixel-level correspondence annotation is extremely labor-intensive and subjective. 2. Keypoint-level correspondence annotation is not sufficient for training CNN. ……. Proposals approach Geometric alignment problem all nodes in a neighborhood has the same label. People surely make progress.

Related Methods Semantic Correspondence(Deep Learning Methods) Proposals approach Generate proposals(Fast R-CNN) => Proposal matching => convert to dense correspondence Matching result is not smooth. A little complicated. all nodes in a neighborhood has the same label

Related Methods Semantic Correspondence(Deep Learning Methods) Geometric alignment problem To estimate parameters of a global transformation (e.g. affine, homography, thin plate spline) Fig.3 Pipeline of a typical geometric alignment approach. The input is two picture, the output is the 6 parameters for affine transformation. all nodes in a neighborhood has the same label Do not rely on annotation. Naturally smoothing. Sensitive to non-rigid deformation. Do not really establish semantic correspondence.

Proposed Method Overview all nodes in a neighborhood has the same label

Proposed Method Feature extraction and matching ℎ ∗𝑤 ∗𝑑 Adaption layer Resnet(conv5) Conv Batch-norm ReLu ℎ ∗𝑤 ∗𝑑 Adaption layer 𝑝: position of source feature map. 𝑞: position of target feature map. all nodes in a neighborhood has the same label

Proposed Method Kernel soft argmax layer 𝑐 𝑝 : correlation map for position 𝑝 at each spatial position 𝑞. (𝑤 ∗ℎ) We have to find the position with max response. Argmax: not differentiable, not enable fine-grained localization at image level. Soft argmax: not accurate for multi-modal distribution. (E = p*index) data=np.array([0.1, 0.3, 0.6, 2.1 ,0.55]) data = np.exp(data)/np.sum(np.exp(data)) # array([0.07795756, 0.09521758, 0.12853029, 0.57603278, 0.12226179]) Position = sum(data * array([0,1,2,3,4])) # Position:2.5694236670240085 all nodes in a neighborhood has the same label

Proposed Method Kernel soft argmax layer Apply Gaussian kernel to suppress others, make the distribution with one-peak. 𝑐 𝑝 : correlation map for position 𝑝 at each spatial position 𝑞. (𝑤 ∗ℎ) 𝑛 𝑝 : L2-normalized 𝑐 𝑝 . [0,0,0,0,…,0] [0,0,0,1,…,0] …… [0,0,0,0,…,0] (w x h) all nodes in a neighborhood has the same label 𝑘 𝑝 is a 2-d Gaussian kernel center at argmax( 𝑛 𝑝 ). 𝛽 is the hyper parameter.

Proposed Method How to use Loss For a pixel located at (x, y) in the source image with 320 x 320 resolution. The position in feature map is int(x/16, y/16). (suppose the feature map is 20 x 20) Using argmax to find the corresponding position in target feature map via 𝜙( 𝑥 16 , 𝑦 16 ). Loss all nodes in a neighborhood has the same label

Proposed Method Mask consistency loss Prevent the matches from foreground to background. First, we define flow filed of source Then, we reconstruct the position of source mask where is the ground truth target mask Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In NIPS, 2015.

Proposed Method a x a x b b Flow consistency loss One-to-one matching a x a x b b Unstable matching Many-to-one matching If the flow fields are consistent with others, have the same magnitude with opposite directions. Tinghui Zhou, Philipp Krahenbuhl, Mathieu Aubry, Qixing Huang, and Alexei A Efros. Learning dense correspondence via 3D-guided cycle consistency. In CVPR, 2016.

Proposed Method Smooth loss The matching within the foreground object should be smooth.

Experiments Dataset WILLOW: 900 image pairs of 4 categories. (key point annotation and bounding box annotation). PASCAL: 1351 image pairs of 20 categories. (key point and bounding box) Caltech101: annotated with object masks. all nodes in a neighborhood has the same label

Experiments Metric 1. PCK (probability of correct keypoint) Geometric alignment approach T: transformation parameters P: Image pairs depicting different instances B: Bounding boxes M: Masks Proposal approach (flow field approach)

Experiments Metric 2. LT-ACC/IoU PCK only cares the keypoint matching. The matching results or dense flow can be used to align image.

Experiments Ablation

Experiments Examples of alignment

Conclusion Present a CNN model for learning semantic flow end-to-end. Using binary masks to learn pixel-to-pixel correspondences. Kernel soft argmax, loss function design.