Deep Cross-media Knowledge Transfer CVPR 2018 Deep Cross-media Knowledge Transfer Xin Huang, and Yuxin Peng* Institute of Computer Science and Technology, Peking University, Beijing 100871, China {huangxin_14@pku.edu.cn, pengyuxin@pku.edu.cn} Hello everyone! Very glad to present our work in CVPR 2018, Deep Cross-media Knowledge Transfer. The authors are Xin Huang and Professor Yuxin Peng. We are from Peking University. I’m Xin Huang.

Cross-media Retrieval Cross-media retrieval aims to perform retrieval across different media types, such as image, text, video, audio, and 3D model Results of various media types Image Text Video Audio 3D Model Multimedia data Query of any media type The topic of this paper is cross-media retrieval, which aims to perform retrieval across different media types. For example, by an image query of Golden Gate Bridge, the system can return relevant data of image, text, video, audio, and 3D model. Image (Golden Gate Bridge)

Motivation Key problem of cross-media retrieval: Different media types have inconsistent representations (i.e., Heterogeneity Gap) Deep network Common representation Color Texture Shape Image Cross-media similarity cannot be computed Heterogeneous The key problem of cross-media retrieval is that different media types have inconsistent representations, so their similarity cannot be directly computed. The current mainstream methods are to use deep learning for generating cross-media common representation, so the cross-media similarity can be computed. Cross-media similarity can be computed Text Word Frequency

Motivation Common challenge: Insufficient labeled cross-media data Retrieval accuracy Usually relies on data amount Deep Network Needs large-scale data Cross-media data labeling Extremely labor-consuming Read text Watch video See image Listen to audio Observe 3D model Contradiction Existing methods face the challenge of insufficient labeled cross-media data. The retrieval accuracy usually relies on the amount of training data. However, it is extremely labor-consuming to label cross-media data. Imagine you are labeling the data of “Golden Gate Bridge”, you have to see image, read text, watch video, listen to audio, and even observe 3D model. This limits the performance of current cross-media retrieval methods. Limit the performance of existing cross-media retrieval methods

Inter-media correlation Motivation Transfer learning is usually used to relieve the problem of insufficient training data However, existing transfer methods are mostly designed within the same media type Source domain Target domain Transfer Intra-media semantic knowledge transfer Inter-media correlation Correlation Source domain Target domain Transfer Transfer learning is usually used to relieve the problem of insufficient training data. However, existing transfer methods are mostly designed within the same media type. They pays little attention to knowledge transfer between two cross-media domains, including Intra-media semantic and Inter-media correlation knowledge. Therefore, we propose deep cross-media knowledge transfer (DCKT), to transfer from cross-media source to target domains. We propose deep cross-media knowledge transfer (DCKT), to transfer from cross-media source to target domains

Network Architecture An end-to-end architecture with 2 transfer levels: Media-level transfer and Correlation-level transfer Media-level Transfer MMD loss Separate Representation Adaptation Pairwise constraint loss Coexistence Information This is the overall architecture of DCKT, which has two transfer levels: First, media-level transfer is proposed to transfer intra-media knowledge between two domains, with MMD loss and pairwise constraint loss.

Network Architecture An end-to-end architecture with 2 transfer levels: Media-level transfer and Correlation-level transfer Correlation-level Transfer MMD loss Common Representation Adaptation Semantic loss Supervised Label Information Second, correlation-level transfer is proposed to align inter-media correlation of the two domains, including MMD loss and semantic loss

Correlation-level Transfer Network Architecture An end-to-end architecture with 2 transfer levels: Media-level transfer and Correlation-level transfer Jointly transfer to boost retrieval accuracy of target domain In this way, knowledge in source domain can be jointly transferred to target domain for boosting retrieval accuracy. Media-level Transfer Correlation-level Transfer

Progressive Transfer Mechanism Problem: Some hard samples in target domain are not so relevant to source domain, leading to negative transfer effect Network pre-training Source data Source cross-media retrieval model Domain consistency metric Target data Correlation prediction Source model For robust transfer and higher retrieval accuracy Cross-media domain gap is very vast, and hard transfer samples lead to negative transfer effect. We further propose Progressive Transfer Mechanism. We let the model trained in source domain serve as a teacher. Then we automatically select samples in target domain during transfer process. In this way, we gradually transfer from easy to hard samples, For robust transfer and higher retrieval accuracy Transfer sample selection Easy samples （High consistency） Hard samples （Low consistency ） … Domain consistency ranking Sample selection probability 𝑃𝑟𝑜𝑏 𝑞 =𝛼[1− 𝑙𝑜𝑔 2 𝑚𝑎𝑥⁡(𝐴𝑃)− 𝐴𝑃 𝑞 𝑚𝑎𝑥⁡(𝐴𝑃)×𝑖𝑡𝑒𝑟 +1 ]

Experiment Dataset Source domain Target (retrieval) domain: XMediaNet (self-constructed): The first large-scale cross-media dataset with 5 media types, and more than 100,000 instances Target (retrieval) domain: Wikipedia: 2,866 image/text pairs NUS-WIDE-10k:10,000 image/text pairs Pascal Sentences: 1,000 image/text pairs Next, experiment. The self-constructed XMediaNet dataset is used as the source domain, which is a large-scale dataset with more than 100 thousand instances. We adopt 3 widely-used, but small-scale datasets as target domain. Compared with the methods which are only trained with the target domain, DCKT can obtain an inspiring improvement. DCKT achieves the highest accuracy compared with state-of-the-art methods

Multimedia Information CVPR 2018 Lab Homepage Github Homepage Thank you for your listening. Here are the QR codes of our lab home page and Github home page. Welcome to our websites for more information. http://www.icst.pku.edu.cn/mipl Multimedia Information Processing Lab (MIPL)