Learning Compositional Visual Concepts with Mutual Consistency Yunye Gong, Srikrishna Karanam, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, and Peter C. Doerschuk CVPR 2018 (spotlight) Concept GAN Presented by: Anand Bhattad (Vision Lunch, April 24 2018)

Image Translation (Pix to Pix and Cycle GAN) Pix2Pix Cycle GAN Source: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial

Cycle GAN

Source: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial

New Unseen Samples from Translations? Colored Bags Edge Bags Colored Shoes Edge Shoes

Overview: Cycle GAN G1 F1 F2 G2 F2 G2 G1 F1 Cycle GAN: Learn and Compose concepts separately G1 F1 G2 F2 G1 F1 G2 F2

Losses: Adversarial G1 F1 G2 F2

Losses: Cycle Consistency

Can we generate sketch shoes by combining them? F1 F2 G2 F2 G2 G1 F1

Proposed Model: Concept GAN Cycle GAN: Learn and Compose concepts separately Concept GAN: Learn and Compose concepts simultaneously G1 F1 F2 F2 G2 F2 G2 G1 F1

Losses: Cycle Consistency

Losses: Cycle Consistency (Dist=4) G1 G1 F1 F1 F2 G2 F2 G2

Losses: Cycle Consistency (Dist=4) G1 G1 F1 F1 F2 G2 F2 G2

Losses: Cycle Consistency (Dist=4) G1 G1 F1 F1 F2 G2 F2 G2

Loss: Commutative G1 G1 F1 F1 F2 G2 F2 G2

Loss: Commutative G1 F1 F2 G2 F2 G2 G1 F1

Loss: Commutative G1 F1 F2 G2 F2 G2 G1 F1

Concept GAN

Comparing Results

Qualitative Results

Qualitative Results

Quantitative Results (Classification) Test Set: edge Shoes only Test Set: with eyeglasses, with bangs 18 60 (Accuracy if L_x removed)

Ablation Study

Re-ID

Generalizing to n(=3) Concepts

Learning from 2 attributes in 2 experiments Experiment1: (smile, eyeglasses) Experiment2: (bangs, eyeglasses) (smile, eyeglasses, bangs)

Yaxing Wang, Joost van de Weijer, Luis Herranz CVPR 2018 Mix and match networks: encoder-decoder alignment for zero-pair image translation Yaxing Wang, Joost van de Weijer, Luis Herranz CVPR 2018

Task Mix and match networks do not require retraining on unseen transformations, in contrast to unpaired translation alternatives (e.g. CycleGAN) leverages depth and semantic segmentation, ignoring the available RGB data and the corresponding pairs

Overview

Losses

Depth –to- Semantic segmentation

Depth-to-Semantic segmentation