Adversarially Tuned Scene Generation Presenter: Kaiyue Zhou

Pipeline Background Motivation Structure Approach Experiments Existing methods for domain shift Existing Scene generation techniques Motivation Structure Approach Experiments

Background Domain Shift: Scene Generation Techniques Manually insert specific attributes such as illumination or pose to achieve invariance. Learning a scene prior to generation for the specific target domain. Scene Generation Techniques Optimal spatial arrangement of 3D models is well studied: Simulated annealing. Reversible-jump MCMC (Monte-Carlo Markov chain) Factor potentials

Motivation Recent works: Augmenting simulated data with a few labelled real samples can ameliorate domain shift. But costly. GAN uses unlabeled samples to obtain better estimation of parameters in generative models. Generator G: generate semantic label maps; Discriminator D: classify real and virtual data.

Structure

Discriminator Discriminator D: AlexNet Standard stochastic gradient descent with backpropagation. Using data augmentation techniques. Tuning G: classifier outputs conditional probability p(v=real|Θ)

Generator Generator G: DeepLab Real world target datasets Using Atrous convolutions for multi-scale objects; Fully connected conditional random field (CRF) for obtain relations of pixel to pixel and instance to instance. Able to generate 7 classes in this case: vehicle, pedestrian, building, vegetation, road, ground, and sky. Real world target datasets CityScapes CamVid

Approach Probabilistic scene generation Geometry: Photometry: two kinds of probabilistic distributions for points and attributes; Photometry: environmental parameters like light, weather and camera. Blender to render. Dynamics (not in this work)

Geometry Spatial non-overlap, cooccurrence and coherence among instances are incorporated with Gibbs potentials. Density of object layouts: Where

Photometry

Experiments

Presenter: Kaiyue Zhou High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs Presenter: Kaiyue Zhou

Pipeline Background and Motivation Structure Approach Experiments Use of GANs and purpose Existing image-to-image techniques and purpose Existing visual manipulation and purpose Structure Approach Experiments

Background and Motivation GANs Unconditional setting Image generation image manipulation representation learning object detection video apps Aim to coarse-to-fine generator and multi-scale discriminator for conditional image generation

Background and Motivation Image-to-image translation Conventional L1 loss Blurry Conditional GANs using adversarial loss function to avoid blurry images Pix2pix framework (hard for high-resolution) Perceptual loss (high-resolution but lack fine details) Aim to high-resolution and fine details

Background and Motivation Deep visual manipulation Works well for style transfer, inpainting, colorization and restoration Lack of interface to operate on the result Or low level manipulation such as color and sketch Aim to provide user interface for object-level semantic editing.

Structure – generator Baseline: pix2pix framework Generator G: U-Net Discriminator D: a patch-based fully convolutional network Resolution: 256*256 Improvement for high-resolution Global generator network G1 Local enhancer network G2

Structure – generator

Structure – discriminator Multi-scale discriminators: 3 discriminators with different scales for different resolution of images. In addition, feature matching loss is incorporated by match intermediate representations of real and synthesized image from multiple layers of discriminator.

Instance map Semantic label map: does not consider the boundary of instances. Concatenated with the one-hot vector representation from semantic label map to be fed to generator. As well to discriminator.

Learning with instance-level feature Using a feature encoder, which is a standard encode- decoder network. Change G(s) to G(s,E(x)) in the loss function.

Experiments (semantic segmentation) PSPNet is used to predict the ground truth label.

Experiments Image generation

Experiments (editing)

Experiments (editing)