1. Adversarially Tuned Scene Generation
Presenter: Kaiyue Zhou

2. Pipeline Background Motivation Structure Approach Experiments
Existing methods for domain shift
Existing Scene generation techniques
Motivation
Structure
Approach
Experiments

3. Background Domain Shift: Scene Generation Techniques
Manually insert specific attributes such as illumination or pose to achieve invariance.
Learning a scene prior to generation for the specific target domain.
Scene Generation Techniques
Optimal spatial arrangement of 3D models is well studied:
Simulated annealing.
Reversible-jump MCMC (Monte-Carlo Markov chain)
Factor potentials

4. Motivation
Recent works: Augmenting simulated data with a few labelled real samples can ameliorate domain shift. But costly.
GAN uses unlabeled samples to obtain better estimation of parameters in generative models.
Generator G: generate semantic label maps;
Discriminator D: classify real and virtual data.

5. Structure

7. Discriminator Discriminator D: AlexNet
Standard stochastic gradient descent with backpropagation.
Using data augmentation techniques.
Tuning G: classifier outputs conditional probability p(v=real|Θ)

8. Generator Generator G: DeepLab Real world target datasets
Using Atrous convolutions for multi-scale objects;
Fully connected conditional random field (CRF) for obtain relations of pixel to pixel and instance to instance.
Able to generate 7 classes in this case: vehicle, pedestrian, building, vegetation, road, ground, and sky.
Real world target datasets
CityScapes
CamVid

9. Approach Probabilistic scene generation Geometry: Photometry:
two kinds of probabilistic distributions for points and attributes;
Photometry:
environmental parameters like light, weather and camera.
Blender to render.
Dynamics (not in this work)

10. Geometry
Spatial non-overlap, cooccurrence and coherence among instances are incorporated with Gibbs potentials. Density of object layouts:
Where

11. Photometry

12. Experiments

14. Presenter: Kaiyue Zhou
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
Presenter: Kaiyue Zhou

15. Pipeline Background and Motivation Structure Approach Experiments
Use of GANs and purpose
Existing image-to-image techniques and purpose
Existing visual manipulation and purpose
Structure
Approach
Experiments

16. Background and Motivation
GANs
Unconditional setting
Image generation
image manipulation
representation learning
object detection
video apps
Aim to coarse-to-fine generator and multi-scale discriminator for conditional image generation

17. Background and Motivation
Image-to-image translation
Conventional L1 loss
Blurry
Conditional GANs using adversarial loss function to avoid blurry images
Pix2pix framework (hard for high-resolution)
Perceptual loss (high-resolution but lack fine details)
Aim to high-resolution and fine details

18. Background and Motivation
Deep visual manipulation
Works well for style transfer, inpainting, colorization and restoration
Lack of interface to operate on the result
Or low level manipulation such as color and sketch
Aim to provide user interface for object-level semantic editing.

19. Structure – generator Baseline: pix2pix framework
Generator G: U-Net
Discriminator D: a patch-based fully convolutional network
Resolution: 256*256
Improvement for high-resolution
Global generator network G1
Local enhancer network G2

20. Structure – generator

21. Structure – discriminator
Multi-scale discriminators: 3 discriminators with different scales for different resolution of images.
In addition, feature matching loss is incorporated by match intermediate representations of real and synthesized image from multiple layers of discriminator.

22. Instance map
Semantic label map: does not consider the boundary of instances.
Concatenated with the one-hot vector representation from semantic label map to be fed to generator.
As well to discriminator.

23. Learning with instance-level feature
Using a feature encoder, which is a standard encode- decoder network.
Change G(s) to G(s,E(x)) in the loss function.

24. Experiments (semantic segmentation)
PSPNet is used to predict the ground truth label.

25. Experiments
Image generation

26. Experiments (editing)

27. Experiments (editing)