David Healey BYU Capstone Course 15 Nov 2018 Lecture 4: VAEs, GANs David Healey BYU Capstone Course 15 Nov 2018

Administrative things

Homework: Kaggle stock exchange prediction Our homework will be a group assignment to do a Kaggle competition involving stock prediction from the new york stock exchange https://www.kaggle.com/dgawlik/nyse/home

About Recursion Pharmaceuticals To get a data science job at Recursion Pharmaceuticals: Pass a resume screen Practical experience Advanced degree helps Looking for depth Pass a work sample 2 hours to complete several questions testing general ability around data analysis ”What kind of analysis would constitute an answer to this question?” Evaluation of the method you used Pass 4 or 5 onsite interviews Light coding, quantitative reasoning, machine learning Kinds of jobs at Recursion: Data scientist Data engineer Software Engineer

How to contact me david.w.healey@gmail.com LinkedIn

Generative models: Variational Autoencoders and GANs

Unsupervised generative models: VAEs and GANs The goal is to create a decoder or a generator: a neural network that maps an arbitrary low- dimensional vector into a very high-dimensional and complicated output that matches a very complicated data distribution like images or language A model that can create something new that looks like its from the data “understands” the distribution of the data very well. Example: can you create the face of a person who doesn’t exist yet z Low dimensional “latent space” High dimensional

Variational autoencoders (VAE) Normal autoencoder Variational autoencoder Encode to a distribution instead of a single point Decoder first samples from the distribution. Try to force the space to be continuous and meaningful so that arbitrary vectors will map to something real-seeming Encode data to a vector whose dimension is less than the

VAE loss 2 parts to the loss Reconstruction loss (mse): did you get the same data out that you put in? This makes sure the model learns the distribution of the data, like a regular autoencoder K-L loss (KL divergence from unit normal): Did you encode the point to a distribution that somewhat resembles a unit normal? (this makes sure the latent space is more or less continuous)

Variational Autoencoders: Generating Data! Data manifold for 2-d z Dim 1 Labeled Faces in the Wild Figures copyright (L) Dirk Kingma et al. 2016; (R) Anders Larsen et al. 2017. Reproduced with permission. Dimension 2 Remember: these are new numbers, not part of the training set Kingma and Welling, “Auto-Encoding Variational Bayes”, ICLR 2014 Used with permission from Justin Johnson

Generative Adversarial Networks Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014 Problem: Want to sample from complex, high-dimensional training distribution. No direct way to do this! Solution: Sample from a simple distribution, e.g. random noise. Learn transformation to training distribution. Output: Sample from training distribution Q: What can we use to represent this complex transformation? A: A neural network! Transform Input: Random noise sample z 10 3 Lecture 12 - May 15, 2018 Used with permission from Justin Johnson

Training GANs: Two-player game Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014 Generator network : try to fool the discriminator by generating real-looking images Discriminator network : try to distinguish between real and fake images Real or Fake Discriminator Network (CNN) Fake Images (from generator) Real Images (from training set) Generator Network (upsampling convolutional network) Random noise z Fake and real images copyright Emily Denton et al. 2015. Reproduced with permission. 10 5 Lecture 12 - May 15, 2018 Used with permission from Justin Johnson

Training GANs Alternate between training the generator and discriminator (you can train for one step each or multiple steps each) Discriminator: Generate some fake examples with the generator Get some real examples Discriminator tries to score the fake examples low and the real ones high So GAN loss punishes discriminator for high scores on fake examples and low scores on real examples Generator: Generate some fake images and give them to the discriminator to score Generator tries to get high scores from the discriminator So GAN loss punishes generator for low scores from the discriminator Warning: notoriously hard to train and unstable. See for tips: https://github.com/soumith/ganhacks

Generative Adversarial Nets: Convolutional Architectures Samples from the model look much better! Radford et al, ICLR 2016 12 Lecture 12 - May 15, 2018 Used with permission from Justin Johnson

Generative Adversarial Nets: Interpretable Vector Math Glasses man No glasses man No glasses woman Radford et al, ICLR 2016 Radford et al, ICLR 2016 Lecture 12 - May 15, 2018 Used with permission from Justin Johnson

Generative Adversarial Nets: Interpretable Vector Math Glasses man No glasses man No glasses woman Radford et al, ICLR 2016 Woman with glasses 12 6 Lecture 12 - May 15, 2018 Used with permission from Justin Johnson

See how far we’ve come since then https://www.youtube.com/watch?v=XOxxPcy5Gr4

Resources to look for for help When you don’t know what approach to take: Talk to someone Examples to work from, when you know what you need to build but don’t know how to code and train it: Keras blog has a lot of examples (our translation example was adapted from one) https://github.com/keras-team/keras/tree/master/examples Google search usually brings up at least one github repo if you search something like Keras GAN Concepts and Courses for learning and clarifying concepts: Deeplearning.ai Andrew Ng (pretty friendly) Cs231n Stanford course (heavy on equations). A lot of the slides here were adapted w/permission from that course. Geoff Hinton’s DL Coursera (downside: OCTAVE not Python)

Some things to say about data science and artificial intelligence

The demand for machine learning is crazy high right now Data scientist is the highest rated job in the country 3 years running Machine learning scientists have higher compensation than comparably-credentialed peers Utah has a particular shortage of data scientists which is about to get worse

The future of Artificial Intelligence

What happens when machines displace human labor?

Real earnings relative to 1963 (men)

“And now behold, my beloved brethren, … do not suppose that this is all; for after ye have done all these things, if ye turn away the needy, and the naked, and visit not the sick and afflicted, and impart of your substance, if ye have, to those who stand in need—I say unto you, if ye do not any of these things, behold, your prayer is vain, and availeth you nothing, and ye are as hypocrites who do deny the faith.” (Alma 34:28.)