LF Deep Learning Foundation Technical Advisory Council Meeting January 17, 2019

Recording of Calls This is a reminder that we have decided to record TAC meetings and store them on the TAC Wiki.

Antitrust Policy Notice Linux Foundation meetings involve participation by industry competitors, and it is the intention of the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Examples of types of actions that are prohibited at Linux Foundation meetings and in connection with Linux Foundation activities are described in the Linux Foundation Antitrust Policy available at http://www.linuxfoundation.org/antitrust-policy. If you have questions about these matters, please contact your company counsel, or if you are a member of the Linux Foundation, feel free to contact Andrew Updegrove of the firm of Gesmer Undergone LLP, which provides legal counsel to the Linux Foundation.

Dial-in Information Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/430697670 Or iPhone one-tap: US: +16465588656,,430697670# or +16699006833,,430697670# Or Telephone: Dial(for higher quality, dial a number based on your current location): US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free) Meeting ID: 430 697 670 International numbers available: https://zoom.us/u/achYtcw7uN

Agenda Roll Call Project Contribution Proposal Review & Discussion: Pyro Next Meeting Information Discussion / Q&A 11/16/18

TAC Member Directory Member TAC Contact TAC Email Amdocs Ofer Hermoni* ofer.hermoni@amdocs.com AT&T Jack Murray jfm@research.att.com Baidu Xin Pan panxin01@baidu.com Ericsson Nimisha Radia nimish.radia@ericsson.com Huawei Yinglong Xia yinglong.xia@huawei.com Nokia Pantelis Monogioudis pantelis.monogioudis@nokia.com Tech Mahindra Manish Kampani manish.kampani@techmahindra.com Tencent XiaoLong Zhu xiaolongzhu@tencent.com ZTE Wei Meng meng.wei2@zte.com.cn Acumos AI Project *TAC Chairperson

TAC Overview LF Deep Learning Foundation is an umbrella project of The Linux Foundation with the mission of supporting AI, ML and DL open source projects. LF DL currently has three projects and we are accepting contributions of additional projects. The TAC is a committee within LF DL that is responsible for assisting in the coordination and communication of projects within LF DL. It is not a technical oversight body, all technical oversight is handled by the individual projects themselves according to their own governance. Since the March 2018 launch of LF DL, activity of the TAC has included: approving a project lifecycle for projects of LF DL (incubation, graduation, archive stages); reviewing and then approving contribution proposals for the Angel and EDL projects; starting work on a landscape analysis diagram for open source AI projects overall; and identifying the area of model interchange formats for exploration by the TAC.

TAC Wiki We have established a Wiki for the TAC which can be found here (login to lists.deeplearning.org required): https://lists.deeplearningfoundation.org/g/tac-general/wiki/home

Project Contribution Proposal Review & Discussion: Pyro Project Pyro Deep universal probabilistic programming with Python and PyTorch http://pyro.ai/ https://github.com/uber/pyro (As of Jan 14th: 50 contributors, 4870 stars) Presenter: Fritz Obermeyer, Pyro Engineering Lead, Uber AI Labs

Pyro: Deep Universal Probabilistic Programming Fritz Obermeyer, Pyro engineering lead, Uber AI Labs

ML Problems at Uber Uber does not have one machine learning problem, it has thousands.

ML Solutions in Practice

ML Solutions in Practice

Probabilistic programming Build custom solutions that combine the best of all worlds: - probabilistic models (handling uncertainty) - high-level programming languages (means of abstraction) - scalable deep learning (learning from large data)

Example application: semi-supervised learning Suppose we want to classify handwritten digits... but we only have a few labeled examples

Pyro: Python with two extra primitives x = pyro.sample(“x”, Bernoulli(0.5)) assert isinstance(x, torch.Tensor) pyro.sample(“data”, Normal(0., 1.), obs=data) theta = pyro.param(“theta”, torch.ones(100), constraint=positive) return value of pyro.sample(...) is a torch.Tensor (not an RV object). pyro.sample(..., obs=...) acts as a loss term. pyro.param(...) persists across function calls.

Pyro models generate data style digit digit decoder = ... # nn def model(image=None): pyro.module("decoder", decoder) style = pyro.sample("style", Normal(zeros(20), ones(20))) digit = pyro.sample("digit", Categorical(ones(10) * 0.1)) image = pyro.sample("image", Bernoulli(decoder(style, digit)), obs=image) return image image Semi-supervised Learning with Deep Generative Models, Diederik P Kingma, Danilo Jimenez Rezende, Shakir Mohamed, Max Welling

Approximate inference as optimization We can do inference by learning a model of the posterior, called a guide: Variational Bayes and Beyond: Bayesian Inference for Big Data, Broderick, T.

Pyro guides infer latent variables style digit digit def model(image=None): ... def guide(image): pyro.module("encoder_digit", encoder_digit) pyro.module("encoder_style", encoder_style) digit = pyro.sample("digit", Categorical(encoder_digit(image))) loc, scale = encoder_style(image, digit) style = pyro.sample("style", Normal(loc, scale)) return digit, style image go into detail

Pyro SVI trains the guide style digit digit def model(image): ... def guide(image): ... svi = SVI(model, guide, optim.Adam(), TraceEnum_ELBO()) for epoch in range(10): for batch in partition(data): svi.step(batch) image infer.step() updates parameters registered with pyro.param() statement. T=this uses monte carlo estimate of a loss function (elbo).

The guide now serves as a classifier style digit digit def model(image): ... def guide(image): ... # ...train... digit, style = guide(image) image

Results style digit digit image

Pyro Pyro stack Generative models Custom inference components Inference building blocks (e.g. loss functions, MCMC kernels) Pyro Runtime Ops, distributions Effect handlers Candidate #1 PyTorch 1.0: JIT, autograd, distributions, tensor math Python C++ CUDA

Research advances must interact correctly Subsampling Reparameterization Sticking the Landing SVI Neural Baselines RSVI ADVI Discrete Enumeration Each of these are papers and months to years of work. They all have to interact with each other Message Passing Rao- Blackwellization DiCE

Documentation catalyzes community growth Design interfaces to be easy to document Document internals, e.g. minipyro.py Follow Python & PyTorch idioms Tutorials, examples, reference, forum, tech reports http://pyro.ai/examples http://docs.pyro.ai Doctests and usage examples Allow researchers to hack Pyro

Community goals: automated inference

Community We collaborate on PyTorch Distributions Open Design Process - university collaborators around the world - contribute code to other OSS

Community pyro.contrib.oed pyro.contrib.tracking goals: automated inference

Community goals: automated inference

Results at Uber and beyond Pyro is used by large companies like Siemens, IBM, and Uber, and startups like Noodle.AI Pyro is widely used in research institutions including Harvard, MIT, Stanford, Oxford, Cambridge, and The Broad Institute Pyro solves a range of Uber problems, spanning: sensor fusion, time series forecasting, ads campaign optimization, and data augmentation for deep image understanding Joint Mapping and Calibration via Differentiable Sensor Fusion, Chen et al. (2018)

2019H1 Roadmap Engineering foundations: - Improve support for deployment/serving with PyTorch 1.0 - Pay down tech debt and simplify user interfaces - OSS citizenship: push code upstream to PyTorch and GPyTorch Supporting applications, at Uber and beyond: - Optimal experimental design - Sensor fusion & perception - Time series segmentation Core PPL research: - Automating inference patterns - Improving inference algorithm scalability and performance

For documentation, examples, and more: https://pyro.ai Thank you! For documentation, examples, and more: https://pyro.ai Alternate end slide design

TAC Vote on Project Proposal ›Proposed Resolution: The TAC approves the Pyro Project as an incubation project of the LF Deep Learning Foundation.

TAC Vote on Project Proposal RESOLVED: The TAC approves the Pyro Project as an incubation project of the LF Deep Learning Foundation.

Next Meeting Information Upcoming Meetings January 31, 2019: Invited Presentation: Pachyderm February 14, 2019: Regular TAC Call (No invited speaker)

Questions / Discussion

Legal Notices The Linux Foundation, The Linux Foundation logos, and other marks that may be used herein are owned by The Linux Foundation or its affiliated entities, and are subject to The Linux Foundation’s Trademark Usage Policy at https://www.linuxfoundation.org/trademark-usage, as may be modified from time to time. Linux is a registered trademark of Linus Torvalds. Please see the Linux Mark Institute’s trademark usage page at https://lmi.linuxfoundation.org for details regarding use of this trademark. Some marks that may be used herein are owned by projects operating as separately incorporated entities managed by The Linux Foundation, and have their own trademarks, policies and usage guidelines. TWITTER, TWEET, RETWEET and the Twitter logo are trademarks of Twitter, Inc. or its affiliates. Facebook and the “f” logo are trademarks of Facebook or its affiliates. LinkedIn, the LinkedIn logo, the IN logo and InMail are registered trademarks or trademarks of LinkedIn Corporation and its affiliates in the United States and/or other countries. YouTube and the YouTube icon are trademarks of YouTube or its affiliates. All other trademarks are the property of their respective owners. Use of such marks herein does not represent affiliation with or authorization, sponsorship or approval by such owners unless otherwise expressly specified. The Linux Foundation is subject to other policies, including without limitation its Privacy Policy at https://www.linuxfoundation.org/privacy and its Antitrust Policy at https://www.linuxfoundation.org/antitrust-policy. each as may be modified from time to time. More information about The Linux Foundation’s policies is available at https://www.linuxfoundation.org. Please email legal@linuxfoundation.org with any questions about The Linux Foundation’s policies or the notices set forth on this slide. The Linux Foundation Internal Use Only 1/3/18