Machine Learning Platform Life-Cycle Management Hope(Xinwei) Wang Software Engineer at Intuit

ABOUT ME Software Engineer at Intuit (Small Business Group) since Jan. 2017 M.S. in Biomedical Engineering from University of Southern California(USC), Los Angeles (Dec. 2016) Self-taught, self-motivated programmer Email: xinweiwang3@gmail.com Twitter: @HopeXinwei Linkedin: Hope(Xinwei) Wang https://www.linkedin.com/in/xinweiwanglinkedin/

ABOUT INTUIT You may have heard of or used some of Intuit’s products. We make financial software for consumers, small businesses and the self-employed. Our mission is to power prosperity around the world. Here’s one of our main product at Intuit: Quickbooks, accounting software for small and medium-sized businesses which offers them accounting applications, including accepting business payments, managing and pay bills, and payroll functions.

What is a machine learning platform? Machine Learning Platform Life-Cycle Management Machine Learning Platform Life-Cycle Management OVERVIEW What is a machine learning platform? What is the ML platform lifecycle? Why ML platform lifecycle management? Artifacts and their associations Use cases at Intuit

What is a Machine Learning Platform? Machine Learning Platform Life-Cycle Management What is a Machine Learning Platform? Manages the entire lifecycle of an ML model Includes automating and accelerating the delivery of ML applications

Machine Learning Platform Life-Cycle Management

Life-Cycle Management Machine Learning Platform Life-Cycle Management ML Platform Life-Cycle Data Ingestion Data Discovery Feature Engineering Model Development Model Training Model Scoring This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation. Life-Cycle Management

Metadata/Catalog tool Machine Learning Platform Life-Cycle Management Data Discovery Metadata/Catalog tool Metadata/catalog tool Accessible data source (Raw attributes & Data lineage) Data Lake This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Metadata/Catalog tool Machine Learning Platform Life-Cycle Management Feature Engineering Metadata/Catalog tool Output : features Reproducible Reusable Feature Repository This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Model Development Collaborative environment Access data lake Machine Learning Platform Life-Cycle Management Model Development Notebooks Collaborative environment Access data lake Data Lake This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Model Training Support ability for: Machine Learning Platform Life-Cycle Management Model Training Support ability for: Being triggered either manually/via automation Creation and management of training sets Re-training Optimizing hyper parameter tuning through parallelization of model training execution This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Model Scoring Support online/offline(depend on use cases) Machine Learning Platform Life-Cycle Management Model Scoring Support online/offline(depend on use cases) Ability to be triggered either manually/via automation This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Big Mess! No central artifact management solution Machine Learning Platform Life-Cycle Management Big Mess! No central artifact management solution Hard to reuse existing features/data/algorithms/toolings Inability to scale for large datasets Lack of automation/orchestration across the ML life-cycle Lack of rigor/discipline in the ML development life-cycle Slow down delivery of Machine Learning applications This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Ideal Status Artifact Management Machine Learning Platform Life-Cycle Management Ideal Status Optimizing data scientists’ engineering process Tie ML components together into a cohesive platform, support the life-cycle of ML artifacts end-to-end Increase efficiency of delivering ML predictions at scale This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation. Artifact Management

Environment Artifacts Machine Learning Platform Life-Cycle Management Data Artifacts Features Training sets Model Artifacts Model code Trained models Performance metrics Hyper parameter values Environment Artifacts Languages & language versions Packages & Package versions This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Life-Cycle Management Machine Learning Platform Life-Cycle Management Data Ingestion Data Discovery Feature Engineering Model Development Model Training Model Scoring This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation. Life-Cycle Management

CONTAINERIZATION! Environment in Container Machine Learning Platform Life-Cycle Management CONTAINERIZATION! This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation. Environment in Container

Benefit Of Containerization Machine Learning Platform Life-Cycle Management Benefit Of Containerization Flexibility : Model has specific environment Consistency : Model has same behavior throughout the life-cycle This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Machine Learning Platform Life-Cycle Management Feature Set Definition Features Model Artifacts Data Artifacts Environment Artifacts Hyper-parameter values Model Training Datasets Scheduled Re-training/ performance benchmarks Metrics Definition Environment Metadata This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation. Performance Metrics Trained Models Container Model Scoring

Model Code Developed in notebooks Multiple versions Machine Learning Platform Life-Cycle Management Model Code Developed in notebooks Multiple versions Each version associate with an externalized environment artifact This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Environment Artifacts Machine Learning Platform Life-Cycle Management Environment Artifacts Environment must be consistent for development, training, scoring Externalized as metadata Model/Execution environments constructed from metadata and deployed into containers (Docker, Yarn, Conda, etc.) This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Examples of containers/virtual environments Machine Learning Platform Life-Cycle Management Examples of containers/virtual environments Tailored to the environment (built based on externalized environment metadata) Used for model development, training, execution Container/virtual environment Usage Docker container Model development Model training Online scoring Yarn container On Spark cluster Distributed training Batch offline scoring Conda environment This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Features Used as data input of the model Machine Learning Platform Life-Cycle Management Features Used as data input of the model Stored in discoverable feature repository Metadata defines the model specific feature-sets This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Trained Models Serialized, weighted model files Machine Learning Platform Life-Cycle Management Trained Models Serialized, weighted model files Associate with a version of model code and training set This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Training Sets Datasets used to train, validate and test the model Machine Learning Platform Life-Cycle Management Training Sets Datasets used to train, validate and test the model Associated to a trained model This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Feature Set Definitions Machine Learning Platform Life-Cycle Management Feature Set Definitions Define what feature sets this model requires This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Hyper Parameter Values Machine Learning Platform Life-Cycle Management Hyper Parameter Values Set up values before learning process Model specific This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Machine Learning Platform Life-Cycle Management Metric Definition Defines the metrics to collect and thresholds to evaluate models against. This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Performance Metrics Metrics to evaluate model effectiveness Machine Learning Platform Life-Cycle Management Performance Metrics Metrics to evaluate model effectiveness Model metrics including: ROC curve, confusion matrix, F1 score, precision, recall, etc. This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Scheduled Re-training & performance benchmarks Machine Learning Platform Life-Cycle Management Scheduled Re-training & performance benchmarks To automate the re-training and deployment of updated models Model specific This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Model Training in Docker Container Machine Learning Platform Life-Cycle Management Model Training in Docker Container Docker Base Image Docker file Environment Artifact Model Docker Image Host Server Feature Discovery & Model Development Model Docker Container Trained Model This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation. Model Code Training Datasets

Life-Cycle Management Machine Learning Platform Life-Cycle Management Data Ingestion Data Discovery Feature Engineering Model Development Model Training Model Scoring This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation. Life-Cycle Management

Model Development & Training & Tuning Machine Learning Platform Life-Cycle Management Model Development & Training & Tuning This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation.

Self-help Service in Quickbooks Machine Learning Platform Life-Cycle Management Get Contextual help Personalization Service Clickstream Data Feature Service Score Data Online Scoring Service Batch Source Kafka Spark Streaming Online Store Online Features This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation. Offline Features Lake Feature Repository Offline Spark Job

Example: Online Scoring Service Deployment Diagram Auto-Deployment Tool Merge Training Environment Model Save Deploy Code Model Repository Deploy Template Wrap Serialized Model Trained Model Serialized Model Code Repository Model Specific Deploy Code/metadata Online Scoring Worker Node Docker Container Deploy Environment Metadata Model Model Specific Docker Image Docker File Build Docker Build Docker Registry Docker Base Image

Life-Cycle Management Machine Learning Platform Life-Cycle Management Data Ingestion Data Discovery Feature Engineering Model Development Model Training Model Scoring This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation. Life-Cycle Management

Example: Online Scoring Service Deployment Diagram Auto-Deployment Tool Merge Training Environment Model Save Deploy Code Model Repository Deploy Template Wrap Serialized Model Trained Model Serialized Model Code Repository Model Specific Deploy Code/metadata Online Scoring Worker Node Docker Container Deploy Environment Metadata Model Model Specific Docker Image Docker File Build Docker Build Docker Registry Docker Base Image

Example: Online Scoring Service Deployment Diagram Auto-Deployment Tool Merge Training Environment Model Save Deploy Code Model Repository Deploy Template Wrap Serialized Model Trained Model Serialized Model Code Repository Model Specific Deploy Code/metadata Online Scoring Worker Node Docker Container Deploy Environment Metadata Model Model Specific Docker Image Docker File Build Docker Build Docker Registry Docker Base Image

Example: Online Scoring Service Deployment Diagram Auto-Deployment Tool Merge Training Environment Model Save Deploy Code Model Repository Deploy Template Wrap Serialized Model Trained Model Serialized Model Code Repository Model Specific Deploy Code/metadata Online Scoring Worker Node Docker Container Deploy Environment Metadata Model Model Specific Docker Image Docker File Build Docker Build Docker Registry Docker Base Image

Example: Online Scoring Service Deployment Diagram Auto-Deployment Tool Merge Training Environment Model Save Deploy Code Model Repository Deploy Template Wrap Serialized Model Trained Model Serialized Model Code Repository Model Specific Deploy Code/metadata Online Scoring Worker Node Docker Container Deploy Environment Metadata Model Model Specific Docker Image Docker File Build Docker Build Docker Registry Docker Base Image

Example: Online Scoring Service Deployment Diagram Auto-Deployment Tool Merge Training Environment Model Save Deploy Code Model Repository Deploy Template Wrap Serialized Model Trained Model Serialized Model Code Repository Model Specific Deploy Code/metadata Online Scoring Worker Node Docker Container Deploy Environment Metadata Model Model Specific Docker Image Docker File Build Docker Build Docker Registry Docker Base Image

CONTAINERIZATION! Environment in Container Machine Learning Platform Life-Cycle Management CONTAINERIZATION! This is a sample INTRODUCTION using the 1-Column layout. Placeholder text boxes that appear as part of the selected layout have predefined fonts, sizes and colors. To change the appearance of any line of text, on the Home tab, in the Paragraph group, click Increase Indent or Decrease Indent. The selected text will reformat to the predefined size according to its indent level. Note: Any changes to the color, size, spacing or font in the placeholder text box will break its connection to the Master style. It will no longer automatically conform when switching between layouts or when imported into another presentation. Environment in Container

Thank you Hope (Xinwei) Wang Email: xinweiwang3@gmail.com Linkedin: https://www.linkedin.com/in/xinweiwanglinkedin/ Twitter: @HopeXinwei