Open Source Toolkit for Turn-Key AI Cluster (Introduction)

Slides:

Advertisements

Similar presentations

Grid Resource Allocation Management (GRAM) GRAM provides the user to access the grid in order to run, terminate and monitor jobs remotely. The job request.

Advertisements

Microsoft virtual machine converter

Configurations Management System Chris Boyd.  Time consuming task of provisioning a number of systems with STIG compliance  Managing a number of systems.

Building service testbeds on FIRE D5.2.5 Virtual Cluster on Federated Cloud Demonstration Kit August 2012 Version 1.0 Copyright © 2012 CESGA. All rights.

ISG We build general capability Introduction to Olympus Shawn T. Brown, PhD ISG MISSION 2.0 Lead Director of Public Health Applications Pittsburgh Supercomputing.

Support in setting up a non-grid Atlas Tier 3 Doug Benjamin Duke University.

GumTree Development Environment Setup Windows Only Compatible with Eclipse 3.2 M3 (Last update: 16/11/05)

VMWare Workstation Installation. Starting Vmware Workstation Go to the start menu and start the VMware Workstation program. *Note: The following instructions.

ISG We build general capability Introduction to Olympus Shawn T. Brown, PhD ISG MISSION 2.0 Lead Director of Public Health Applications Pittsburgh Supercomputing.

Claudio Grandi INFN Bologna Virtual Pools for Interactive Analysis and Software Development through an Integrated Cloud Environment Claudio Grandi (INFN.

EPAM Cloud Orchestration

Condor Week Apr 30, 2008Pseudo Interactive monitoring - I. Sfiligoi1 Condor Week 2008 Pseudo-interactive monitoring in Condor by Igor Sfiligoi.

Job Scheduling and Runtime in DLWorkspace

Cloud Computing for Science

4/19/ :02 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN.

Open OnDemand: Open Source General Purpose HPC Portal

ONAP/K8S Deployment OOM Team

Here’s an overview of the three-part setup process.

Early Results of Deep Learning on the Stampede2 Supercomputer

AI How to: System Update and Additional Software

Efficient development and deployment of Hydra projects using Vagrant

DL (Deep Learning) Workspace

BigDL Deep Learning Library on HDInsight

Working With Azure Batch AI

Docker Birthday #3.

Open Source Toolkit for Turn-Key AI Cluster (Introduction)

AI development using Data Science Virtual Machines (DSVM) in Azure

Building Applications with LUIS

Platform as a Service.

TensorFlow on Kubernetes with GPU Enabled

Azure CLI Deep Dive Neil Peterson Content Developer Microsoft.

DL (Deep Learning) Workspace

Machine Learning Workshop

MaterialsHub - A hub for computational materials science and tools.

Configuration Management with Azure Automation DSC

DL (Deep Learning) Workspace

Virtualization in the gLite Grid Middleware software process

RMS Architecture EMS Partner Bootcamp TechReady 18 9/17/2018

Service Fabric Patterns & Best Practices

Open Source Toolkit for Turn-Key AI Cluster (Introduction)

Coding in the Cloud This slide deck includes recorded video demonstrations of content from the live presentation. Joon-Yee.

OPNFV Arno Installation & Validation Walk-Through

Enterprise security for big data solutions on Azure HDInsight

Microsoft Bot Framework: changing how we communicate with users

Unit 8 NT1330 Client-Server Networking II Date: 8/2/2016

Using docker containers

High Performance Data Scientist

How To: Deep Learning Workspace

Logsign All-In-One Security Information and Event Management (SIEM) Solution Built on Azure Improves Security & Business Continuity MICROSOFT AZURE APP.

Haiyan Meng and Douglas Thain

Intro to Docker Containers and Orchestration in the Cloud

Data Security for Microsoft Azure

12/5/ :36 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN.

Microsoft Virtual Academy

SAS Deep Learning: From Toolkit to Fast Model Prototyping

Using the Microsoft AI Platform for next generation applications

Microsoft Virtual Academy

Workshop Day 1, 5/2 Machine Learning Studio: Use Azure Machine Learning Studio to develop and deploy ML models, then consume the models using Excel Time.

5 Azure Services Every .NET Developer Needs to Know

LearnAI - Machine Learning on Azure

Simplify the management of your M365 workplace with analytics

H2O is used by more than 14,000 companies

Abel Sanchez, John Williams

IBM C IBM Big Data Engineer. You want to train yourself to do better in exam or you want to test your preparation in either situation Dumpspedia’s.

Deploying Deep Learning Models on GPU Enabled Kubernetes Cluster

A DevOps process for deploying R to production

Deploying machine learning models at scale

Ready Pre-day Azure Monitoring Workshop

Containers on Azure Peter Lasne Sr. Software Development Engineer

Presentation transcript:

Open Source Toolkit for Turn-Key AI Cluster (Introduction) DL Workspace This video introduces DL Workspace, an open source toolkit for turn-key AI Cluster setup and operation. Open Source Toolkit for Turn-Key AI Cluster (Introduction)

DL Workspace is … Open source toolkit for turn-key AI cluster setup Used for daily development/production in Microsoft internal groups (e.g., Microsoft Cognitive Services, SwiftKey, Bing Relevance) Allow AI scientist to run jobs (interactive exploration, training, inferencing, data analytics) Resource managed by cluster Turn-key operation (automatic software setup & cluster configuration) Out-of-box support All major DL toolkits (TensorFlow, CNTK, Caffe, MxNet, etc..) Big data analytics (Hadoop/Spark) DL Workspace provides out-of-box support for multiple Deep Learning toolkits, and big data analytical kits. It is used daily by Microsoft employees, and allows AI scientists to run both interactive and batch jobs on cluster.

Installation on Azure The rest of the video explains the process to install DL Workspace on Azure

OR Prepare Dev Box Installation script for You need a development machine running Ubuntu OS. You may then either install docker, and build DL workspace dev docker, or run installation scripts that will install docker, python and Azure CLI on your machine. Installation script for

prepare configuration file src/ClusterBootstrap/config.yaml Prepare configuration file. You need the name of the cluster, # of infrastructure and worker node, deployed azure location, and type of VM size.

Depending on your Open ID provider, configure the Open ID endpoint, and put the information into configuration file. Authentication for Microsoft corp users have been pre-configured, please contact authors for information.

Run these scripts one-by-one to: Generate Azure VM Generate configuration file for the Azure Cluster Deploy Kubernete Cluster & DL Workspace runtime Log in to your azure account via az login. Then run the following scripts in sequence to create Azure VM, generate configuration file for the Azure Cluster, and deploy kubernete cluster & DL Workspace runtime.

You should have a fully functional cluster in about half an hour. After the scripts run through, please wait a few minutes for the container to start. You should have a fully function cluster.