Open Source Toolkit for Turn-Key AI Cluster (Introduction)

Slides:



Advertisements
Similar presentations
Grid Resource Allocation Management (GRAM) GRAM provides the user to access the grid in order to run, terminate and monitor jobs remotely. The job request.
Advertisements

Microsoft virtual machine converter
Configurations Management System Chris Boyd.  Time consuming task of provisioning a number of systems with STIG compliance  Managing a number of systems.
Building service testbeds on FIRE D5.2.5 Virtual Cluster on Federated Cloud Demonstration Kit August 2012 Version 1.0 Copyright © 2012 CESGA. All rights.
ISG We build general capability Introduction to Olympus Shawn T. Brown, PhD ISG MISSION 2.0 Lead Director of Public Health Applications Pittsburgh Supercomputing.
Support in setting up a non-grid Atlas Tier 3 Doug Benjamin Duke University.
GumTree Development Environment Setup Windows Only Compatible with Eclipse 3.2 M3 (Last update: 16/11/05)
VMWare Workstation Installation. Starting Vmware Workstation Go to the start menu and start the VMware Workstation program. *Note: The following instructions.
ISG We build general capability Introduction to Olympus Shawn T. Brown, PhD ISG MISSION 2.0 Lead Director of Public Health Applications Pittsburgh Supercomputing.
Claudio Grandi INFN Bologna Virtual Pools for Interactive Analysis and Software Development through an Integrated Cloud Environment Claudio Grandi (INFN.
EPAM Cloud Orchestration
Condor Week Apr 30, 2008Pseudo Interactive monitoring - I. Sfiligoi1 Condor Week 2008 Pseudo-interactive monitoring in Condor by Igor Sfiligoi.
Job Scheduling and Runtime in DLWorkspace
Cloud Computing for Science
4/19/ :02 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN.
Open OnDemand: Open Source General Purpose HPC Portal
ONAP/K8S Deployment OOM Team
Here’s an overview of the three-part setup process.
Early Results of Deep Learning on the Stampede2 Supercomputer
AI How to: System Update and Additional Software
Efficient development and deployment of Hydra projects using Vagrant
DL (Deep Learning) Workspace
BigDL Deep Learning Library on HDInsight
Working With Azure Batch AI
Docker Birthday #3.
Open Source Toolkit for Turn-Key AI Cluster (Introduction)
AI development using Data Science Virtual Machines (DSVM) in Azure
Building Applications with LUIS
Platform as a Service.
TensorFlow on Kubernetes with GPU Enabled
Azure CLI Deep Dive Neil Peterson Content Developer Microsoft.
DL (Deep Learning) Workspace
Machine Learning Workshop
MaterialsHub - A hub for computational materials science and tools.
Configuration Management with Azure Automation DSC
DL (Deep Learning) Workspace
Virtualization in the gLite Grid Middleware software process
RMS Architecture EMS Partner Bootcamp TechReady 18 9/17/2018
Service Fabric Patterns & Best Practices
Open Source Toolkit for Turn-Key AI Cluster (Introduction)
Coding in the Cloud This slide deck includes recorded video demonstrations of content from the live presentation. Joon-Yee.
OPNFV Arno Installation & Validation Walk-Through
Enterprise security for big data solutions on Azure HDInsight
Microsoft Bot Framework: changing how we communicate with users
Unit 8 NT1330 Client-Server Networking II Date: 8/2/2016
Using docker containers
High Performance Data Scientist
How To: Deep Learning Workspace
Logsign All-In-One Security Information and Event Management (SIEM) Solution Built on Azure Improves Security & Business Continuity MICROSOFT AZURE APP.
Haiyan Meng and Douglas Thain
Intro to Docker Containers and Orchestration in the Cloud
Data Security for Microsoft Azure
12/5/ :36 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN.
Microsoft Virtual Academy
SAS Deep Learning: From Toolkit to Fast Model Prototyping
Using the Microsoft AI Platform for next generation applications
Microsoft Virtual Academy
Workshop Day 1, 5/2 Machine Learning Studio: Use Azure Machine Learning Studio to develop and deploy ML models, then consume the models using Excel Time.
5 Azure Services Every .NET Developer Needs to Know
LearnAI - Machine Learning on Azure
Simplify the management of your M365 workplace with analytics
H2O is used by more than 14,000 companies
Abel Sanchez, John Williams
IBM C IBM Big Data Engineer. You want to train yourself to do better in exam or you want to test your preparation in either situation Dumpspedia’s.
Deploying Deep Learning Models on GPU Enabled Kubernetes Cluster
A DevOps process for deploying R to production
Deploying machine learning models at scale
Ready Pre-day Azure Monitoring Workshop
Containers on Azure Peter Lasne Sr. Software Development Engineer
Presentation transcript:

Open Source Toolkit for Turn-Key AI Cluster (Introduction) DL Workspace This video introduces DL Workspace, an open source toolkit for turn-key AI Cluster setup and operation. Open Source Toolkit for Turn-Key AI Cluster (Introduction)

DL Workspace is … Open source toolkit for turn-key AI cluster setup Used for daily development/production in Microsoft internal groups (e.g., Microsoft Cognitive Services, SwiftKey, Bing Relevance) Allow AI scientist to run jobs (interactive exploration, training, inferencing, data analytics) Resource managed by cluster Turn-key operation (automatic software setup & cluster configuration) Out-of-box support All major DL toolkits (TensorFlow, CNTK, Caffe, MxNet, etc..) Big data analytics (Hadoop/Spark) DL Workspace provides out-of-box support for multiple Deep Learning toolkits, and big data analytical kits. It is used daily by Microsoft employees, and allows AI scientists to run both interactive and batch jobs on cluster.

Installation on Azure The rest of the video explains the process to install DL Workspace on Azure

OR Prepare Dev Box Installation script for You need a development machine running Ubuntu OS. You may then either install docker, and build DL workspace dev docker, or run installation scripts that will install docker, python and Azure CLI on your machine. Installation script for

prepare configuration file src/ClusterBootstrap/config.yaml Prepare configuration file. You need the name of the cluster, # of infrastructure and worker node, deployed azure location, and type of VM size.

Depending on your Open ID provider, configure the Open ID endpoint, and put the information into configuration file. Authentication for Microsoft corp users have been pre-configured, please contact authors for information.

Run these scripts one-by-one to: Generate Azure VM Generate configuration file for the Azure Cluster Deploy Kubernete Cluster & DL Workspace runtime Log in to your azure account via az login. Then run the following scripts in sequence to create Azure VM, generate configuration file for the Azure Cluster, and deploy kubernete cluster & DL Workspace runtime.

You should have a fully functional cluster in about half an hour. After the scripts run through, please wait a few minutes for the container to start. You should have a fully function cluster.