Scientific Computing on Amazon Web Services Dave Cuthbert Solutions Architect

Slides:

Advertisements

Similar presentations

Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009.

Advertisements

Amazon Web Services Justin DeBrabant CIS Advanced Systems - Fall 2013.

EHarmony in Cloud Subtitle Brian Ko. eHarmony Online subscription-based matchmaking service Available in United States, Canada, Australia and United Kingdom.

Amazon Web Services and Eucalyptus

Rhea Analysis & Post-processing Cluster Robert D. French NCCS User Assistance.

Using ArcGIS for Server in the Amazon Cloud

Cloud Computing Resource provisioning Keke Chen. Outline  For Web applications statistical Learning and automatic control for datacenters  For data.

Infrastructure as a Service (IaaS) Amazon EC2

Marihebert Leal. Alteryx is the fastest analytics plataform that is purpose- built to empower data analysts & their productivity. It blend complex data,

RCAC Research Computing Presents: DiaGird Overview Tuesday, September 24, 2013.

© 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. Developing on AWS © 2014 Amazon Web Services, Inc. and its affiliates. All rights.

Virtual Geophysics Laboratory (VGL) VGL v1.2 NeCTAR Project Close R.Fraser, T.Rankine, J.Vote, L.Wyborn, B.Evans, R.Woodcock, C.Kemp July 2013 CSIRO |

Cloud Computing using AWS C. Edward Chow. Advanced Internet & Web Systems chow2 Outline of the Talk Introduction to Cloud Computing AWS EC2 EC2 API A.

A Brief Overview by Aditya Dutt March 18 th ’ Aditya Inc.

TeraGrid Gateway User Concept – Supporting Users V. E. Lynch, M. L. Chen, J. W. Cobb, J. A. Kohl, S. D. Miller, S. S. Vazhkudai Oak Ridge National Laboratory.

Experiences with AWS and RightScale By: Max Gribov Presented at New York PHP, March 22, 2011

Jamie Kinney, AWS Scientific Computing

Research Business Technology Pfizer Enterprise Elastic HPC Mike Miller Pfizer Research Business Technology May 18 th Prism Meeting Stockholm Sweden.

MaterialsHub - A hub for computational materials science and tools.  MaterialsHub aims to provide an online platform for computational materials science.

AWS Simple Icons v15.9 AWS Simple Icons: Usage Guidelines Check to make sure you have the most recent set of AWS Simple Icons This version was last updated.

Describe workflows used to maintain and provide the RDA to users – Both are 24x7 operations Transition to the NWSC with zero downtime NWSC is new environment.

AWS Elastic Beanstalk and Docker: High Fidelity, High Velocity Deployments in the Cloud Evan Senior Developer Advocate, AWS.

Sponsored by the National Science Foundation University of Massachusetts Amherst November 2 nd, 2011 GENI DiCloud.

Template This is a template to help, not constrain, you. Modify as appropriate. Move bullet points to additional slides as needed. Don’t cram onto a single.

TeraGrid Gateway User Concept – Supporting Users V. E. Lynch, M. L. Chen, J. W. Cobb, J. A. Kohl, S. D. Miller, S. S. Vazhkudai Oak Ridge National Laboratory.

AN ORGANISATION FOR A NATIONAL EARTH SCIENCE INFRASTRUCTURE PROGRAM Virtual Geophysics Laboratory (VGL): Scientific workflows Exploiting the Cloud Josh.

Cloud Computing Paradigms for Pleasingly Parallel Biomedical Applications Thilina Gunarathne, Tak-Lon Wu Judy Qiu, Geoffrey Fox School of Informatics,

Template This is a template to help, not constrain, you. Modify as appropriate. Move bullet points to additional slides as needed. Don’t cram onto a single.

ISG We build general capability Introduction to Olympus Shawn T. Brown, PhD ISG MISSION 2.0 Lead Director of Public Health Applications Pittsburgh Supercomputing.

Computing Issues for the ATLAS SWT2. What is SWT2? SWT2 is the U.S. ATLAS Southwestern Tier 2 Consortium UTA is lead institution, along with University.

What’s Coming? What are we Planning?. › Better docs › Goldilocks – This slot size is just right › Storage › New.

Homework 4 Responses Most people really did make this assignment an illusion – not good –tempted to now cancel this class and just give everyone an A The.

Event Service Wen Guan University of Wisconsin 1.

Large-scale accelerator simulations: Synergia on the Grid turn 1 turn 27 turn 19 turn 16 C++ Synergia Field solver (FFT, multigrid) Field solver (FFT,

100% Exam Passing Guarantee & Money Back Assurance

Terraform at Adobe Kelvin Jasperson. Introduction 2 Systems Adobe Audience Manager (AAM) Been with Adobe for 18 months AAM was acquired by.

GETTING STARTED WITH AWS AND PYTHON. OUTLINE  Intro to Boto  Installation and configuration  Working with AWS S3 using Bot  Working with AWS SQS using.

100% Exam Passing Guarantee & Money Back Assurance

Introduction to Data Analysis with R on HPC Texas Advanced Computing Center Feb

Amazon Web Services. Amazon Web Services (AWS) - robust, scalable and affordable infrastructure for cloud computing. This session is about:

INTRODUCTION TO AMAZON WEB SERVICES (EC2). AMAZON WEB SERVICES  Services  Storage (Glacier, S3)  Compute (Elastic Compute Cloud, EC2)  Databases (Redshift,

S3 Lifecycle Policies to Glacier

Mastering Spark Data Masters. Special Thanks To…

Petr Škoda, Jakub Koza Astronomical Institute Academy of Sciences

High Performance Computing (HPC)

AWS Simple Icons v AWS Simple Icons: Usage Guidelines

Security Group Amazon RDS Mysql Media Request S3

Open OnDemand: Open Source General Purpose HPC Portal

What is HPC? High Performance Computing (HPC)

Data Platform and Analytics Foundational Training

Hydrodynamic Galactic Simulations

Geoffrey Fox, Shantenu Jha, Dan Katz, Judy Qiu, Jon Weissman

Provisioning 160,000 cores with HEPCloud at SC17

Joker: Getting the most out of the slurm scheduler

MaterialsHub - A hub for computational materials science and tools.

AWS COURSE DEMO BY PROFESSIONAL-GURU. Amazon History Ladder & Offering.

2018 Amazon AWS DevOps Engineer Professional Dumps - DumpsProfessor

Buy September 2018 Valid Amazon AWS-SysOps Dumps Questions - Amazon AWS-SysOps Braindumps Realexamdumps.com

AWS Administrator overview  SV Trainings AWS Training –provides real time and placement oriented Amazon Web Services (AWS) Online Training. Our AWS Course.

Homework 4 Responses There was no expectation of completeness assignment – this was meant to give you the opportunities to a) start and manage a github.

CCR Advanced Seminar: Running CPLEX Computations on the ISE Cluster

Advanced Computing Facility Introduction

CS110: Discussion about Spark

What’s Different About Overlay Systems?

Overview of big data tools

Deploying Your First Full Stack Application to the Cloud

Tutorial 1: Python, Numpy, and AWS Tutorial

Machine Learning for Cyber

Introduction to research computing using Condor

Presentation transcript:

Scientific Computing on Amazon Web Services Dave Cuthbert Solutions Architect

Two Facets (That I’ll Mention Today) Facet 1: Availability of scientific applications General purpose analysis Python (SciPy, NumPy, iPython notebooks). Octave, R, … C, C++, Fortran, … Databases/data formats NetCDF, HDF, … Cassandra, MongoDB, CouchDB, Redis, Berkeley DB, … MySQL/MariaDB, PostgreSQL, … Commercial Applications are widely available. Licensing can be thorny.

Two Facets (That I’ll Mention Today) Facet 2: Cycles What everyone thinks: HPC. Mental trap 1: It’s not “real” science if it’s not running on an HPC cluster. Mental trap 2: If your lab has an HPC cluster, you should be coding for it. So everyone demands cluster time, and…

A Typical HPC Cluster Workload

But What Is HPC, Anyway? If I wanted to start a flame war: “What is ‘real’ HPC?”

HPC Is Not A Panacea! Hadoop GPU Low Latency Hadoop Low Latency GPU

It’s A Trap! Facet 2: Cycles What everyone thinks: HPC. Mental trap 1: It’s not “real” science if it’s not running on an HPC cluster. Mental trap 2: If your lab has an HPC cluster, you should be coding for it. The right systems for the job.

HOW AWS IS ATTACKING THE PROBLEM

AmazonLinux with SLURM AMI Availability Zone us-west-2a Region: us-west-2 (Oregon) controller VPC Space: /16 Subnet /24 Availability Zone us-west-2b Subnet /24 Availability Zone us-west-2c Subnet /24 node-0 node-1 node-2 node-3 node-4 node-5 node-6 node-7 node-8 node-9 node-10 node-11 VBL S3 Bucket Scripts Code Input Decks Output Files CloudFormation Template Internet gateway Work Request Queue Work Response Queue SQS Queues CloudFormation (Bootstrap controller)

AmazonLinux with SLURM AMI Availability Zone us-west-2a Region: us-west-2 (Oregon) controller VPC Space: /16 Subnet /24 Availability Zone us-west-2b Subnet /24 Availability Zone us-west-2c Subnet /24 node-0 node-1 node-2 node-3 node-4 node-5 node-6 node-7 node-8 node-9 node-10 node-11 VBL S3 Bucket Scripts Code Input Decks Output Files CloudFormation Template Internet gateway Work Request Queue Work Response Queue SQS Queues CloudFormation (Bootstrap controller)

AmazonLinux with SLURM AMI Availability Zone us-west-2a Region: us-west-2 (Oregon) controller VPC Space: /16 Subnet /24 Availability Zone us-west-2b Subnet /24 Availability Zone us-west-2c Subnet /24 node-0 node-1 node-2 node-3 node-4 node-5 node-6 node-7 node-8 node-9 node-10 node-11 VBL S3 Bucket Scripts Code Input Decks Output Files CloudFormation Template Internet gateway Work Request Queue Work Response Queue SQS Queues CloudFormation (Bootstrap controller)

AmazonLinux with SLURM AMI Availability Zone us-west-2a Region: us-west-2 (Oregon) controller VPC Space: /16 Subnet /24 Availability Zone us-west-2b Subnet /24 Availability Zone us-west-2c Subnet /24 node-0 node-1 node-2 node-3 node-4 node-5 node-6 node-7 node-8 node-9 node-10 node-11 VBL S3 Bucket Scripts Code Input Decks Output Files CloudFormation Template Internet gateway Work Request Queue Work Response Queue SQS Queues CloudFormation (Bootstrap controller) min229 µs p50239 µs p90258 µs p99280 µs max472 µs min229 µs p50239 µs p90258 µs p99280 µs max472 µs min329 µs p50340 µs p90354 µs p99377 µs max611 µs min329 µs p50340 µs p90354 µs p99377 µs max611 µs min1048 µs p µs p µs p µs max2125 µs min1048 µs p µs p µs p µs max2125 µs

AmazonLinux with SLURM AMI Availability Zone us-west-2a Region: us-west-2 (Oregon) controller VPC Space: /16 Subnet /24 Availability Zone us-west-2b Subnet /24 Availability Zone us-west-2c Subnet /24 node-0 node-1 node-2 node-3 node-4 node-5 node-6 node-7 node-8 node-9 node-10 node-11 VBL S3 Bucket Scripts Code Input Decks Output Files CloudFormation Template Internet gateway Work Request Queue Work Response Queue SQS Queues CloudFormation (Bootstrap controller)

AmazonLinux with SLURM AMI Availability Zone us-west-2a Region: us-west-2 (Oregon) controller VPC Space: /16 Subnet /24 node-0node-1node-2node-3node-4node-5node-6node-7node-8node-9node-10node-11 VBL S3 Bucket Scripts Code Input Decks Output Files CloudFormation Template Internet gateway Work Request Queue Work Response Queue SQS Queues CloudFormation (Bootstrap controller) Placement Group A min85 µs p5096 µs p90106 µs p99189 µs max233 µs min85 µs p5096 µs p90106 µs p99189 µs max233 µs min87 µs p5099 µs p90174 µs p99189 µs max246 µs min87 µs p5099 µs p90174 µs p99189 µs max246 µs

Is AWS The Silver Bullet? No silver bullets – Fred Brooks Commonly heard latency number: 10 µs Proximity to other resources might be an issue. People-hours are more expensive than core- hours. Enable facilities like NERSC to focus on harder problems not served (or currently served) by COTS.

THANK YOU!