WRF in the Cloud: An introduction to Big Compute on Windows Azure Wenming Ye Research Program Manager Microsoft Research

What is Cloud Computing?

Software-as-a-Service SaaS Platform-as-a-Service Stateless, easy to scale, manage PaaS Infrastructure-as-a-Service with Persistent remote disks IaaS

Cloud servicesWeb sitesVirtual machines

DELL Compute/Storage Container Transformer & UPS Air Flow Vents

Inside a DELL Container Compute Rack 2 “Side by Side” Servers Disk Sled Rack On the truck At DELL’s Austin Plant

DLA Architecture (Old)Quantum10 Architecture (New) TOR Spine … … DCR BL Spine DC Routers BL 30,000 Gbps 120 Gbs

Internet of things Audio / Video Log Files Text/Image Social Sentiment Data Market Feeds eGov Feeds Weather Wikis / Blogs Click Stream Sensors / RFID / Devices Spatial & GPS Coordinates WEB 2.0 Mobile Advertisin g CollaborationeCommerce Digital Marketing Search Marketing Web Logs Recommendation s ERP / CRM Sales Pipeline Payables Payroll Inventory Contacts Deal Tracking Terabytes (10E12) Gigabytes (10E9) Exabytes (10E18) Petabytes (10E15) Velocity - Variety - variability Volume ,000$ $ ,000$ $ Storage/GB ERP / CRM WEB 2.0 Internet of things

Scenarios

Design Optimization Financial and insurance risk calculation Engineering modeling and simulation Computational life sciences Earth sciences Data analytics

Invisible devices Trillions of networked nodes Low bandwidth last- mile connection Mostly addressed by local schemes Machine-centricSensing-focus Global addressingUser-centric Communication- focus Laptops / tablets / smartphones Billions of networked devices High-bandwidth access

How do I optimize my services based on patterns of weather, traffic. How do I build a recommendation engine? What’s the social sentiment of my product? How do I better predict future outcomes?

Multiple/Any languages (Fortran too) 

en-us/develop/nodejs/how-to- guides/command-line-tools/

Distributed Storage (HDFS) Query (Hive) Distributed Processing (MapReduce) HDINSIGHT / HADOOP Eco-System Legend Red = Core Hadoop Blue = Data processing Purple = Microsoft integration points and value adds Orange = Data Movement Green = Packages

Front end Stream Layer Partition Layer Name Node de Data Node Front end HDFS API DFS (1 Data Node per Worker Role) and Compute Cluster Azure Storage (ASV) … Azure Blob Storage

A few Demos

Demo: Weather Forecast as a Service

Azure Table storage Front end Stream Layer Partition Layer Azure Storage Front end

HeatMap: 5 Compute Nodes running at 100%

Geogrid: Defines the model horizontal domain Horizontally interpolates static data to the model domain Output conforms to the WRF I/O API Ungrib: Decodes Grib Edition 1 and 2 data (Grib is the file format of the NOAA forecast data) Uses tables to decide which variables to extract from the data Supports isobaric and generalized vertical coordinates Output is in a non-WRF-I/O-API form, referred to as an intermediate format Metgrid: Ingest static data and raw meteorological fields Horizontally interpolate meteorological fields to the model domain Output conforms to WRF I/O API

Gdal_translate: Converts file from png to geospatial Tiff (geoTiff) format Assigns geo-referencing (“ground control points”) to specified pixel values Labels the output (geoTiff) file’s header with map projection information Gdalwarp: Geographically reprojects the output file from gdal_translate to the map projection of Bing Maps Gdal2tiles.py: A python script that generates tiles from the output file from gdalwarp Zoom levels are specified on the command line

Demo: Interactive computing and data analysis with IPython NotebookIPython Notebook

PaaS IaaS

Two MODIS satellites Terra, launched 12/1999 Aqua, launched 05/2002 Near polar orbits Global coverage two days Sensitive in 36 spectral bands

© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION. © 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.