Sponsored by the National Science Foundation University of Massachusetts Amherst November 2 nd, 2011 GENI DiCloud

Sponsored by the National Science Foundation 2 Project Summary DiCloud enables GENI experiments to incorporate cloud computing resources –Focuses on Amazon web services –EC2 (VMs), EBS (block storage), S3 (object storage) How are cloud resources different from other GENI resources? –They cost money! Requires active Amazon account and credit card –Require authentication with Amazon

Sponsored by the National Science Foundation 3 Project Summary DiCloud tracks per GENI-user Amazon costs –Amazon charges can be complicated Pay per VM time used Pay for data transfer in and out Pay for IOs to block devices –No current facilities for tracking per user costs Enables controlled access to cloud resources –Stand-alone: setup your own DiCloud instance to track your own costs –GENI CF-mode: place “underneath” GENI CF, where AM pays the bills, but gives GENI users access

Sponsored by the National Science Foundation 4 Elastic Compute Cloud (EC2) Rent VMs from Amazon “on-demand” –

Sponsored by the National Science Foundation 5 Elastic Compute Cloud (EC2) Rent VMs from Amazon “on-demand” –Pay per hour based on VM performance –Pay for data transfer in/out

Sponsored by the National Science Foundation 6 Elastic Block Store (EBS) Rent block storage from Amazon –Pay per GB-month (currently $0.10) –Also pay per 1 million IO requests (currently $0.10)

Sponsored by the National Science Foundation 7 Simple Storage Service (S3) Rent object storage from Amazon –Pay per GB-month –Pay for data transfer in/out

Sponsored by the National Science Foundation 8 Web Portal

Sponsored by the National Science Foundation 9 Web Portal

Sponsored by the National Science Foundation 10 Web Portal

Sponsored by the National Science Foundation 11 Tutorial Step 5: Web Portal

Sponsored by the National Science Foundation 12 Tutorial Step 5: Web Portal Now logout and log back in as user “geni” –Then click on “Request EC2 resources” –Use ami-013d6c44 as the AMI id

Sponsored by the National Science Foundation 13 Tutorial Step 5: Web Portal Now go back to your user page –Click on status next to the VM you just created

Sponsored by the National Science Foundation 14 Tutorial Step 5: Web Portal Now lets create an S3 bucket –Click on “Request new S3 buckets”

Sponsored by the National Science Foundation 15 Tutorial Step 5: Web Portal Once back at the user page, put something in your bucket –Select the s3_test file in ~/Tutorials/DiCloud/s3_test

Sponsored by the National Science Foundation 16 Tutorial Step 5: Web Portal Now get that same thing back from your bucket

Sponsored by the National Science Foundation 17 Tutorial Step 5: Web Portal Now lets create an EBS volume and attach it to our VM –Click request new EBS volumes –Make it 1GB, in us-west-1, in us-west-1

Sponsored by the National Science Foundation 18 Tutorial Step 5: Web Portal Attach the volume to our VM –Click on attach, enter instance id, and sdg for device

Sponsored by the National Science Foundation 19 Tutorial Step 5: Web Portal If we ssh into our VM we can see that the volume is now attached

Sponsored by the National Science Foundation 20 Demo Data Flow Dynamic end-to-end Nowcasting –Mapping Nowcast Workflows onto GENI Archival Storage Radar Nodes “raw” live data Upstream LDM feed archived netcdf data archived netcdf data Nowcast Processing aggregated multi-radar data aggregated multi-radar data Post to Web Nowcast images for display

Sponsored by the National Science Foundation 21 Multi-radar NetCDF Data Nowcast Processing 1.DiCloud Archival Service (S3) 2.LDM Data Feed (EC2) 1.DiCloud Archival Service (S3) 2.LDM Data Feed (EC2) “raw” live data Generate “raw” live data ViSE/CASA radar nodes Generate “raw” live data ViSE/CASA radar nodes Use proxy to track usage-based spending on Amazon and enforce quotas and limits Use proxy to track usage-based spending on Amazon and enforce quotas and limits 1.Ingest mulit-radar data feeds 2.Merge and grid multi-radar data 2.Generate 1min, 5min, and 10min Nowcasts 3.Send results over NLR to Umass 4.Repeat 1.Ingest mulit-radar data feeds 2.Merge and grid multi-radar data 2.Generate 1min, 5min, and 10min Nowcasts 3.Send results over NLR to Umass 4.Repeat ViSE views steerable radars as shared, virtualized resources ViSE views steerable radars as shared, virtualized resources Nowcast images for display Nowcast images for display

Sponsored by the National Science Foundation 22 Demo Overview Dynamic end-to-end Nowcasting on GENI –Slice of sensing, networking, computing, and storage Archival Storage Radar Nodes Upstream LDM feed Nowcast Processing Post to Web Generate “raw” data ViSE/CASA radars Archive radar data Amazon S3 “raw” live data Archived data available to downstream nodes archived netcdf data archived netcdf data aggregated multi-radar data aggregated multi-radar data Nowcast images for display Generate Nowcasts