Sponsored by the National Science Foundation GEC 12 DiCloud Tutorial University of Massachusetts Amherst November 2 nd, 2011 GEC12 DiCloud Tutorial

Sponsored by the National Science Foundation 2 Project Summary DiCloud enables GENI experiments to incorporate cloud computing resources –Focuses on Amazon web services –EC2 (VMs), EBS (block storage), S3 (object storage) How are cloud resources different from other GENI resources? –They cost money! Requires active Amazon account and credit card –Require authentication with Amazon

Sponsored by the National Science Foundation 3 Project Summary DiCloud tracks per GENI-user Amazon costs –Amazon charges can be complicated Pay per VM time used Pay for data transfer in and out Pay for IOs to block devices –No current facilities for tracking per user costs Enables controlled access to cloud resources –Stand-alone: setup your own DiCloud instance to track your own costs –GENI CF-mode: place “underneath” GENI CF, where AM pays the bills, but gives GENI users access

Sponsored by the National Science Foundation 4 Elastic Compute Cloud (EC2) Rent VMs from Amazon “on-demand” –

Sponsored by the National Science Foundation 5 Elastic Compute Cloud (EC2) Rent VMs from Amazon “on-demand” –Pay per hour based on VM performance –Pay for data transfer in/out

Sponsored by the National Science Foundation 6 Elastic Block Store (EBS) Rent block storage from Amazon –Pay per GB-month (currently $0.10) –Also pay per 1 million IO requests (currently $0.10)

Sponsored by the National Science Foundation 7 Simple Storage Service (S3) Rent object storage from Amazon –Pay per GB-month –Pay for data transfer in/out

Sponsored by the National Science Foundation 8 Tutorial Overview Setup DiCloud server for stand-alone mode Access DiCloud through console –Learn how to create VMs, block devices, storage objects –Learn how to check our balance and set quotas Access DiCloud through web portal –Learn how to create users and edit quotas Learn how to give users access to cloud resources –Learn how to console actions through web portal Time permitting –Use your Amazon resources in a simple GENI experiment using Gush

Sponsored by the National Science Foundation 9 Tutorial Step 1: Let’s get started! Trac page with both setup and tutorial instructions – lhttp://groups.geni.net/geni/wiki/DICLOUD/GEC12tutoria l –Setup instructions done for you –Tutorial instructions step-by-step in slides –Download GEC12tutorial.txt file with commands Pre-requisites –Download/start your Virtual Box image Login as user: geni –Open a Firefox web browser –Open 3 terminal windows and go to ~/Tutorials/DiCloud $ cd ~/Tutorials/DiCloud

Sponsored by the National Science Foundation 10 Tutorial Step 1a: Common Errors Be careful cutting and pasting –Doesn’t work from Trac! –Sometimes even txt file inserts hidden characters –Some lines require edits; don’t cut and paste the newline Make terminal windows big –Sometimes terminal gets corrupted when pasting a line that runs to the next line (especially when backspacing) On the website –Remember to hit down before hitting enter if the browser auto-fills in a text box (from its history)

Sponsored by the National Science Foundation 11 Tutorial Step 2: Getting your Keys We’ve created temporary keys for you to access Amazon using a UMass account –Run the following commands: $ cd ~/Tutorials/DiCloud $./key_install.sh geni.cs.umass.edu –Verify Success: ls -la ec2/keys/ total 40 drwxr-xr-x 2 geni geni :30. drwxr-xr-x 6 geni geni :21.. -rw geni geni :26 credentials.csv -rw-r--r-- 1 geni geni :29 geni_aws_credentials.txt -rw-r--r-- 1 geni geni :26 geni_cert.pem -rw-r--r-- 1 www-data www-data :30 geni_cert_web.pem -rw geni geni :26 geni-keypair.pem -rw-r--r-- 1 geni geni :26 geni-keypair.pub -rw-r--r-- 1 geni geni :26 geni_pk.pem -rw-r--r-- 1 www-data www-data :30 geni_pk_web.pem

Sponsored by the National Science Foundation 12 Tutorial Step 3: DiCloud Server Run following command to start DiCloud server: –$ cd ~/Tutorials/DiCloud –$ source./environment –$ cd dicloud/bin –$./dicloud_server.sh –Verify Success 11:20:31,376 INFO root Reading properties from dicloud.properties 11:20:31,466 INFO root Starting RMI registry on port :20:31,474 INFO root Registering DiCloud server 11:20:33,253 INFO root Registering AWS accouting service 11:20:33,260 INFO root Registering EC2 handler 11:20:33,275 INFO root Registering S3 handler 11:20:33,289 INFO root Registering EBS handler 11:20:33,302 INFO root Waiting for console connections Move terminal to background but keep DiCloud server running

Sponsored by the National Science Foundation 13 Tutorial Step 3: DiCloud Server Switch to another terminal –$ cd ~/Tutorials/DiCloud –$ source./environment –$ cd dicloud/bin –$ cat ec2pricing.properties cloudwatch= network.in.per.gb=-.10 network.out.per.gb=-.15 s3.storage.per.gb=-.15 s3.put= s3.get= ebs.storage.per.gb=-.10 ebs.io.per.million=-.10 t1.micro,us-west-1=-.01 m1.small,us-west-1=-.042 m1.large,us-west-1=-.16 m1.xlarge,us-west-1=-.325 m2.xlarge,us-west-1=-.23 m2.2xlarge,us-west-1=-.571 ……..

Sponsored by the National Science Foundation 14 Tutorial Step 4a: DiCloud Console Now open a new terminal and start console: –$ cd ~/Tutorials/DiCloud –$ source./environment –$./dicloud_console.sh –Verify Success dicloud - not connected> Now we are in the DiCloud console –Type “help” to see a list of the possible console commands

Sponsored by the National Science Foundation 15 Tutorial Step 4b: DiCloud Console Now connect to server and start creating resources –dicloud - not connected> connect localhost 1099 Connecting to //localhost:1099/dicloud/server dicloud - //localhost:1099> –dicloud - //localhost:1099> get balance Current balance is: $0.0 –dicloud - //localhost:1099> add credit 100 Current balance is: $0.0 New balance is: $100.0 –dicloud - //localhost:1099> get balance Current balance is: $100.0 Next we are going to create an S3 storage object, an EC2 VM, and an EBS storage volume

Sponsored by the National Science Foundation 16 Tutorial Step 4c: S3 objects must be random (S3 namespace globally shared) Create S3 storage object –dicloud - //localhost:1099> s3 create gec12-tutorial- us –dicloud - //localhost:1099> s3 put gec12-tutorial- test /home/geni/Tutorials/DiCloud/s3_test Uploading file /home/geni/Tutorials/DiCloud/s3_test as object test in bucket gec12-tutorial- –dicloud - //localhost:1099> s3 get gec12-tutorial- test /home/geni/Tutorials/DiCloud/test Downloading file /home/geni/Tutorials/DiCloud/test from object test in bucket gec12-tutorial- –dicloud - //localhost:1099> s3 delete gec12-tutorial- us –dicloud - //localhost:1099>get balance Current balance is: $

Sponsored by the National Science Foundation 17 Tutorial Step 4d: EC2 VMs Create EC2 VM –dicloud - //localhost:1099> ec2 start /geni_cert.pem /geni_pk.pem ami- 013d6c44 1 m1.small us-west-1 geni-keypair i-970bb3d0 –dicloud - //localhost:1099> ec2 probe /geni_cert.pem /geni_pk.pem i- 970bb3d0 us-west-1 ascii Instance Id: i-970bb3d0AMI Id: ami-013d6c44 Public name: ec us-west-1.compute.amazonaws.com Private name: ip us-west-1.compute.internal Public IP: Private IP: ebs Instance State: runningInstance Type: m1.small Key pair: geni-keypair Start time: T18:02: Availability zone: us-west-1a

Sponsored by the National Science Foundation 18 Tutorial Step 4e: EC2 VMs Logging into your EC2 VM –Open another terminal window –$ ssh –i /geni-keypair.pem 1.compute.amazonaws.com __| __|_ ) _| ( / Amazon Linux AMI ___|\___|___| See /usr/share/doc/system-release/ for latest release notes. No packages needed for security; 10 packages available ~]$ Next lets create an EBS storage volume and attach it to our VM

Sponsored by the National Science Foundation 19 Tutorial Step 4f: EBS storage volume Create EBS storage volume –dicloud - //localhost:1099> ebs create /geni_cert.pem /geni_pk.pem 1 us-west-1 us-west-1 vol-7f2ca91e –dicloud - //localhost:1099> ebs attach /geni_cert.pem /geni_pk.pem us- west-1 vol-7f2ca91e i-970bb3d0 sdg attaching Switch back to the EC2 VM’s terminal ~]$ ls /dev/sdg /dev/sdg

Sponsored by the National Science Foundation 20 Tutorial Step 4g: Console Scripting DiCloud’s console is scriptable –$ cd ~/Tutorials/DiCloud/dicloud/bin –$ nano getbalance.txt connect localhost 1099 get balance quit –$./dicloud_console_script.sh < getbalance.txt dicloud - not connected>Connecting to //localhost:1099/dicloud/server dicloud - //localhost:1099>Current balance is: $0.0 dicloud - //localhost:1099> Also able to hook into DiCloud programmatically using Java

Sponsored by the National Science Foundation 21 Tutorial Step 4h: Clean-up Detach and destroy EBS storage volume –dicloud - //localhost:1099> ebs detach /geni_cert.pem /geni_pk.pem us- west-1 vol-7f2ca91e detaching –dicloud - //localhost:1099> ebs delete /geni_cert.pem /geni_pk.pem vol-7f2ca91e us-west-1 Terminate EC2 VM instance –dicloud - //localhost:1099> ec2 stop /geni_cert.pem /geni_pk.pem i- 970bb3d0 us-west-1 geni-keypair –dicloud - //localhost:1099> get balance Current balance is: $

Sponsored by the National Science Foundation 22 Tutorial Step 5: Web Portal Open Firefox web browser –Go to –Login as user:admin, pw:geni First, lets create a new non-admin user

Sponsored by the National Science Foundation 23 Tutorial Step 5: Web Portal Give the new user the name geni with pw geniuser –Change access rights to view billing history

Sponsored by the National Science Foundation 24 Tutorial Step 5: Web Portal Now click on “change” under “Certificates” for user geni Enter file names and click “Update user certificates” –cert file: /geni_cert_web.pem –key file: /geni_pk_web.pem –keypair name: geni-keypair

Sponsored by the National Science Foundation 25 Tutorial Step 5: Web Portal Now logout and log back in as user “geni” –Then click on “Request EC2 resources” –Use ami-013d6c44 as the AMI id

Sponsored by the National Science Foundation 26 Tutorial Step 5: Web Portal Now logout and log back in as user “geni” –Then click on “Request EC2 resources” –Use ami-013d6c44 as the AMI id

Sponsored by the National Science Foundation 27 Tutorial Step 5: Web Portal Now go back to your user page –Click on status next to the VM you just created

Sponsored by the National Science Foundation 28 Tutorial Step 5: Web Portal Now lets create an S3 bucket –Click on “Request new S3 buckets”

Sponsored by the National Science Foundation 29 Tutorial Step 5: Web Portal Once back at the user page, put something in your bucket –Select the s3_test file in ~/Tutorials/DiCloud/s3_test

Sponsored by the National Science Foundation 30 Tutorial Step 5: Web Portal Now get that same thing back from your bucket

Sponsored by the National Science Foundation 31 Tutorial Step 5: Web Portal Now lets create an EBS volume and attach it to our VM –Click request new EBS volumes –Make it 1GB, in us-west-1, in us-west-1

Sponsored by the National Science Foundation 32 Tutorial Step 5: Web Portal Attach the volume to our VM –Click on attach, enter instance id, and sdg for device

Sponsored by the National Science Foundation 33 Tutorial Step 5: Web Portal If we ssh into our VM we can see that the volume is now attached

Sponsored by the National Science Foundation 34 Tutorial Step 5: Web Portal Cleanup EBS –Detach and delete EBS volume S3 –Delete S3 storage bucket For EC2….. –Lets create 1 more VM, so we can test our Gush experiments across 2 VMs –Take note of the IP addresses for each machine ec us-west-1.compute.amazonaws.com ec us-west-1.compute.amazonaws.com

Sponsored by the National Science Foundation 35 Tutorial Step 6: Using Gush Setup keys and fix hostnames –$ cp ~/Tutorials/DiCloud/ec2/keys/geni-keypair.pem ~/.ssh/id_rsa –$ cp ~/Tutorials/DiCloud/ec2/keys/geni-keypair.pub ~/.ssh/id_rsa.pub –$ keychain ~/.ssh/id_rsa –$ ssh “hostname –$ ssh “hostname Add EC2 VMs to Gush’s directory.xml file –$ cd ~/Tutorials/DiCloud/gush –$ nano directory.xml

Sponsored by the National Science Foundation 36 Tutorial Step 6: Using Gush Run Gush experiment –$ cd ~/Tutorials/DiCloud/gush –$./gush –P –gush> info nodes There are 2 known nodes: [ U ] (Disconnected.) [ U ] (Disconnected.) –gush> load simple.xml Project "simple" is selected.Experiment "simple" is selected. –gush> run > –gush> disconnect gush> has decided to leave the mesh. has decided to leave the mesh. –gush> quit

Sponsored by the National Science Foundation 37 Tutorial Step 6: Using Gush Run Gush experiment –gush> run Starting experiment run. Running experiment simple... gush> The configuration matcher has finished matching.The resource allocator has finished successfully. has joined the mesh. has joined the mesh. The file transfer of Package to ec us-west-1.compute.amazonaws.com has been completed. The software installation of Package on ec us-west-1.compute.amazonaws.com was successful. The file transfer of Package to ec us-west-1.compute.amazonaws.com has been completed. The software installation of Package on ec us-west-1.compute.amazonaws.com was successful. Hello World! Your experiment was successful Hello World! Your experiment was successful! The experiment has ended.

Sponsored by the National Science Foundation 38 Tutorial Step 6: Using Gush with Orca Run Gush experiment using Orca and DiCloud resources –$ cd ~/Tutorials/DiCloud/gush –$ cat omni_config [ setup omni_config with correct ProtoGENI credentials ] –$ nano directory.xml [ change directory.xml to add orca slice ]

Sponsored by the National Science Foundation 39 Tutorial Step 6: Using Gush with Orca Add both Amazon and Orca keys to keychain –$ cd ~/.ssh/; keychain --clear –$ keychain ~/.ssh/id_rsa_geni; keychain ~/.ssh/id_rsa –$ cd ~/Tutorials/DiCloud/gush –$ cp simple.xml simple-orca.xml; nano simple-orca.xml [ Add orca resources to simple.xml example ] …. 4 ….

Sponsored by the National Science Foundation 40 Tutorial Step 6: Using Gush with Orca Create orca-new.rspec –$ nano orca-new.rspec <rspec type="request" xsi:schemaLocation=" xmlns:flack=" xmlns:xsi=" xmlns=

Sponsored by the National Science Foundation 41 Tutorial Step 6: Using Gush with Orca Create Orca slice and slivers; check status –$ helper-scripts/handle-geni.py -n -f pg -a createslice gush-orca- david INFO:omni:Loading config file omni_config INFO:omni:Using control framework pg Created slice with Name gush-orca-david, URN urn:publicid:IDN+emulab.net+slice+gush-orca-david –$ helper-scripts/handle-geni.py -n -f pg -a createsliver gush-orca-david orca-new.rspec INFO:omni:Loading config file omni_config INFO:omni:Using control framework pg Asked to reserve resources. Result: Request id: 8eaacd08-4f4b-4ac6-be0a- ce78dbd77893[ Slice UID: 4e8cec5d-a28a-49df-a485-0a03c46f3a95 | Reservation UID: f3cc03c fac-9f96- d6f94d1a43a1 | Resource Type: acisrencivmsite.vm | Resource Units: 1 ] [ Slice UID: 4e8cec5d-a28a-49df-a485-0a03c46f3a95 | Reservation UID: 2d6fc0e5-77bc-41a9-9c93-7e2d46134b9a | Resource Type: acisrencivmsite.vm | Resource Units: 1 ] [[ Slice UID: 4e8cec5d-a28a-49df-a485-0a03c46f3a95 | Reservation UID: 02db97c4-1ff1-492d-9c c0 | Resource Type: acisrencivmsite.vlan | Resource Units: 1 ]

Sponsored by the National Science Foundation 42 Tutorial Step 6: Using Gush with Orca Create Orca slice and slivers; check status (cont’d) –$ helper-scripts/handle-geni.py -n -f pg -a sliverstatus gush-orca-david –$ helper-scripts/handle-geni.py -n -f pg -a sliverstatus gush-orca-david 99999

Sponsored by the National Science Foundation 43 Tutorial Step 6: Using Gush with Orca Must fix hostnames as before –$ ssh “hostname –$ ssh “hostname

Sponsored by the National Science Foundation 44 Tutorial Step 6: Using Gush with Orca Run Gush experiment as before –$./gush –P –gush> Gush has learned about the slice gush-orca-david. –gush> Updated information on the slice gush-orca-david –gush> info nodes There are 4 known nodes: [ P ] (Disconnected.) [ P ] (Disconnected.) [ U ] (Disconnected.) [ U ] (Disconnected.) –gush> load simple-orca.xml Project "simple" is selected. Experiment "simple" is selected. –gush> run [ output on next slide ]

Sponsored by the National Science Foundation 45 Tutorial Step 6: Using Gush with Orca Run Gush experiment as before –gush> run Starting experiment run. Running experiment simple... gush> The configuration matcher has finished matching. The resource allocator has finished successfully. has joined the mesh. has joined the mesh. The file transfer of Package to ec us-west-1.compute.amazonaws.com has been completed. The software installation of Package on ec us-west-1.compute.amazonaws.com was successful. The file transfer of Package to ec us-west-1.compute.amazonaws.com has been completed. The software installation of Package on ec us-west-1.compute.amazonaws.com was successful. has joined the mesh. The file transfer of Package to has been completed. The software installation of Package on was successful. has joined the mesh. The file transfer of Package to has been completed. The software installation of Package on was successful. Hello World! Your experiment was successful! Hello World! Your experiment was successful! Hello World! Your experiment was Hello World! Your experiment was successful! The experiment has ended.

Sponsored by the National Science Foundation 46 Tutorial Step 6: Using Gush with Orca Disconnect from gush and delete sliver –gush> disconnect gush> has decided to leave the mesh. has decided to leave the mesh. has decided to leave the mesh. has decided to leave the mesh. –gush> quit –$ helper-scripts/handle-geni.py -n -f pg -a deletesliver gush-orca-david INFO:omni:Loading config file omni_config INFO:omni:Using control framework pg Deleted sliver urn:publicid:IDN+emulab.net+slice+gush-orca-david on unspecified_AM_URN at

Sponsored by the National Science Foundation 47 Tutorial Step 6: Terminating EC2 VMs Go back to the Firefox browser and terminate your VMs