1. GETTING STARTED WITH AWS AND PYTHON

2. OUTLINE  Intro to Boto  Installation and configuration  Working with AWS S3 using Bot  Working with AWS SQS using Boto  Working with AWS EC2 using Boto  Geoprocessing tools for AWS

3. INTRO TO BOTO  An integrated interface to current and future infrastructural services offered by Amazon Web Services.  All features work with Python 2.6 and 2.7.  Official documentation: https://boto.readthedocs.org/en/latest/

4. INSTALLATION AND CONFIGURATION  Download boto package http://boto.googlecode.com/files/boto-2.6.0.tar.gz  Run setup.py to install  Credential Configuration  aws_access_key_id={ACCESS KEY ID}  aws_secret_access_key={SECRET ACCESS KEY} Note: can be found on your AWS account  Security Credentials

5. BOTO FOR AMAZON S3  Store files onto S3  Connect  Create Bucket  New Key  Set Contents import boto S3=boto.connect_s3() bucket=s3.create_bucket('media.yourdomain.com') key=bucket.new_key('examples/first_file.csv') key.set_contents_from_filename('/home/patrick/first _file.csv') key.set_acl('public-read')

6.  File download  Connect  Get Bucket  Get Key  Get Contents import boto s3=boto.connect_s3() key=s3.get_bucket('media.yourdomain.com').get_key ('examples/first_file.csv') key.get_contents_to_filename('/myfile.csv')

7.  Move file to bucket  Connect  Get Bucket  Get Key  Copy  Delete Key import boto s3=boto.connet_s3() key=s3.get_bucket('media.yourdomain.com').get_key ('examples/first_file.csv') new_key=key.copy('media2.yourdomain.com', 'sample/file.csv') if new_key.exists: key.delete()

8. WORKING WITH AMAZON SQS  A message-queuing service that acts as glue for building applications in the cloud  Write and read messages  Count existing and deleted messages from the queue

9.  Write messages  Connect  Create Queue  New Message  Write Msg into Queue yourmain

10.  Read and delete messages  Connect  Get Message in Queue  Load Message  Delete Message

11. WORKING WITH AMAZON EC2  Use a prebuilt or build a new image  Boot up nodes on Amazon EC2  Execute a script to read these messages  Use the data as input to process into output leveraging the cloud nodes

12. BOOT AN IMAGE  Activate an Amazon EC2 connection object  Under the support of EC2, AWS generates the key pair import boto ec2 = boto.connect_ec2() key_pair = ec2.create_key_pair('ec2-sample-key') key_pair.save('/Users/patrick/.ssh') reservation = ec2.run_instances(image_id='ami-bb709dd2', key_name='ec2-sample-key')

13.  Wait a min or two while it loads for r in ec2.get_all_instances(): if r.id == reservation.id: break print r.instances[0].public_dns_name # output: ec2-184-73-24- 97.compute-1.amazonaws.com $ chmod 600 ~/.ssh/ec2-sample-key.pem $ ssh -i ~/.ssh/ec2-sample-key.pem ubuntu@ec2-184-73-24-97.compute- 1.amazonaws.com

14. AUTOMATIC INSTALLATION  Install what you want automatically, saving time  Run a script to process the queue without your having to log on reservation = ec2.run_instances( image_id = "ami-bb709dd2", key_name = "ec2-sample-key", user_data = """#!/bin/bash apt-get update apt-get install -y imagemagick """)

15. SAMPLE APPLICATION  Process PDF documents into preview images for display on a Web site

16. GP TOOLS FOR AWS EMR  GPTools for AWS by ESRI  Enable ArcGIS users to leverage AWS through GP tools  Available on GitHub https://github.com/Esri/gptools-for-aws#works  Not updated recently.

17.  Install Boto on Windows using win-pip  Add GP Tools for AWS  Download  Unzip  Connect to Folder in ArcGIS

18.  Upload data and hive script that points to it to S3  Start an EMR cluster (Hadoop, hive, GIS tools)

19.  Run a hive query and use the output to define the output directory in S3  Terminate the cluster after query is executed

20.  Download your output from S3

21. REFERENCES  Altman, P. Getting started with AWS and Python https://aws.amazon.com/articles/Python/3998  GP Tools for Amazon Web Services Elastic Map Reduce (Hosted Hadoop Framework) https://github.com/Esri/gptools-for-aws