Lecture: Amazon AWS Instructor: Weidong Shi (Larry), PhD

Slides:



Advertisements
Similar presentations
Ivan Pleština Amazon Simple Storage Service (S3) Amazon Elastic Block Storage (EBS) Amazon Elastic Compute Cloud (EC2)
Advertisements

Amazon Web Services (aws) B. Ramamurthy. Introduction  Amazon.com, the online market place for goods, has leveraged the services that worked for their.
University of Notre Dame
© 2010 VMware Inc. All rights reserved Amazon Web Services.
Amazon Web Services and Eucalyptus
1 NETE4631 Cloud deployment models and migration Lecture Notes #4.
Webscale Computing Mike Culver Amazon Web Services.
Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License.
Nikolay Tomitov Technical Trainer SoftAcad.bg.  What are Amazon Web services (AWS) ?  What’s cool when developing with AWS ?  Architecture of AWS 
Google AppEngine. Google App Engine enables you to build and host web apps on the same systems that power Google applications. App Engine offers fast.
Amazon EC2 Quick Start adapted from EC2_GetStarted.html.
Cloud Computing using AWS C. Edward Chow. Advanced Internet & Web Systems chow2 Outline of the Talk Introduction to Cloud Computing AWS EC2 EC2 API A.
A Brief Overview by Aditya Dutt March 18 th ’ Aditya Inc.
Customized cloud platform for computing on your terms !
The Blue Book pages 19 onwards
Cloud Computing Amazon Web Services - introduction Keke Chen.
1 Apache. 2 Module - Apache ♦ Overview This module focuses on configuring and customizing Apache web server. Apache is a commonly used Hypertext Transfer.
1 NETE4631 Amazon Cloud Offerings Lecture Notes #6.
Cloud Computing & Amazon Web Services – EC2 Arpita Patel Software Engineer.
Webscale Computing Mike Culver Amazon Web Services.
Microsoft Azure SoftUni Team Technical Trainers Software University
ArcGIS Server for Administrators
AWS Amazon Web Services Georges Akpoly CS252. Overview of AWS Amazon Elastic Compute Cloud (EC2) Amazon Simple Storage Service (S3) Amazon Simple Queue.
CLOUD WITH AMAZON. Amazon Web Services AWS is a collection of remote computing services Elastic Compute Cloud (EC2) provides scalable virtual private.
Powered by Microsoft Azure, PointMatter Is a Flexible Solution to Move and Share Data between Business Groups and IT MICROSOFT AZURE ISV PROFILE: LOGICMATTER.
Cloud Computing is a Nebulous Subject Or how I learned to love VDF on Amazon.
Launch Amazon Instance. Amazon EC2 Amazon Elastic Compute Cloud (Amazon EC2) provides resizable computing capacity in the Amazon Web Services (AWS) cloud.
Cloud Computing from a Developer’s Perspective Shlomo Swidler CTO & Founder mydrifts.com 25 January 2009.
© 2015 MetricStream, Inc. All Rights Reserved. AWS server provisioning © 2015 MetricStream, Inc. All Rights Reserved. By, Srikanth K & Rohit.
GETTING STARTED WITH AWS AND PYTHON. OUTLINE  Intro to Boto  Installation and configuration  Working with AWS S3 using Bot  Working with AWS SQS using.
SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI CloudBroker usage Zoltán Farkas MTA SZTAKI LPDS
INTRODUCTION TO AMAZON WEB SERVICES (EC2). AMAZON WEB SERVICES  Services  Storage (Glacier, S3)  Compute (Elastic Compute Cloud, EC2)  Databases (Redshift,
DreamFactory for Microsoft Azure Is an Open Source REST API Platform That Enables Mobilization of Data in Minutes across Frameworks and Storage Methods.
SEMINAR ON.  OVERVIEW -  What is Cloud Computing???  Amazon Elastic Cloud Computing (Amazon EC2)  Amazon EC2 Core Concept  How to use Amazon EC2.
AWS Simple Icons v AWS Simple Icons: Usage Guidelines
Univa Grid Engine Makes Work Management Automatic and Efficient, Accelerates Deployment of Cloud Services with Power of Microsoft Azure MICROSOFT AZURE.
Amazon Web Services (aws)
Deploying Web Application
DocFusion 365 Intelligent Template Designer and Document Generation Engine on Azure Enables Your Team to Increase Productivity MICROSOFT AZURE APP BUILDER.
Customized cloud platform for computing on your terms !
Amazon Storage- S3 and Glacier
Platform as a Service.
Logo here Module 3 Microsoft Azure Web App. Logo here Module Overview Introduction to App Service Overview of Web Apps Hosting Web Applications in Azure.
Veeam Backup Repository
AWS COURSE DEMO BY PROFESSIONAL-GURU. Amazon History Ladder & Offering.
Utilization of Azure CDN for the large file distribution
OpenNebula Offers an Enterprise-Ready, Fully Open Management Solution for Private and Public Clouds – Try It Easily with an Azure Marketplace Sandbox MICROSOFT.
2018 Amazon AWS DevOps Engineer Professional Dumps - DumpsProfessor
Intelledox Infiniti Helps Organizations Digitally Transform Paper and Manual Business Processes into Intuitive, Guided User Experiences on Azure MICROSOFT.
AWS: EC2, S3 and Other Services
20409A 7: Installing and Configuring System Center 2012 R2 Virtual Machine Manager Module 7 Installing and Configuring System Center 2012 R2 Virtual.
Zadara™ Virtual Private Storage Arrays™: High Performance, High Availability NAS & SAN by the Hour, with Private, Dedicated Resources MICROSOFT AZURE.
Data Security for Microsoft Azure
Accelerate Your Self-Service Data Analytics
HC Hyper-V Module GUI Portal VPS Templates Web Console
AWS: EC2, S3 and Other Services
Dell Data Protection | Rapid Recovery: Simple, Quick, Configurable, and Affordable Cloud-Based Backup, Retention, and Archiving Powered by Microsoft Azure.
Introduction to Apache
Keep Your Digital Media Assets Safe and Save Time by Choosing ImageVault to be Your Digital Asset Management Solution, Hosted in Microsoft Azure Partner.
Appcelerator Arrow: Build APIs in Minutes. Connect to Any Data Source
Microsoft Virtual Academy
Media365 Portal by Ctrl365 is Powered by Azure and Enables Easy and Seamless Dissemination of Video for Enhanced B2C and B2B Communication MICROSOFT AZURE.
Abiquo’s Hybrid Cloud Management Solution Helps Enterprises Maximise the Full Potential of the Microsoft Azure Platform MICROSOFT AZURE ISV PROFILE: ABIQUO.
AWS Cloud Computing Masaki.
Saranya Sriram Developer Evangelist | Microsoft
Last.Backend is a Continuous Delivery Platform for Developers and Dev Teams, Allowing Them to Manage and Deploy Applications Easier and Faster MICROSOFT.
The Blue Book pages 19 onwards
Features Overview.
COMPANY PROFILE: REELWAY
Presentation transcript:

Lecture: Amazon AWS Instructor: Weidong Shi (Larry), PhD COSC6376 Cloud Computing Lecture: Amazon AWS Instructor: Weidong Shi (Larry), PhD Computer Science Department University of Houston

Outline AWS Elastic Compute Cloud (EC2) Simple Storage Services (S3) Elastic Mapreduce CloudFront Simple Queue Service

Reading Assignment Google bigtable Thursday in class

Overview Amazon Web Services Infrastructure As a Service Amazon Simple Storage Service Amazon Elastic Compute Cloud Amazon Simple Queue Service Amazon SimpleDB Amazon CloudFront Commerce As a Service Amazon Flexible Payments Service Fulfillment Web Service People As a Service Amazon Mechanical Turk Alexa Web Services Alexa Web Information Service Alexa Top Sites Alexa Site Thumbnail Alexa Web Search Platform

EC2

EC2 A typical example of utility computing Functionality: launch instances with a variety of operating systems (windows/linux) load them with your custom application environment (customized AMI) Full root access to a blank Linux machine manage your network’s access permissions run your image using as many or few systems as you desire (scaling up/down)

Backyard… Powered by Xen – Virtual Machine Different from Vmware - high performance Hardware contributions by Intel (VT-x/Vanderpool) and AMD (AMD-V) Supports “Live Migration” of a virtual machine between hosts We will dedicate one class to virtualization ...

Amazon Machine Images Public AMIs: Use pre-configured, template AMIs to get up and running immediately. Choose from Fedora, Movable Type, Ubuntu configurations, and more Private AMIs: Create an Amazon Machine Image (AMI) containing your applications, libraries, data and associated configuration settings Paid AMIs: Set a price for your AMI and let others purchase and use it (Single payment and/or per hour) AMIs with commercial DBMS

Normal way to use EC2 For web applications For data intensive analysis Run your base system in minimum # of VMs Monitoring the system load (user traffic) Load is distributed to VMs If over some threshold  increase # of VMs If lower than some thresholds  decrease # of VMs For data intensive analysis Estimate the optimal number of nodes (tricky!) Load data Start processing

Tools (most are for web apps) Elastic Block Store: mountable storage, local to each VM instance Elastic IP address: programmatically remap public IP to any instance Virtual private cloud: bridge private cloud and AWS resources CloudWatch: monitoring EC2 resouces Elastic load balancing: automatically distribute incoming traffic across instances

Type of instances Standard instances (micro, small, large, extra) E.g., small: 1.7GB Memory, 1EC2 Compute Unit (1 2ghz core?), 160 GB instance storage High-CPU instances More CPU with same amount of memory

AMIs with special software IBM DB2, Informix Dynamic Server, Lotus Web Content Management, WebSphere Portal Server MS SQL Server, IIS/Asp.Net Hadoop Open MPI Apache web server MySQL Oracale 11g …

Pricing (2010)

Preparation Security credentials Access Key ID Security access key X.509 certificate “create a certificate” Download the private key and the certificate (i.e., the public key) and save them to ~/.ec2/

AWS keys AWS access key This is actually username . It is alphanumeric text string that uniquely identifies user who owns account. No two accounts can have same AWS Access Key. AWS secret key This key plays role of password . It's called secret because it is assumed to be known to owner only.

AWS keys

Preparation Methods for accessing EC2 Command line tools boto python library

Preparation Ec2 command line tools have been installed at /usr/local/ec2 You have to set up env variables JAVA_HOME EC2_HOME Add $EC2_HOME/bin to PATH EC2_PRIVATE_KEY=~/.ec2/pk-XXXXX.pem EC2_CERT=~/.ec2/cert-XXXXXXX.pem Both pk*.pem and cert*.perm are from the x.509 certificate you downloaded from your account)

Ready to start! Check AMIs ec2-describe-images –o self –o amazon | grep machine|less Looking for … IMAGE ami-3c47a355 ec2-public-images/getting-started.manifest.xml amazon available public i386

Generate key pair 1. ec2-add-keypair gsg-keypair 2. Paste the following part to the file ~/.ec2/id_rsa-gsg-keypair -----BEGIN RSA PRIVATE KEY----- …. -----END RSA PRIVATE KEY----- 3. chmod 600 id_rsa-gsg-keypair

Run an instance ec2-run-instances ami-3c47a355 –k gsg-keypair ec2-describe-instances i-395bf151 RESERVATION r-29f61541 425984194090 default INSTANCE i-395bf151 ami-3c47a355 pending gsg-keypair 0 m1.small 2009-10-13T05:16:54+0000 us-east-1b aki-a71cf9ce ari-a51cf9cc monitoring-disabled RESERVATION r-29f61541 425984194090 default INSTANCE i-395bf151 ami-3c47a355 ec2-67-202-28-87.compute-1.amazonaws.com domU-12-31-39-06-AC-33.compute-1.internal running gsg-keypair 0m1.small 2009-10-13T05:16:54+0000 us-east-1b aki-a71cf9ce ari-a51cf9cc monitoring-disabled 67.202.28.87 10.208.179.193

Get connected Authorize accesses to ports Connect to your instance ec2-authorize default –p 22 ec2-authorize default –p 80 -- enable ssh and web Connect to your instance http://ec2-67-202-28-87.compute-1.amazonaws.com ec2-get-console-output i-395bf151 ssh -i ~/.ec2/id_rsa-gsg-keypair root@ec2-67-202-28-87.compute-1.amazonaws.com

Clean up Terminate the instance ec2-terminate-instances i-395bf151 Or in the instance, run shutdown –h now

Use boto to access EC2 Create connection >>> from boto.ec2.connection import EC2Connection >>> conn = EC2Connection('<aws access key>', '<aws secret key>') Or if you have set the keys in AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY >>> import boto >>> conn = boto.connect_ec2()

Images >>> images = conn.get_all_images() >>> images >>> for i in range(len(images)): ... print i, images[i].location

Run instance >>> image = images[xxx] # some selected image >>> reservation = image.run() # have various parameter settings, such as key, security group, instance type, etc. >>> reservation.instances [Instance:i-6761850e] >>> instance = reservation.instances[0] >>> instance.state u'pending‘ >>> instance.update() u'pending' >>> # wait a few seconds to minutes u'running'

Retrieve information of instance >>> instance.dns_name u'ec2-72-44-40-153.z-2.compute-1.amazonaws.com' >>> instance.public_dns_name >>> instance.private_dns_name u'domU-12-31-35-00-42-33.z-2.compute-1.internal'

Run multiple instances >>> reservation.image.run(2,2,'gsg-keypair') >>> reservation.instances [Instance:i-5f618536, Instance:i-5e618537] >>> for i in reservation.instances: ... print i.status u'pending' >>>

Terminate instances >>> instance.stop() >>> instance.update() >>> instance.state u'shutting-down' >>> # wait a minute u'terminated' For multiple instances >>> reservation.stop_all() >>> instances = conn.get_all_instances() >>># then check each instance

Security Set launch permission for private AMIs image.get_launch_permission() image.set_launch_permission(list_of_AWS_user_IDs) image.remove_launch_permission(list_of_AWS_user_IDs) Image.reset_launch_permission()

Security >>> sg = rs[1] >>> sg.name u'default' >>> sg.rules [IPPermissions:tcp(0-65535), IPPermissions:udp(0-65535), IPPermissions:icmp(-1--1)] >>>

Create a security group >>> web = conn.create_security_group('apache', 'Our Apache Group') >>> web SecurityGroup:apache >>> web.authorize('tcp', 80, 80, '0.0.0.0/0') True >>> web.authorize(ip_protocol='tcp', from_port=22, to_port=22, cidr_ip='192.168.1.130/32')

Revoke permission >>> web.rules [IPPermissions:tcp(80-80), IPPermissions:tcp(22-22)] >>> web.revoke('tcp', 22, 22, cidr_ip='192.168.1.130/32') True >>> web.rules [IPPermissions:tcp(80-80)] >>>

Simple AWS S3 Tutorial

S3 Write,read,delete objects 1byte-5gb Namespace: buckets, keys, objects Accessible using URLs

S3 – quick review Objects are organized in a two-level directory Bucket container of objects Global unique name Key Like file names Unique in the same bucket Object Indexed by (bucket, key) http://bucket.s3.amazonaws.com/key

# of objects stored 14 Billion 10 Billion 5 Billion 800 Million August 06 April 07 October 07 January 08

S3 namespace Amazon S3 bucket bucket object object object object

S3 namespace Amazon S3 mculver-images media.mydomain.com Beach.jpg 2005/party/hat.jpg img1.jpg img2.jpg public.blueorigin.com index.html img/pic1.jpg

Accessing objects Bucket: my-images, key: jpg1, object: a jpg image accessible with https://my-images.s3.amazonaws.com/jpg1 mapping your subdomain to S3 with DNS CNAME configuration e.g. media.yourdomain.com  media.yourdomain.com.s3.amazonaws.com/

Access control Access log Objects are private to the user account Authentication Authorization ACL: AWS users, users identified by email, any user … Digital signature to ensure integrity Encrypted access: https

Open Source Backup

Access methods Python library Boto Including access methods for almost all AWS services

S3 Programming tools Language API Style Tools Java REST Java compiler and runtime that supports JDK 1.4.2 or later. C# A C# compiler and runtime environment that supports .Net Version 2.0 (e.g., Visual C# 2005 or 2008 Express Edition). Perl Perl 5 with CPAN Digest::SHA1, Bundle::LWP, and XML::Simple modules (go to http://www.perl.org and http://www.cpan.org). PHP The PHP example uses the base installation of PHP5. Python Boto

Check out AWS Developer Resource Center, for more programming examples We will take a look at boto library

Create a connection >>> from boto.s3.connection import S3Connection >>> conn = S3Connection('<aws access key>', '<aws secret key>') These two keys can be found in your security credentials

Keys If you have set the keys in AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY >>> import boto >>> conn=boto.connect_s3()

Creating a bucket >>> bucket = conn.create_bucket(‘mybucket’) Note that mybucket is globally (in the entire S3 system) uniuqe

Storing data >>> from boto.s3.key import Key >>> k = Key(bucket) >>> k.key = 'foobar' >>> k.set_contents_from_string('This is a test of S3')

Retrieve data >>> import boto >>> c = boto.connect_s3() >>> b = c.create_bucket('mybucket') # substitute your bucket name here >>> from boto.s3.key import Key >>> k = Key(b) >>> k.key = 'foobar' >>> k.get_contents_as_string() 'This is a test of S3'

Work on files >>> k = Key(b) >>> k.key = 'myfile' >>>k.set_contents_from_filename('foo.jpg') >>> k.get_contents_to_filename('bar.jpg')

Check all created buckets >>> rs = conn.get_all_buckets() Rs is a list of buckets >>> len(rs) >>> for b in rs: … print b.name … Listing of all available buckets

Set access control Set public readable for entire bucket >>> b.set_acl('public-read') For one object >>> b.set_acl('public-read‘, ‘foobar’) Or if k is a Key >>>k.set_acl(‘public-read’)

Meta data with objects >>> k = Key(b) >>> k.key = 'has_metadata' >>> k.set_metadata('meta1', 'This is the first metadata value') >>> k.set_metadata('meta2', 'This is the second metadata value') >>>k.set_contents_from_filename('foo.txt') >>> k = b.lookup('has_metadata) >>> k.get_metadata('meta1') 'This is the first metadata value'

Amazon Elastic-mapreduce

Elastic Mapreduce Based on hadoop AMI Data stored on S3 “job flow”

Example elastic-mapreduce --create --stream \ --mapper s3://elasticmapreduce/samples/wordcount/wordSplitter.py \ --input s3://elasticmapreduce/samples/wordcount/input --output s3://my-bucket/output --reducer aggregate

Wiki page access statistics

Elsatic mapreduce elastic-mapreduce is a command-line interface to Amazon's elastic mapreduce http://aws.amazon.com/code/Elastic-MapReduce/2264

Launching a virtual hadoop cluster $ elastic-mapreduce --create --name “Wiki log crunch” --alive --num-instances –instance-type c1.medium 20 Created job flow <job flow id> The --alive option tells the job flow to keep running even when it has finished all its steps.

Hadoop NameNode JobTracker DataNode + TaskTracker

Add a step $ elastic-mapreduce --jobflow <jfid> --stream \ --step-name “Wiki log crunch” \ --input s3n://dsikar-wikilogs-2009/dec/ \ --output s3n://dsikar-wikilogs-output/21 \ --mapper s3n://dsikar-wiki-scripts/wikidictionarymap.pl \ --reducer s3n://dsikar-wiki-scripts/wikireduce.pl \ http://<instance public dns>:9100

S3cmd s3cmd is a command-line interface to Amazon's S3 cloud-storage. Download http://sourceforge.net/projects/s3tools/files/s3cmd/1.0.0/s3cmd-1.0.0.tar.gz/download

s3cmd # make bucket $ s3cmd mb s3://dsikar-wikilogs # put log files $ s3cmd put pagecounts-200912*.gz s3://dsikar-wikilogs/dec $ s3cmd put pagecounts-201004*.gz s3://dsikar-wikilogs/apr # list log files $ s3cmd ls s3://dsikar-wikilogs/ # put scripts $ s3cmd put *.pl s3://dsikar-wiki-scripts/ # delete log files $ s3cmd del --recursive --force s3://dsikar-wikilogs/ # remove bucket $ s3cmd rb s3://dsikar-wikilogs/

Elastic mapReduce --create --list --jobflow --describe --stream --terminate

Output files part-00000 part-00001 part-00002 (...)

Aggregation

DNA sequencing example http://aws.amazon.com/articles/2272?_encoding=UTF8&jiveRedirect=1

Word count example http://aws.amazon.com/articles/2273?_encoding=UTF8&jiveRedirect=1

ItemSimilarity http://aws.amazon.com/articles/2274?_encoding=UTF8&jiveRedirect=1

CloudFront

CloudFront For content delivery: distribute content to end users with a global network of edge locations. “Edges”: servers close to user’s geographical location Objects are organized into distributions Each distribution has a domain name Distributions are stored in a S3 bucket

Edge servers US EU Hongkong Japan US and EU are partitioned to different regions Hongkong Japan

Use cases Hosting your most frequently accessed website components Small pieces of your website are cached in the edge locations, and are ideal for Amazon CloudFront. Distributing software distribute applications, updates or other downloadable software to end users. Publishing popular media files If your application involves rich media – audio or video – that is frequently accessed

Simple Queue Service

Simple Queue Service Store messages traveling between computers Make it easy to build automated workflows Implemented as a web service read/add messages easily Scalable to millions of messages a day

Some features Message body : <8Kb in any format Message is retained in queues for up to 4days Messages can be sent and read simultaneously Can be “locked”, keeping from simultaneous processing Accessible with SOAP/REST Simple: Only a few methods Secure sharing

A typical workflow

Workflow with AWS