Download presentation
Presentation is loading. Please wait.
Published bySaige Neblett Modified over 10 years ago
1
JamesRH 2007.06.11 http://msblogs/jamesrh
2
7 major AWS Services (http://aws.amazon.com)http://aws.amazon.com Amazon E-Commerce Service (ECS) Amazon Elastic Compute Cloud Beta (EC2) Amazon Historical Pricing Amazon Mechanical Turk Beta Amazon Simple Storage Service (S3) Amazon Simple Queue Service (SQS) Alexa Web Services Originally planning to write against both EC2 & S3 but EC2 beta was closed until recently There are EC2 scaling issues (confirmed by Bezos at Web2.0) S3 Scaling well: 5B objects Peak transaction rate: Day: 923M Second: 16,67 This demo app was written to learn the S3 interfaces, their strengths, and weaknesses 4/17/20152
3
Went to aws.amazon.com Signed into Amazon account Same one used for Amazon shopping Fill in additional contact information agreed to terms and conditions AWS provides Access Key & Secret Key Tested using www.awszone.comwww.awszone.com Simple web app supporting most AWS operations via SOAP Failed with: “Client.NotSignedUp” Email showed up later complaining about credit card out of date I corrected the payment issue and was up and running User provisioning is fast and efficient Overall, quick and easy process 4/17/20153
4
Blob store with per blob metadata Blobs supported up to 5GB ~36.4 hour upload via Comcast “broadband” Metadata up to 4k in name/value pairs (2k of user metadata) Up to 100 ACLs on each object (part of system metadata) Read, write, read_ACP, write_ACP, and full_control Owner has full control and is not changeable Each blob stored in a bucket Bucket name space shared by ALL S3 users All buckets have a single unchanging owner Each blob has a URI identifier in a flat per-bucket namespace Directory hierarchy simulated Get(key=“C:/X/Y/Z.txt”) gets the file Get(key=“C:/X”) gets all objects with leading “C:/X/” in URI Get(key=“C:/X/”, separator=“/”) gets all objects with leading “C:/X/” but no subsequent “/” 4/17/20154
5
Client libraries offered as samples for: Java SOAP, Java REST, C# SOAP, C# REST, Ruby REST, Python REST, & Perl REST BitTorent also supported Docs at protocol layer rather than APIs API implementations uneven without docs C# SOAP has max send of 1M (can use DIME attachment for larger) C# REST doesn’t support direct streaming Source provided so quality & functionality issues all have solution 4/17/20155
6
conn = new AWSAuthConnection(key, secretKey); res = conn.delete(Bucket, key, null); Soap request Headers: - 10T01CX2MA2M9WD25002 NbT52AjYRzhHdA490B+T20KBYYg= 2007-03-14T20:10:28.000Z 10T01CX2MA2M9WD25002-Archiver C:/JamesRH/dev/AWS/tst/test.tst*1 SOAP response Headers - 204 No Content 4/17/20156 REST Access Key:AWS : Base64(hmac-sha1(VERB, Content-MD5, Content type, Date, AMZ headers, resourceKey )) SOAP Model Doesn’t sign
7
S3 Archiver is a version store Registered directories & sub-directories contents are stored in S3 Each new file version is stored separately Any version of any file can be restored in place Multiple versions of a file can be restored with version suffixes on filename Any file version can be deleted from S3 Roughly 3,500 lines of C# written against C# REST interface 4/17/20157 Performance: 12,220 files, 12,500 versions totaling 5.1 GB synchronized in under 30 seconds (local- only operation) Actual transfers very slow ranging from roughly 1GB/hour to as low as 100 MB/hour (28 KBs) Issues mostly network bandwidth-related on the perf side and standard internet flakiness on the reliability side
8
Files stored in S3 with version numbers C:\JamesRH\OpsFriendlyApps.docsx*10 Metadata as name/value pairs with each file: Version Size Hash Last Write Create Time Access Time Archive Time Fetching metadata for 1,000s of files slow so have a cache stored in S3 S3_Archiver.xml Large scale testing required real data so implemented encryption Single file operations slow and very uneven in performance so need to run all non-metadata-only operations asynchronously On scale-up to 1000s of files S3_Archiver.xml gets large and slow so implemented compression On scale-up to 10s of 1000s of files S3_Archiver.xml still slow to transfer so check hash and avoid transfers when possible 4/17/20158 C:/user/file1*1 C:/user/file1*2 C:/user/file2*1 C:/file3*1 cache cache*C:/usr/libary1 cache*C:/usr/library2
9
4/17/20159
10
Error rate ~4 in 1,000 and similar across different networks: Comcast: Recv: 43 to 567.19 KBs Send: 5 to 52 KBs Microsoft: Recv : 317 to 10,935 KBs Send: 437 to 5,556 KBs Home Large File: Send Fast/Slow ratio: 9.8x Recv Fast/Slow ratio: 12.9x Recv/Send ratio: 10.8x Work Large File: Send Fast/Slow ratio: 12.7x Recv Fast/Slow ratio: 34.4x Recv/Send ratio: 1.9x Conclusion: Send to receive difference ranges from a factor of 2 to an order of magnitude 2 orders of magnitude spread in throughput Internet error rate high 5+ orders of magnitude higher latency than local disk 4/17/201510 Source: one week hourly perf & reliability probe of S3 at 1k, 10k, 100k, 1m, & 25m
11
4/17/201511 Kbs 4 errors Errors show as negative throughput
12
4/17/201512 Kbs Errors show as negative throughput 1 error
13
4/17/201513 Kbs Errors show as negative throughput 2 errors 3 errors
14
4/17/201514 Kbs Errors show as negative throughput 1 error
15
Summary of This Month's Activity as of March 14, 2007 Billing Cycle for this Report: March 1 - March 31, 2007 Usage Charges Rate Usage Totals Amazon Simple Storage Service $0.20 per GB of data transferred (details) 15.174 GB $3.04 details $0.15 per GB-Month of storage used (details) 0.214 GB-Mo $0.04 details Total:$3.08 All web services are sold by Amazon Digital Services, Inc. Summary of Last Month's Statement: February 1 - February 28, 2007 Total Charges for February 1 - February 28, 2007: $0.07 Amazingly inexpensive: Developed a 3,000 line interactive system storing 1,000s of files and several GB stored with many test cycles at $3.11 4/17/201515
16
Current bandwidth price (through May 31, 2007) $0.20 / GB – uploaded $0.20 / GB - downloaded New bandwidth price (effective June 1, 2007) $0.10 per GB - all data uploaded $0.18 per GB - first 10 TB / month data downloaded $0.16 per GB - next 40 TB / month data downloaded $0.13 per GB - data downloaded / month over 50 TB Data transferred between Amazon S3 and Amazon EC2 will remain free New request-based price (effective June 1, 2007) $0.01 per 1,000 PUT or LIST requests $0.01 per 10,000 GET and all other requests No charge for delete requests Storage will continue to be charged at $0.15 / GB-month used 4/17/201516
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.