1. Python on AWS Lambda: Practical Applications
Brian Morton
Chief Information Officer - DVIDS

2. Online, searchable archive for DoD public media
1.5 million images
300K+ videos
Fully open source stack*
All content copyright free
Publicly accessible API

3. *Classic City Computing Partnership

4. AWS Basic Glossary AWS = Amazon Web Services (the cloud)
EC2 = Elastic Compute (virtual hosts)
S3 = Simple Storage Service (storage in the cloud)
ELB = Elastic Load Balancer (Haproxy in the cloud)
SES = Simple Service ( in the cloud)
SQS = Simple Queue Service (RabbitMQ in the cloud)
RDS = Relational Database Service (Mysql in the cloud)
Lambda = “Serverless” compute

5. AWS Lambda “Function” as infrastructure
Uses spare cycles on AWS compute nodes (EC2 virtualization hosts)
Bills in 100ms increments of CPU time based on memory profile allocated
Runs in response to many types of events (S3 file delivery, SNS/SQS, SES)
Designed to run up to 100 concurrent instances by default (can be exceeded)
Languages currently supported: Java, Node.JS, Python
More info:

6. Lambda Example
virtualenv zip
import MySQLdb
import PIL
import django

7. Practical Application 1: Image Resizing
Performs read/write with S3 bucket
Reads original image from bucket path
Performs resizing according to json config in package
Can be overridden through event arguments
Writes resized images back to bucket
Could be called from S3 upload, API Gateway, Simple Service and others

8. Allocated
Used
Image Size
Memory (MB)
Price per 100ms ($)
Time (ms)
Cost
Total Cost
128
n/a
192
256
210
17000
320
13000
384
10900
163.5
448
9500
512
8500
576
229
7800
640
7200
1152
4200
189
1216
5200
1280
3800
1344
3400
1408
3600
1472
3300
1536
3200
Asset count

9. Practical Application 2: Social Data Mining
Searches Twitter for posts containing words or hashtags
Topic and item count can be overridden through event args
Returns number of Tweets, input topic, and length of time to search
Wired to API Gateway with public endpoint
Next logical step might be storage in Solr or RDS for analysis
Could be called from API endpoint, or perhaps SES?

10. Sample Application: ETL Data Processing
Following diagram shows ETL data processing pipeline
S3 file upload triggers Lambda
Initial function parses file, writes records to SQS
Each SNS from queue instantiates a new Lambda
Processing function does work (validation, lookup, soundex, etc)
Data written to RDS or Redshift

11. 1. File upload to S3 bucket
2. Parsing/validate Lambda invoked
3. Records queued to SQS
4. SNS notifications instantiate processing Lambdas
5. Processed data written to persistent storage

12. Questions: bmorton@dvidshub.net