1. Messaging Services and Distributed Systems in the Cloud
Jed Sundwall, Global Open Data Lead

2. What is cloud computing?
The on-demand delivery of IT resources over public or private networks with zero up-front costs, no long-term contracts, and pay-as-you-go pricing. 2

3. “Invention requires two things: the ability to try a lot of experiments, and not having to live with the collateral damage of failed experiments.”
— Andy Jassy, CEO, AWS

4. “Everything fails, all the time.”
— Werner Vogels, CTO, Amazon

5. Loosely-coupled systems
The looser they are coupled, the bigger they will scale, the more fault tolerant they will be, the fewer dependencies they will have, the faster you will innovate.
One of the many challenges we face as architects and developers as we start to build applications for the cloud or migrate our existing applications, is making them distributed. We need to address scalability, fault tolerance, high availability, and start to think more deeply about things like the CAP Theorem, eventual consistency, distributed transactions, and design patterns that support distributed systems.
Messaging can really help us achieve our goals by providing a mechanism for:
Reliable durable, fault tolerant delivery of messages
Create unidirectional – non-blocking operations
Enables focus on logical decomposition of systems and increase autonomy of our components
Decrease the dependencies they have on each other 5

6. CAP* Theorem
Consistency - A read is guaranteed to return the most recent write for a given client. Availability - A non-failing node will return a reasonable response within a reasonable amount of time (no error or timeout). Partition Tolerance - The system will continue to function when network partitions occur. * A different CAP than what we’ve been talking about!
One of the many challenges we face as architects and developers as we start to build applications for the cloud or migrate our existing applications, is making them distributed. We need to address scalability, fault tolerance, high availability, and start to think more deeply about things like the CAP Theorem, eventual consistency, distributed transactions, and design patterns that support distributed systems.
Messaging can really help us achieve our goals by providing a mechanism for:
Reliable durable, fault tolerant delivery of messages
Create unidirectional – non-blocking operations
Enables focus on logical decomposition of systems and increase autonomy of our components
Decrease the dependencies they have on each other 6

7. Messaging enables decoupling
Message Queueing
Publish-subscribe (pub-sub)
Asynchronous
Point-to-point
Broadcast
Point-to-multipoint
Producer
Queue
Consumer
Subscriber
Topic
Publisher
Messaging Enables the creation of the these decoupled architectures.
In queuing, messages are sent from producers to consumers using message queues. It’s an asynchronous model that provides buffering. This is for messaging from point A to point B, although a single queue can have multiple producers and consumers.
Message queuing
Messages are sent from a single producer to a single consumer using queues
Asynchronous—provides buffering between producer and consumer
Point-to-point
Pub/Sub
In the Pub-sub model, messages are broadcast, or “fanned-out” from a publisher to one or more subscribers using “topics”. Point to multipoint. The classic example here is the stock ticker, where prices are published and multiple readers want to subscribe to that stream of information.
Messages are broadcast from a publisher to one or more subscribers using topics
Point-to-multipoint
Subscriber

8. When to use messaging Separate parts of an application
Web tier instances create work, worker instances complete it
Scale and manage tiers separately
Perform tasks asynchronously
Long-running tasks (e.g. transcoding, transactions)
Don’t need to wait for a response (e.g. JS web apps)
Independent and fault-tolerant

9. When to use messaging Connect multiple components
Send individual messages or fan-out to many recipients
Provide instant or delayed notification
Batch and burst processing
Be resilient to spikes in traffic
Perform work only as fast as necessary to lower costs
Don’t lose data

10. Amazon Simple Notification Service (Amazon SNS)
Fast, reliable, scalable fully managed pub-sub service
Message notifications pushed to subscribers
Use topics to fan out messages to:
Amazon SQS queues
HTTP endpoints (web servers)
AWS Lambda functions
Mobile push, SMS, and
Amazon SNS is a managed pub-sub service. It provides the “topic” construct for fan-out. You can fan-out to queues, as well as HTTP endpoints and Lambda functions. This same service is also used for mobile push and SMS notifications.

11. Amazon SNS: key features
Proven reliability with messages are stored across multiple AZs
Flexible message delivery over multiple transport protocols
Nearly unlimited throughput

12. Amazon SNS: key features
Instantaneous or delayed, push-based delivery
Simple APIs and easy integration
Amazon CloudWatch metrics and alerts
Message payloads up to 256 KB

13. SNS is Simple to Use Create Topic Subscribe Publish M M 1 2 3
Endpoint subscribes to a Topic
SNS Topic
Subscriber
Endpoint 3 M
Publisher sends message “M” to a Topic
SNS Topic
See full API list:
Publisher M
Topic pushes message to Subscriber
SNS Topic
Subscriber
Endpoint
You can get started with just four APIs.

14. Earthquake Response Prototype

15. Goal: To provide meaningful, actionable information as quickly as possible to respond to disaster situations arising from earthquakes.

16. Inputs/Outputs earthquake.usgs.gov
USGS Landsat satellite imagery
ESA Sentinel-1 satellite imagery
ESA Sentinel-2 satellite imagery
OpenStreetMap data
Bonus: bring your own high resolution imagery
A publicly accessible S3 bucket with:
pre and post event optical and SAR imagery
a list of medical facilities within a 1,000 km radius
available within seconds of reporting

17. Infrastructure Diagram, SNS feed
Use an object key to determine new/old feed item
earthquake.usgs.gov
Polls for new data on a regular basis
SNS topic for any new earthquakes
Lambda function polls for data every minute
It looks at the keys in the S3 bucket to determine if the feed item is new or old
If it is a new item, message is sent to SNS for downstream consumption

18. Infrastructure Diagram, data handler
Query metadata tables with Athena
Landsat on AWS
Sentinel-2 on AWS
Sentinel-1 on AWS
SNS topic
Runs on SNS message
Final shareable bucket
Lambda function runs whenever a new message is passed through SNS
Function first checks earthquake magnitude against a threshold (e.g. magnitude 5.0)
Queries imagery metadata tables via Athena to determine available imagery over epicenter
Images are copied from source bucket to destination bucket
Queries OSM data stored on S3 via Athena and saves results in JSON to final S3 bucket
Query OSM with Athena
OSM on AWS

19. See it live! http://earthquake-data-test.s3.amazonaws.com/index.html
Uses the publicly available aws-js-s3-explorer

20. aws.amazon.com/earth Climate Models Aerial Imagery Elevation Models
We want anyone to be build their own planetary-scale applications.
In September, we launched Earth on AWS, which is a new landing page for large geospatial and environmental datasets.
These are just a few examples of the kind of data that are available today, and we’re continually looking to bring on more.
This data is available for you to do whatever you want with. You own your applications. You own your IP. Go forth.
Having this data in the cloud allows people to close the gap between data and action. People can ask questions that they simply wouldn’t ask before.
Climate
Models
Aerial
Imagery
Elevation
Models
Satellite
Imagery
High-resolution
Radar

21. aws.amazon.com/earth/research-credits
AWS Cloud Credits for Research provide promotional AWS cloud credits for anyone to conduct research using Earth Observation data.
aws.amazon.com/earth/research-credits
We want *everyone* to think big about how we can work together and find ways to solve the environmental challenges we’re facing today.
Students, researchers, amateurs, and businesses all have ideas to contribute here.
We don't want access to data or computing resources to stop them from getting started and trying something new.
We can't wait to see how it's used, and we hope it inspires others to build new solutions for other challenges.