Messaging Services and Distributed Systems in the Cloud Jed Sundwall, Global Open Data Lead

What is cloud computing? The on-demand delivery of IT resources over public or private networks with zero up-front costs, no long-term contracts, and pay-as-you-go pricing. 2

“Invention requires two things: the ability to try a lot of experiments, and not having to live with the collateral damage of failed experiments.” — Andy Jassy, CEO, AWS

“Everything fails, all the time.” — Werner Vogels, CTO, Amazon

Loosely-coupled systems The looser they are coupled, the bigger they will scale, the more fault tolerant they will be, the fewer dependencies they will have, the faster you will innovate. One of the many challenges we face as architects and developers as we start to build applications for the cloud or migrate our existing applications, is making them distributed. We need to address scalability, fault tolerance, high availability, and start to think more deeply about things like the CAP Theorem, eventual consistency, distributed transactions, and design patterns that support distributed systems. Messaging can really help us achieve our goals by providing a mechanism for: Reliable durable, fault tolerant delivery of messages Create unidirectional – non-blocking operations Enables focus on logical decomposition of systems and increase autonomy of our components Decrease the dependencies they have on each other 5

CAP* Theorem Consistency - A read is guaranteed to return the most recent write for a given client. Availability - A non-failing node will return a reasonable response within a reasonable amount of time (no error or timeout). Partition Tolerance - The system will continue to function when network partitions occur. * A different CAP than what we’ve been talking about! One of the many challenges we face as architects and developers as we start to build applications for the cloud or migrate our existing applications, is making them distributed. We need to address scalability, fault tolerance, high availability, and start to think more deeply about things like the CAP Theorem, eventual consistency, distributed transactions, and design patterns that support distributed systems. Messaging can really help us achieve our goals by providing a mechanism for: Reliable durable, fault tolerant delivery of messages Create unidirectional – non-blocking operations Enables focus on logical decomposition of systems and increase autonomy of our components Decrease the dependencies they have on each other 6

Messaging enables decoupling Message Queueing Publish-subscribe (pub-sub) Asynchronous Point-to-point Broadcast Point-to-multipoint Producer Queue Consumer Subscriber Topic Publisher Messaging Enables the creation of the these decoupled architectures. In queuing, messages are sent from producers to consumers using message queues. It’s an asynchronous model that provides buffering. This is for messaging from point A to point B, although a single queue can have multiple producers and consumers. Message queuing Messages are sent from a single producer to a single consumer using queues Asynchronous—provides buffering between producer and consumer Point-to-point Pub/Sub In the Pub-sub model, messages are broadcast, or “fanned-out” from a publisher to one or more subscribers using “topics”. Point to multipoint. The classic example here is the stock ticker, where prices are published and multiple readers want to subscribe to that stream of information. Messages are broadcast from a publisher to one or more subscribers using topics Point-to-multipoint Subscriber

When to use messaging Separate parts of an application Web tier instances create work, worker instances complete it Scale and manage tiers separately Perform tasks asynchronously Long-running tasks (e.g. transcoding, transactions) Don’t need to wait for a response (e.g. JS web apps) Independent and fault-tolerant

When to use messaging Connect multiple components Send individual messages or fan-out to many recipients Provide instant or delayed notification Batch and burst processing Be resilient to spikes in traffic Perform work only as fast as necessary to lower costs Don’t lose data

Amazon Simple Notification Service (Amazon SNS) Fast, reliable, scalable fully managed pub-sub service Message notifications pushed to subscribers Use topics to fan out messages to: Amazon SQS queues HTTP endpoints (web servers) AWS Lambda functions Mobile push, SMS, and email Amazon SNS is a managed pub-sub service. It provides the “topic” construct for fan-out. You can fan-out to queues, as well as HTTP endpoints and Lambda functions. This same service is also used for mobile push and SMS notifications.

Amazon SNS: key features Proven reliability with messages are stored across multiple AZs Flexible message delivery over multiple transport protocols Nearly unlimited throughput

Amazon SNS: key features Instantaneous or delayed, push-based delivery Simple APIs and easy integration Amazon CloudWatch metrics and alerts Message payloads up to 256 KB

SNS is Simple to Use Create Topic Subscribe Publish M M 1 2 3 Endpoint subscribes to a Topic SNS Topic Subscriber Endpoint 3 M Publisher sends message “M” to a Topic SNS Topic See full API list: http://docs.aws.amazon.com/sns/latest/api/Welcome.html Publisher M Topic pushes message to Subscriber SNS Topic Subscriber Endpoint You can get started with just four APIs.

Earthquake Response Prototype

Goal: To provide meaningful, actionable information as quickly as possible to respond to disaster situations arising from earthquakes.

Inputs/Outputs earthquake.usgs.gov USGS Landsat satellite imagery ESA Sentinel-1 satellite imagery ESA Sentinel-2 satellite imagery OpenStreetMap data Bonus: bring your own high resolution imagery A publicly accessible S3 bucket with: pre and post event optical and SAR imagery a list of medical facilities within a 1,000 km radius available within seconds of reporting

Infrastructure Diagram, SNS feed Use an object key to determine new/old feed item earthquake.usgs.gov Polls for new data on a regular basis SNS topic for any new earthquakes Lambda function polls for data every minute It looks at the keys in the S3 bucket to determine if the feed item is new or old If it is a new item, message is sent to SNS for downstream consumption

Infrastructure Diagram, data handler Query metadata tables with Athena Landsat on AWS Sentinel-2 on AWS Sentinel-1 on AWS SNS topic Runs on SNS message Final shareable bucket Lambda function runs whenever a new message is passed through SNS Function first checks earthquake magnitude against a threshold (e.g. magnitude 5.0) Queries imagery metadata tables via Athena to determine available imagery over epicenter Images are copied from source bucket to destination bucket Queries OSM data stored on S3 via Athena and saves results in JSON to final S3 bucket Query OSM with Athena OSM on AWS

See it live! http://earthquake-data-test.s3.amazonaws.com/index.html Uses the publicly available aws-js-s3-explorer https://github.com/awslabs/aws-js-s3-explorer http://earthquake-data-test.s3.amazonaws.com/index.html

aws.amazon.com/earth Climate Models Aerial Imagery Elevation Models We want anyone to be build their own planetary-scale applications. In September, we launched Earth on AWS, which is a new landing page for large geospatial and environmental datasets. These are just a few examples of the kind of data that are available today, and we’re continually looking to bring on more. This data is available for you to do whatever you want with. You own your applications. You own your IP. Go forth. Having this data in the cloud allows people to close the gap between data and action. People can ask questions that they simply wouldn’t ask before. Climate Models Aerial Imagery Elevation Models Satellite Imagery High-resolution Radar

aws.amazon.com/earth/research-credits AWS Cloud Credits for Research provide promotional AWS cloud credits for anyone to conduct research using Earth Observation data. aws.amazon.com/earth/research-credits We want *everyone* to think big about how we can work together and find ways to solve the environmental challenges we’re facing today. Students, researchers, amateurs, and businesses all have ideas to contribute here. We don't want access to data or computing resources to stop them from getting started and trying something new. We can't wait to see how it's used, and we hope it inspires others to build new solutions for other challenges.