1. U.S. Department of the Interior U.S. Geological Survey Dealing with Streaming Data CDI Connected Devices Working Group Jeff Allen and Dell Long USGS Fort Collins Science Center

2. Technology Challenge  Lots of data now available in streams  Social media  Environmental parametrics  Observations from many simultaneous sources  More variables as part of stream  Much higher expectations for technology  Deal more sources  Provide more analyses  Present analyzed results sooner

3. Traditional Workflow  Source sends signal to server  Signal filtered, preprocessed, and pushed to database for persistence  Data post-processed  Products generated -- dispatch, display, and distribution  Excruciatingly sequential  Scale by buying bigger servers, multi-threading  Memory allocation always an issue

4. Rethinking Options – Reactive Applications  Based on Reactive Manifesto  Architecture to build event-driven, scaleable, resilient, and responsive applications  Designed to take advantage of cloud environments  Works off Actors  Lightweight concurrent entities  Each has a distinct state and behavior  Fail fast, recover quickly, can run remotely

5. Legacy @USGSTED Workflow

6. Reactive Approach  Akka – toolkit and runtime  Build highly concurrent, distributed, and fault tolerant event-driven applications  Small memory footprint; ~2.5 million actors per GB of heap  Apache Storm – distributed computation  consumes streams of data  processes streams in arbitrarily complex ways  repartitions streams between computation stages  Hadoop analog for streaming data

7. Storm Spouts and Bolts

8. @USGSTED Alternative Workflow

9. Extending USGS/DOI/Federal Capabilities  Provide @USGSTED partners with additional capabilities  FEMA needs beyond initial dispatch  Multiple social media feeds  Proximity sensors  Observation networks  Way to deal with almost any flood of distributed simultaneous signals