1. ANF Detection Network Dave Foster dfoster@asascience.comdfoster@asascience.com Guy de Wardener gdewardener@asascience.com gdewardener@asascience.com

2. Resources Confluence page: https://confluence.oceanobservatories.org/di splay/CIDev/ANF-SQLstream+Integration https://confluence.oceanobservatories.org/di splay/CIDev/ANF-SQLstream+Integration ANF Detection Network code: https://github.com/daf/anf-detection- network https://github.com/daf/anf-detection- network

3. Introduction / Goals Using elastic processing (cloud computing) to operate on seismic data Drive development and test ION (https://confluence.oceanobservatories.org/di splay/CIDev/Home) Data from Array Network Facility (http://anf.ucsd.edu/)http://anf.ucsd.edu/ Uses SQLstream (http://sqlstream.com/) for processing for real time processinghttp://sqlstream.com/

4. (con’t) Continuation of POC/Prototype (Aug 2009, https://confluence.oceanobservatories.org/di splay/CIDev/SQLstream+Prototype) https://confluence.oceanobservatories.org/di splay/CIDev/SQLstream+Prototype

5. Architecture

6. (con’t) ANF Application Controller – Part of four op units in a deployment – Listens for instrument announcements, partitions those instruments into AMQP queues, requests SQLstream worker op units be created ANF Application Agent – Runs on SQLstream worker op unit – Manages SQLstream instances, communicates status with Application Controller

7. AppAgent Installs/configures SQLstream on demand from AppController – Allows us to use common base image – More than one SQLstream instance (listening to different queues) can be run on a worker op unit – Downsides: time spent transferring installer from S3, time spent installing SQLstream

8. SQLstream Listens to AMQP queue assigned to it by App Controller Transmits detection events over AMQP topic “anf.detections”

9. Deployment Uses CEI developed tools cloudinit.d, epumgmt – cloudinit.d: https://github.com/nimbusproject/cloudinit.d https://github.com/nimbusproject/cloudinit.d – epumgmt: https://github.com/nimbusproject/epumgmt https://github.com/nimbusproject/epumgmt Actual startup commands: –cloudinitd boot launch-plans/anf/main.conf -n anf -v -v -f /tmp/anf.log –epumgmt.sh -a find-workers -n anf --wholerun –epumgmt.sh -a logfetch -n anf --wholerun

10. Demo Live system – 30 stations Captured Data – 10 stations, showing events

11. Metrics Cloudinit.d (6:30-9:30) Worker Booted (~ 2:52) SQLstream running (~ 4:58) Instrument Announce ~ 7:20 startup

12. (con’t) Install Ready DefinedRunning 36s 12.5s 44.47s175.40s (2:55) 4:58 SQLstream running

13. Future Goals Data Preservation – Detections, possibly input events Service Hardening – Distributed, stateless processes Replacement for instrument announcement True load balancing – Need more metrics with live data

14. (con’t) Monitoring – Graphical dashboard showing SQLstream op units, queues/bindings, consumption rates