1. ANF Detection Network Dave Foster dfoster@asascience.comdfoster@asascience.com Guy de Wardener gdewardener@asascience.com gdewardener@asascience.com

2. Resources Confluence page: https://confluence.oceanobservatories.org/di splay/CIDev/ANF-SQLstream+Integration https://confluence.oceanobservatories.org/di splay/CIDev/ANF-SQLstream+Integration ANF Detection Network code: https://github.com/daf/anf-detection- network https://github.com/daf/anf-detection- network

3. Array Network Facility Deployment

4. ANF Event Detection thresh threshoff ratio1 ratio2 filtered data STA LTA detection on detection off onset (pick) time LTA STA look for onset time here compute ratio2 noise floor here

5. Introduction / Goals Using elastic processing (cloud computing) to operate on seismic data Drive development and test ION (https://confluence.oceanobservatories.org/di splay/CIDev/Home) Data from Array Network Facility (http://anf.ucsd.edu/)http://anf.ucsd.edu/ Uses SQLstream (http://sqlstream.com/) for processing for real time processinghttp://sqlstream.com/

6. (con’t) Continuation of POC/Prototype (Aug 2009, https://confluence.oceanobservatories.org/di splay/CIDev/SQLstream+Prototype) https://confluence.oceanobservatories.org/di splay/CIDev/SQLstream+Prototype

7. Architecture

8. (con’t) ANF Application Controller – Part of four op units in a deployment – Listens for instrument announcements, partitions those instruments into AMQP queues, requests SQLstream worker op units be created ANF Application Agent – Runs on SQLstream worker op unit – Manages SQLstream instances, communicates status with Application Controller

9. AppAgent Installs/configures SQLstream on demand from AppController – Allows us to use common base image – More than one SQLstream instance (listening to different queues) can be run on a worker op unit – Downsides: time spent transferring installer from S3, time spent installing SQLstream

10. SQLstream Listens to AMQP queue assigned to it by App Controller Transmits detection events over AMQP topic “anf.detections”

11. Deployment Uses CEI developed tools cloudinit.d, epumgmt – cloudinit.d: https://github.com/nimbusproject/cloudinit.d https://github.com/nimbusproject/cloudinit.d – epumgmt: https://github.com/nimbusproject/epumgmt https://github.com/nimbusproject/epumgmt Actual startup commands: –cloudinitd boot launch-plans/anf/main.conf -n anf -v -v -f /tmp/anf.log –epumgmt.sh -a find-workers -n anf --wholerun –epumgmt.sh -a logfetch -n anf --wholerun

12. Demo Live system – 30 stations Captured Data – 10 stations, showing events

13. Metrics Cloudinit.d (6:30-9:30) Worker Booted (~ 2:52) SQLstream running (~ 4:58) Instrument Announce ~ 7:20 startup

14. (con’t) Install Ready DefinedRunning 36s 12.5s 44.47s175.40s (2:55) 4:58 SQLstream running

15. 15 Operational Architecture (OV-2)

16. Deployment Architecture (OV-2) EPU (Elastic Processing Unit)

17. Future Goals Data Preservation – Detections, possibly input events Service Hardening – Distributed, stateless processes Replacement for instrument announcement True load balancing – Need more metrics with live data

18. (con’t) Monitoring – Graphical dashboard showing SQLstream op units, queues/bindings, consumption rates