ANF Detection Network Dave Foster Guy de Wardener

Resources Confluence page: splay/CIDev/ANF-SQLstream+Integration splay/CIDev/ANF-SQLstream+Integration ANF Detection Network code: network network

Introduction / Goals Using elastic processing (cloud computing) to operate on seismic data Drive development and test ION ( splay/CIDev/Home) Data from Array Network Facility ( Uses SQLstream ( for processing for real time processinghttp://sqlstream.com/

(con’t) Continuation of POC/Prototype (Aug 2009, splay/CIDev/SQLstream+Prototype) splay/CIDev/SQLstream+Prototype

Architecture

(con’t) ANF Application Controller – Part of four op units in a deployment – Listens for instrument announcements, partitions those instruments into AMQP queues, requests SQLstream worker op units be created ANF Application Agent – Runs on SQLstream worker op unit – Manages SQLstream instances, communicates status with Application Controller

AppAgent Installs/configures SQLstream on demand from AppController – Allows us to use common base image – More than one SQLstream instance (listening to different queues) can be run on a worker op unit – Downsides: time spent transferring installer from S3, time spent installing SQLstream

SQLstream Listens to AMQP queue assigned to it by App Controller Transmits detection events over AMQP topic “anf.detections”

Deployment Uses CEI developed tools cloudinit.d, epumgmt – cloudinit.d: – epumgmt: Actual startup commands: –cloudinitd boot launch-plans/anf/main.conf -n anf -v -v -f /tmp/anf.log –epumgmt.sh -a find-workers -n anf --wholerun –epumgmt.sh -a logfetch -n anf --wholerun

Demo Live system – 30 stations Captured Data – 10 stations, showing events

Metrics Cloudinit.d (6:30-9:30) Worker Booted (~ 2:52) SQLstream running (~ 4:58) Instrument Announce ~ 7:20 startup

(con’t) Install Ready DefinedRunning 36s 12.5s 44.47s175.40s (2:55) 4:58 SQLstream running

Future Goals Data Preservation – Detections, possibly input events Service Hardening – Distributed, stateless processes Replacement for instrument announcement True load balancing – Need more metrics with live data

(con’t) Monitoring – Graphical dashboard showing SQLstream op units, queues/bindings, consumption rates