SSEC Satellite Data Services RabbitMQ Messaging System

Slides:

Advertisements

Similar presentations

An Erlang Implementation of Restms. Why have messaging? Separates applications cheaply Feed information to the right applications cheaply Interpret feed.

Advertisements

GOES-R User Data Types and Structures

McIDAS-Lite Dave Santek Program Manager Space Science & Engineering Center University of Wisconsin-Madison 5 December 2002.

SDI-104 Status Jerrold Robaidek & Dave Santek 9 September 2013.

1 Scott Gudes Vice President, Legislative Affairs Space Systems & Operations Interdepartmental Hurricane Conference March 2, 2011 W A S H I N G T O N O.

Greg Mandt GOES-R System Program Director 2011 Satellite Direct Readout Conference April 5, 2011.

Agency, version?, Date 2012 Coordination Group for Meteorological Satellites - CGMS Add CGMS agency logo here (in the slide master) Coordination Group.

Greg Mandt GOES-R System Program Director 2011 GOES-R Proving Ground Annual Meeting Boulder, CO May 17, 2011.

An Overview of the GOES-R Program

Himawari-8 Project Plans and Status. Background 2 NESDIS is implementing a capability to ingest Advanced Himawari Imager data from JMA, process and generate.

Satellite Communication

SDI (Satellite Data Ingestor) 2015 McIDAS Users Group Meeting June 8, 2015.

Storm Tracker – a Web-Based Tool for Working with the Automated Tracking of Thunderstorm Clusters Bob Rabin NOAA / NSSL / CIMSS Tom Whittaker Space Science.

1 GOES-R Update Greg Mandt System Program Director GOES User Conference 3 Nov 2009.

McIDAS Advisory Committee (MAC) Sponsored: Group Discussion Joleen Feltz McIDAS Advisory Committee Chair from Cooperative Institute for Meteorological.

2013 GOES-R AWG 2 nd Validation Workshop January 2014 Status of Activities and Plans for the GOES-R Core Ground System 1 Satya Kalluri Ph.D. GOES-R.

Robin Pfister (GOES-R Ground Segment Deputy Project Manager)

Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.

Japan Meteorological Agency, November, 2012 Coordination Group for Meteorological Satellites - CGMS Status Report on the Current and Future Satellite Systems.

Some Design Notes Iteration - 2 Method - 1 Extractor main program Runs from an external VM Listens for RabbitMQ messages Starts a light database engine.

Breakout Session Topic: Atmosphere A and B 1.During the GOES-R era, NOAA/NESDIS hopes to move away from system “stovepipes” and into the realm of integrated.

2015 AMS Annual Meeting Non Export-Controlled Information Progress in Development and Integration of the Product Generation Capabilities of the GOES- R.

2015 OCONUS Meeting Non Export-Controlled Information Progress in Development and Integration of the Product Generation Capabilities of the GOES-R Ground.

Cooperative Institute for Meteorological Satellite Studies University of Wisconsin - Madison ABI and AIRS Retrievals in McIDAS-V Kaba Bah.

Geo-distributed Messaging with RabbitMQ

SDI-104 Status Dave Santek SDI-104 Program Manager 2 June 2009.

AMQP, Message Broker Babu Ram Dawadi. overview Why MOM architecture? Messaging broker like RabbitMQ in brief RabbitMQ AMQP – What is it ?

Spring RabbitMQ Martin Toshev.

What does it cover? This session addresses “Why?”, “When?”, and “What Sensors?” will be on GOES- R, and presents examples of what to expect. If is a look.

CGMS-43-NOAA-WP-03 Coordination Group for Meteorological Satellites - CGMS PDA Product Distribution & Acquisition by NOAA Presented to CGMS-43 Plenary.

Agency, version?, Date 2012 Coordination Group for Meteorological Satellites - CGMS Add CGMS agency logo here (in the slide master) Coordination Group.

CGMS-43-NOAA-WP-10 Coordination Group for Meteorological Satellites - CGMS NOAA Transition to New Direct Readout Systems for GOES-R & JPSS NOAA Transition.

McIDAS Transition Summary Tom Achtor McIDAS Users’ Group Meeting Madison, WI - 17 October 2007 Space Science & Engineering Center (SSEC) at the University.

Status of JMA’s activities on Virtual Laboratory Fifth Virtual Laboratory Management Group (VLMG-5) WMO-CGMS Virtual Laboratory for Education and Training.

GOES-R Product Distribution Logistics Before and After Handover Kathryn Miretzky GOES-R User Services Coordinator Product Readiness and Operations (PRO)

Band 14 (11um) Winds Low-Level >700 mb Mid-Level mb High-Level mb   NPP VIIRS Polar Winds Products The GOES-R AWG Derived Motion Winds.

CGMS-43-NOAA-WP-02 Coordination Group for Meteorological Satellites - CGMS Transition from GVAR to GOES-R GRB – Service Continuity Presented to CGMS-43.

Matt Seybold (OSPO/SPSD) GOES-R Data Operations Manager & PRO Team Lead NOAA Satellite Proving Ground and User Readiness Meeting May 9, /9/2016NOAA.

GOES-R in Max The benefits are well known, 5x faster data, 4x better resolution, ~3x more spectral channels What challenges does this new data present?

Messaging in Distributed Systems

GOES Data Status Mutual Benefits of NASA THEMIS and NOAA GOES

NWA Annual Meeting Joint Satellite Workshop

SCALE: The Safe Community Awareness and Alerting Network

Fangfang Yu, Fuzhong Weng, Xiangqian Wu, and Ninghai Sun

GOES-R Resources from NOAA, NASA & CIMSS

4:40 pm EST Wednesday November 16th

SDI (Satellite Data Ingestor)

NESDIS-JMA AHI Collaboration

NOAA Transition to New Direct Readout Systems for GOES-R Presented to CGMS-44 WGI session, agenda item 3.2.

The Road to Using Python with McIDAS-X

GOES-R Observational Requirements: Alternative 1 (no sounder)

Satellite Data Resources: Browsing and Accessing Archived Datasets

IP Publishing From IP Data Base to IP list to IP catalog

Who We Are SSEC (Space Science and Engineering Center) is part of the Graduate School of the University of Wisconsin-Madison (UW). SSEC hosts CIMSS (Cooperative.

14-мавзу. Cookie, сеанс, FTP и технологиялари

GOES-R Hyperspectral Environmental Suite (HES) Requirements

Tool for Storm Analysis Using Multiple Data Sets

Replication Middleware for Cloud Based Storage Service

SDI-104 Status Dave Santek SDI-104 Program Manager 25 October 2010.

CMIP6 / ENES Data TF Meeting: DKRZ

Microservices, Messaging and Science Gateways

Jerrold Robaidek & Dave Santek

Scott Mindock, Jonathan Beavers, Rick Kohrs

Evolution of messaging systems and event driven architecture

Enterprise Infrastructure

McIDAS-X Software Development and Demonstration

Meteorological Satellite Center, Japan Meteorological Agency

David Santek 2018 McIDAS Users’ Group Meeting 22 May 2018

McIDAS Advisory Committee (MAC) Update and Group Discussion

Securing HTCondor Flocking

Presentation transcript:

SSEC Satellite Data Services RabbitMQ Messaging System Rick Kohrs Satellite Data Services Space Science & Engineering Center University of Wisconsin – Madison 2018 McIDAS Users’ Group Meeting 22-23 May 2018 Madison, WI

GOES-R Rebroadcast (GRB) ADDE SFTP/LDM NFS Ground Station Server Antenna Electronics Ingest System

What is RabbitMQ Benefits Satellite Data Services Paradigm Use Case

What is RabbitMQ Publisher Publisher Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Publisher (currently ~200 msgs/minute) Broker Message Queue Messaging Queue Publisher Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Publisher (currently ~200 msgs/minute) Message Queue – 'The Broker' Client/Consumer Messaging Queue Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Publisher (currently ~200 msgs/minute) Message Queue Client/Consumer image start band end image complete image complete image start Messaging Queue band end Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Publisher (currently ~200 msgs/minute) Message Queue Client/Consumer image start band end image complete Advanced Message Queueing Protocol (AMQP) Pika – python implementation of AMQP image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Publisher (currently ~200 msgs/minute) Exchange / Topic satellite Topic Keys sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari g16 g17 h8 image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari g16 g17 h8 abi glm exis magnetometer seiss suvi ahi image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari g16 g17 h8 abi glm exis magnetometer seiss suvi ahi adde file image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari g16 g17 h8 abi glm exis magnetometer seiss suvi ahi adde file sdi image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari g16 g17 h8 abi glm exis magnetometer seiss suvi ahi adde file sdi ncdf hsf image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari g16 g17 h8 abi glm exis magnetometer seiss suvi ahi adde file sdi ncdf hsf band image product granule image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari g16 g17 h8 abi glm exis magnetometer seiss suvi ahi adde file sdi ncdf hsf band image product granule start end complete image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Publisher Exchange / Topic Topic Keys Body/Payload satellite Topic Keys Body/Payload satellite_family: GOES satellite_location: GOES-East coverage: Mesoscale-1 satellite_ID: G16 start_time: 2018-04-30 19:52:29.2 mode: 3 instrument: ABI end_time: 2018-04-30 19:52:35.0 signal_type: grb medium: adde create_time: 2018-04-30 19:52:39.3 data_type: RadM1 server_type: sdi server_ip: satbuf1.ssec.wisc.edu title: ABI L1b Radiances message_type: band adde_dataset: EASTA/M1 status: end band: 3 path: /data/goes/grb/goes16/2018/2018_04_30_120/abi/L1b/RadM1/ OR_ABI-L1b-RadM1-M3C03_G16_s20181201952292_e20181201952350_c20181201952393.nc image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

What is RabbitMQ Client Exchange / Topic Subscribe to Keys amqpfind http://www.ssec.wisc.edu/datacenter/amqpfind/ sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo.goes.g16.abi.adde.sdi.ncdf.band.end image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

RabbitMQ Benefits Avoid multiple data request cron Immediate Access to data adde https No need for e-mail messages Client 00 12 06 18 new data available wait no Server adde

RabbitMQ Benefits Avoid multiple data request cron Immediate Access to data adde https No need for e-mail messages Client 00 12 06 18 wait new data available Server adde no

RabbitMQ Benefits Avoid multiple data request cron Immediate Access to data adde https No need for e-mail messages Client process data 00 12 06 18 no wait new data available Server adde yes

RabbitMQ Benefits Avoid multiple data request cron Immediate Access to data adde https No need for e-mail messages Client process data 00 12 06 18 you've got data Broker Server adde

Satellite Data Service GOES-16 Data Flow GRB RF demod CADU fanout 1

Satellite Data Services GOES-16 Data Flow fanout 1 demod GRB RF CADU Satellite Data Services GOES-16 Data Flow SDI cspp-geo rsync SATBUF1

Satellite Data Services GOES-16 Data Flow fanout 1 demod GRB RF CADU Satellite Data Services GOES-16 Data Flow SDI cspp-geo SATBUF1 rsync rsync arcwrite1 Lustre arcdata publish messages mq1 RabbitMQ server publish messages

Satellite Data Services GOES-16 Data Flow mq1 RabbitMQ server publish messages rsync arcwrite1 Lustre arcdata fanout 1 demod GRB RF CADU Satellite Data Services GOES-16 Data Flow SDI cspp-geo SATBUF1 ADDE amqpfind Outside Users In-House nfs https/ADDE/nfs https/ADDE

Satellite Data Services GOES-16 Data Flow fanout 1 demod GRB RF CADU Satellite Data Services GOES-16 Data Flow demod Amazon CADU fanout 2 rsync/nfs rsync/nfs SDI sdigrbbeta GRBDELTA GRBGAMMA csppalpha GEOCAT/AIT Level 2+ Products dbINGE cspp-geo cspp-geo dbINGE csppbeta GEOCAT/AIT Level 2+ Products score score Lustre AWG SDI sdigrbbeta publish messages publish messages rsync rsync publish messages rsync rsync rsync rsync nfs SATBUF1 SATBUF2 mq1 RabbitMQ server Lustre Users publish messages mq2 RabbitMQ server amqpfind ADDE nfs Outside Users In-House arcwrite1 arcwrite2 amqpfind ADDE In-House Users nfs https/ADDE/nfs Lustre arcdata Outside Users https/ADDE https/ADDE

Satellite Data Services GOES-16 Data Flow fanout 1 demod GRB RF CADU Satellite Data Services GOES-16 Data Flow GRBGAMMA arcwrite2 SDI sdigrbbeta csppbeta GEOCAT/AIT Level 2+ Products fanout 2 CADU demod mq2 RabbitMQ server SATBUF2 rsync rsync/nfs Lustre AWG Users Outside In-House https/ADDE ADDE nfs publish messages NOAA Star NOAA PDA hwcloudtop2 rsync (push) Amazon cspp-geo dbINGE GRBDELTA score csppalpha hwcloudtop2 rsync (push) NOAA Star UNIDATA NOAA PDA SDI sdigrbbeta publish messages rsync rsync rsync rsync SATBUF1 mq1 RabbitMQ server ADDE nfs amqpfind amqpfind Outside Users In-House arcwrite1 https/ADDE/nfs Lustre arcdata https/ADDE

Satellite Data Services GOES-16 Data Flow fanout 1 demod GRB RF CADU Satellite Data Services GOES-16 Data Flow GRBGAMMA arcwrite2 SDI sdigrbbeta csppbeta GEOCAT/AIT Level 2+ Products fanout 2 CADU demod mq2 RabbitMQ server SATBUF2 rsync rsync/nfs Lustre AWG Users Outside In-House https/ADDE ADDE nfs publish messages NOAA Star NOAA PDA hwcloudtop2 rsync (push) Amazon cspp-geo dbINGE GRBDELTA score csppalpha hwcloudtop2 rsync (push) NOAA Star UNIDATA NOAA PDA SDI sdigrbbeta publish messages rsync rsync rsync rsync SATBUF1 mq1 RabbitMQ server ADDE nfs amqpfind amqpfind Outside Users In-House arcwrite1 https/ADDE/nfs Lustre arcdata https/ADDE

Single Server Multi-Server ./amqpfind -H mq2.ssec.wisc.edu -u sdsuser -p sdsmq \ -X satellite \ -C 'geo.goes.*.*.*.*.*.*.*' \ -j '{start_time} {coverage} {instrument} {message_type} {status}' Single Server ./amqpfind -H mq2.ssec.wisc.edu -u sdsuser -p sdsmq \ -H mq1.ssec.wisc.edu -u sdsuser -p sdsmq \ -X satellite \ -w 60.0 \ -C 'geo.goes.*.*.*.*.*.*.*' \ -j '{start_time} {coverage} {instrument} {message_type} {status}' Multi-Server

Wrapper Script $HOME/amqpfind/amqpfind -X satellite -C 'geo.goes.*.*.*.*.*.image.complete' \ -H mq1.ssec.wisc.edu -p sdsmq -u sdsuser \ -j '{start_time} {server_ip} {adde_dataset}'|\ grep --line-buffered -i CONUS| \ xargs -P1 –n4 python $HOME/bin/mc-example.py

mc-example.py #! /usr/bin/python def main(cmdArgs): import mcidasx import datetime imageDate = cmdArgs[1] imageTime = cmdArgs[2] server = cmdArgs[3] addeDataset = cmdArgs[4] group = addeDataset.split('/')[0] mcenv = mcidasx.mcenv(f=['3@750x1250'], i=228, g=8) mccmdout = mcenv.dataloc('ADD ' + group + ' ' + server) dateTimeStr = imageDate + ' ' + imageTime dateTimeObj = datetime.datetime.strptime(dateTimeStr,'%Y-%m-%d %H:%M:%S.%f') dateTimeKeyword = dateTimeObj.strftime(' DAY=%Y%j TIME=%H:%M:%S ') mcCmd = ' USER 1234' mcCmdOut = mcenv.logon(mcCmd) mcCmd = addeDataset + dateTimeKeyword + ' BAND=13' mcCmdOut = mcenv.imgdisp(mcCmd) mcCmd = ' X current-G16-band13.gif' mcCmdOut = mcenv.frmsave(mcCmd) if __name__ == '__main__': import sys main(sys.argv)

Thank You! Questions?