Presentation is loading. Please wait.

Presentation is loading. Please wait.

SSEC Satellite Data Services RabbitMQ Messaging System

Published byKoray Ceylan Modified over 6 years ago

Similar presentations

Presentation on theme: "SSEC Satellite Data Services RabbitMQ Messaging System"— Presentation transcript:

1 SSEC Satellite Data Services RabbitMQ Messaging System
Rick Kohrs Satellite Data Services Space Science & Engineering Center University of Wisconsin – Madison 2018 McIDAS Users’ Group Meeting 22-23 May 2018 Madison, WI

2 GOES-R Rebroadcast (GRB) ADDE SFTP/LDM NFS Ground Station Server Antenna Electronics Ingest System

3 What is RabbitMQ Benefits Satellite Data Services Paradigm Use Case

4 What is RabbitMQ Publisher Publisher Satellite Ingestor ADDE Server
Archive Server Product Generator

5 What is RabbitMQ Publisher (currently ~200 msgs/minute)
Broker Message Queue Messaging Queue Publisher Satellite Ingestor ADDE Server Archive Server Product Generator

6 What is RabbitMQ Publisher (currently ~200 msgs/minute)
Message Queue – 'The Broker' Client/Consumer Messaging Queue Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

7 What is RabbitMQ Publisher (currently ~200 msgs/minute) Message Queue
Client/Consumer image start band end image complete image complete image start Messaging Queue band end Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

8 What is RabbitMQ Publisher (currently ~200 msgs/minute) Message Queue
Client/Consumer image start band end image complete Advanced Message Queueing Protocol (AMQP) Pika – python implementation of AMQP image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

9 What is RabbitMQ Publisher (currently ~200 msgs/minute)
Exchange / Topic satellite Topic Keys sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

10 What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite
sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

11 What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite
sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

12 What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite
sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari g16 g17 h8 image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

13 What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite
sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari g16 g17 h8 abi glm exis magnetometer seiss suvi ahi image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

14 What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite
sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari g16 g17 h8 abi glm exis magnetometer seiss suvi ahi adde file image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

15 What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite
sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari g16 g17 h8 abi glm exis magnetometer seiss suvi ahi adde file sdi image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

16 What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite
sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari g16 g17 h8 abi glm exis magnetometer seiss suvi ahi adde file sdi ncdf hsf image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

17 What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite
sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari g16 g17 h8 abi glm exis magnetometer seiss suvi ahi adde file sdi ncdf hsf band image product granule image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

18 What is RabbitMQ Publisher Exchange / Topic Topic Keys satellite
sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo leo goes himawari g16 g17 h8 abi glm exis magnetometer seiss suvi ahi adde file sdi ncdf hsf band image product granule start end complete image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

19 What is RabbitMQ Publisher Exchange / Topic Topic Keys Body/Payload
satellite Topic Keys Body/Payload satellite_family: GOES satellite_location: GOES-East coverage: Mesoscale-1 satellite_ID: G16 start_time: :52:29.2 mode: 3 instrument: ABI end_time: :52:35.0 signal_type: grb medium: adde create_time: :52:39.3 data_type: RadM1 server_type: sdi server_ip: satbuf1.ssec.wisc.edu title: ABI L1b Radiances message_type: band adde_dataset: EASTA/M1 status: end band: 3 path: /data/goes/grb/goes16/2018/2018_04_30_120/abi/L1b/RadM1/ OR_ABI-L1b-RadM1-M3C03_G16_s _e _c nc image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

20 What is RabbitMQ Client Exchange / Topic Subscribe to Keys
amqpfind sat_type.sat_family.sat_ID.sat_instrument.medium.server_type.format.classification.status geo.goes.g16.abi.adde.sdi.ncdf.band.end image complete image start Messaging Queue pika AMQP band end pika AMQP Publisher Client Satellite Ingestor ADDE Server Archive Server Product Generator

21 RabbitMQ Benefits Avoid multiple data request cron
Immediate Access to data adde https No need for messages Client 00 12 06 18 new data available wait no Server adde

22 RabbitMQ Benefits Avoid multiple data request cron
Immediate Access to data adde https No need for messages Client 00 12 06 18 wait new data available Server adde no

23 RabbitMQ Benefits Avoid multiple data request cron
Immediate Access to data adde https No need for messages Client process data 00 12 06 18 no wait new data available Server adde yes

24 RabbitMQ Benefits Avoid multiple data request cron
Immediate Access to data adde https No need for messages Client process data 00 12 06 18 you've got data Broker Server adde

25 Satellite Data Service GOES-16 Data Flow
GRB RF demod CADU fanout 1

26 Satellite Data Services GOES-16 Data Flow
fanout 1 demod GRB RF CADU Satellite Data Services GOES-16 Data Flow SDI cspp-geo rsync SATBUF1

27 Satellite Data Services GOES-16 Data Flow
fanout 1 demod GRB RF CADU Satellite Data Services GOES-16 Data Flow SDI cspp-geo SATBUF1 rsync rsync arcwrite1 Lustre arcdata publish messages mq1 RabbitMQ server publish messages

28 Satellite Data Services GOES-16 Data Flow
mq1 RabbitMQ server publish messages rsync arcwrite1 Lustre arcdata fanout 1 demod GRB RF CADU Satellite Data Services GOES-16 Data Flow SDI cspp-geo SATBUF1 ADDE amqpfind Outside Users In-House nfs https/ADDE/nfs https/ADDE

29 Satellite Data Services GOES-16 Data Flow
fanout 1 demod GRB RF CADU Satellite Data Services GOES-16 Data Flow demod Amazon CADU fanout 2 rsync/nfs rsync/nfs SDI sdigrbbeta GRBDELTA GRBGAMMA csppalpha GEOCAT/AIT Level 2+ Products dbINGE cspp-geo cspp-geo dbINGE csppbeta GEOCAT/AIT Level 2+ Products score score Lustre AWG SDI sdigrbbeta publish messages publish messages rsync rsync publish messages rsync rsync rsync rsync nfs SATBUF1 SATBUF2 mq1 RabbitMQ server Lustre Users publish messages mq2 RabbitMQ server amqpfind ADDE nfs Outside Users In-House arcwrite1 arcwrite2 amqpfind ADDE In-House Users nfs https/ADDE/nfs Lustre arcdata Outside Users https/ADDE https/ADDE

30 Satellite Data Services GOES-16 Data Flow
fanout 1 demod GRB RF CADU Satellite Data Services GOES-16 Data Flow GRBGAMMA arcwrite2 SDI sdigrbbeta csppbeta GEOCAT/AIT Level 2+ Products fanout 2 CADU demod mq2 RabbitMQ server SATBUF2 rsync rsync/nfs Lustre AWG Users Outside In-House https/ADDE ADDE nfs publish messages NOAA Star NOAA PDA hwcloudtop2 rsync (push) Amazon cspp-geo dbINGE GRBDELTA score csppalpha hwcloudtop2 rsync (push) NOAA Star UNIDATA NOAA PDA SDI sdigrbbeta publish messages rsync rsync rsync rsync SATBUF1 mq1 RabbitMQ server ADDE nfs amqpfind amqpfind Outside Users In-House arcwrite1 https/ADDE/nfs Lustre arcdata https/ADDE

31 Satellite Data Services GOES-16 Data Flow
fanout 1 demod GRB RF CADU Satellite Data Services GOES-16 Data Flow GRBGAMMA arcwrite2 SDI sdigrbbeta csppbeta GEOCAT/AIT Level 2+ Products fanout 2 CADU demod mq2 RabbitMQ server SATBUF2 rsync rsync/nfs Lustre AWG Users Outside In-House https/ADDE ADDE nfs publish messages NOAA Star NOAA PDA hwcloudtop2 rsync (push) Amazon cspp-geo dbINGE GRBDELTA score csppalpha hwcloudtop2 rsync (push) NOAA Star UNIDATA NOAA PDA SDI sdigrbbeta publish messages rsync rsync rsync rsync SATBUF1 mq1 RabbitMQ server ADDE nfs amqpfind amqpfind Outside Users In-House arcwrite1 https/ADDE/nfs Lustre arcdata https/ADDE

32 Single Server Multi-Server
./amqpfind H mq2.ssec.wisc.edu -u sdsuser -p sdsmq \ -X satellite \ -C 'geo.goes.*.*.*.*.*.*.*' \ -j '{start_time} {coverage} {instrument} {message_type} {status}' Single Server ./amqpfind H mq2.ssec.wisc.edu -u sdsuser -p sdsmq \ -H mq1.ssec.wisc.edu -u sdsuser -p sdsmq \ -X satellite \ -w 60.0 \ -C 'geo.goes.*.*.*.*.*.*.*' \ -j '{start_time} {coverage} {instrument} {message_type} {status}' Multi-Server

33 Wrapper Script $HOME/amqpfind/amqpfind -X satellite -C 'geo.goes.*.*.*.*.*.image.complete' \ -H mq1.ssec.wisc.edu -p sdsmq -u sdsuser \ -j '{start_time} {server_ip} {adde_dataset}'|\ grep --line-buffered -i CONUS| \ xargs -P1 –n4 python $HOME/bin/mc-example.py

34 mc-example.py #! /usr/bin/python def main(cmdArgs): import mcidasx
import datetime imageDate = cmdArgs[1] imageTime = cmdArgs[2] server = cmdArgs[3] addeDataset = cmdArgs[4] group = addeDataset.split('/')[0] mcenv = i=228, g=8) mccmdout = mcenv.dataloc('ADD ' + group + ' ' + server) dateTimeStr = imageDate + ' ' + imageTime dateTimeObj = datetime.datetime.strptime(dateTimeStr,'%Y-%m-%d %H:%M:%S.%f') dateTimeKeyword = dateTimeObj.strftime(' DAY=%Y%j TIME=%H:%M:%S ') mcCmd = ' USER 1234' mcCmdOut = mcenv.logon(mcCmd) mcCmd = addeDataset + dateTimeKeyword + ' BAND=13' mcCmdOut = mcenv.imgdisp(mcCmd) mcCmd = ' X current-G16-band13.gif' mcCmdOut = mcenv.frmsave(mcCmd) if __name__ == '__main__': import sys main(sys.argv)

35 Thank You! Questions?

Download ppt "SSEC Satellite Data Services RabbitMQ Messaging System"

Similar presentations

An Erlang Implementation of Restms. Why have messaging? Separates applications cheaply Feed information to the right applications cheaply Interpret feed.

An Erlang Implementation of Restms. Why have messaging? Separates applications cheaply Feed information to the right applications cheaply Interpret feed.
GOES-R User Data Types and Structures

GOES-R User Data Types and Structures
McIDAS-Lite Dave Santek Program Manager Space Science & Engineering Center University of Wisconsin-Madison 5 December 2002.

McIDAS-Lite Dave Santek Program Manager Space Science & Engineering Center University of Wisconsin-Madison 5 December 2002.
SDI-104 Status Jerrold Robaidek & Dave Santek 9 September 2013.

SDI-104 Status Jerrold Robaidek & Dave Santek 9 September 2013.
1 Scott Gudes Vice President, Legislative Affairs Space Systems & Operations Interdepartmental Hurricane Conference March 2, 2011 W A S H I N G T O N O.

1 Scott Gudes Vice President, Legislative Affairs Space Systems & Operations Interdepartmental Hurricane Conference March 2, 2011 W A S H I N G T O N O.
Greg Mandt GOES-R System Program Director 2011 Satellite Direct Readout Conference April 5, 2011.

Greg Mandt GOES-R System Program Director 2011 Satellite Direct Readout Conference April 5, 2011.
Agency, version?, Date 2012 Coordination Group for Meteorological Satellites - CGMS Add CGMS agency logo here (in the slide master) Coordination Group.

Agency, version?, Date 2012 Coordination Group for Meteorological Satellites - CGMS Add CGMS agency logo here (in the slide master) Coordination Group.
Greg Mandt GOES-R System Program Director 2011 GOES-R Proving Ground Annual Meeting Boulder, CO May 17, 2011.

Greg Mandt GOES-R System Program Director 2011 GOES-R Proving Ground Annual Meeting Boulder, CO May 17, 2011.
An Overview of the GOES-R Program

An Overview of the GOES-R Program
Himawari-8 Project Plans and Status. Background 2 NESDIS is implementing a capability to ingest Advanced Himawari Imager data from JMA, process and generate.

Himawari-8 Project Plans and Status. Background 2 NESDIS is implementing a capability to ingest Advanced Himawari Imager data from JMA, process and generate.
Satellite Communication

Satellite Communication
SDI (Satellite Data Ingestor) 2015 McIDAS Users Group Meeting June 8, 2015.

SDI (Satellite Data Ingestor) 2015 McIDAS Users Group Meeting June 8, 2015.
Storm Tracker – a Web-Based Tool for Working with the Automated Tracking of Thunderstorm Clusters Bob Rabin NOAA / NSSL / CIMSS Tom Whittaker Space Science.

Storm Tracker – a Web-Based Tool for Working with the Automated Tracking of Thunderstorm Clusters Bob Rabin NOAA / NSSL / CIMSS Tom Whittaker Space Science.
1 GOES-R Update Greg Mandt System Program Director GOES User Conference 3 Nov 2009.

1 GOES-R Update Greg Mandt System Program Director GOES User Conference 3 Nov 2009.
McIDAS Advisory Committee (MAC) Sponsored: Group Discussion Joleen Feltz McIDAS Advisory Committee Chair from 2013-2015 Cooperative Institute for Meteorological.

McIDAS Advisory Committee (MAC) Sponsored: Group Discussion Joleen Feltz McIDAS Advisory Committee Chair from Cooperative Institute for Meteorological.
2013 GOES-R AWG 2 nd Validation Workshop 9 - 10 January 2014 Status of Activities and Plans for the GOES-R Core Ground System 1 Satya Kalluri Ph.D. GOES-R.

2013 GOES-R AWG 2 nd Validation Workshop January 2014 Status of Activities and Plans for the GOES-R Core Ground System 1 Satya Kalluri Ph.D. GOES-R.
Robin Pfister (GOES-R Ground Segment Deputy Project Manager)

Robin Pfister (GOES-R Ground Segment Deputy Project Manager)
Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.

Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.
Japan Meteorological Agency, November, 2012 Coordination Group for Meteorological Satellites - CGMS Status Report on the Current and Future Satellite Systems.

Japan Meteorological Agency, November, 2012 Coordination Group for Meteorological Satellites - CGMS Status Report on the Current and Future Satellite Systems.
Some Design Notes Iteration - 2 Method - 1 Extractor main program Runs from an external VM Listens for RabbitMQ messages Starts a light database engine.

Some Design Notes Iteration - 2 Method - 1 Extractor main program Runs from an external VM Listens for RabbitMQ messages Starts a light database engine.

Similar presentations

Ads by Google