Presentation is loading. Please wait.

Presentation is loading. Please wait.

LRGS & OPENDCS NRT Data Acquisition Systems

Similar presentations


Presentation on theme: "LRGS & OPENDCS NRT Data Acquisition Systems"— Presentation transcript:

1 LRGS & OPENDCS NRT Data Acquisition Systems
Enterprise Data and Information Management Section Informatics Branch, Cooperate Services Division Alberta Environment and Sustainable Resource Development March 11, 2015 WISKI Conference, Edmonton

2 Enterprise Data & Information Management Staff
Vacant Vacant Bob Werner Vacant

3 NRT Data Acquisition Team
Bob Werner Today we are focused on the NRT/Historical Water Data Group consisting of four members with Adam Minke the newest who started focus is on this smaller group. Most of them are here today and I hope you have a chance to meet and talk to them.

4 Near-Real-Time Monitoring
Hydrometric/Lakes/Ice Water level Stream Flow Storage Meteorological Precipitation Air temperature Snow depth/water Wind speed/direction Humidity Water Quality Conductivity Dissolved oxygen pH Turbidity Ground Water Temperature

5 Data Acquisition Group Responsibilities
Manage the real-time data collection systems (LRGS, OPENDCS) Provide data support for Field Staff (GOES, problems log) Provide 24x7 NRT support during major events (ice breakup, floods) Provide access to hydrometric, meteorological, groundwater, & water quality data Archive historical ESRD data Main responsibilities are:

6 WISKI: ESRD Enterprise System
WISKI serves many users & functions within ESRD Primary Users: River Forecast & Engineering Dam Operations WISKI allows: Modeling Graphing Export & Analysis NRT & Archival Continuing to expand…

7 Data Collection Working Together
External Partners Monitoring Data Management SDO WISKI Support River Engineering River Forecasting Dam Operations Irrigation Operations Public Field Office Results

8 Data Acquisition (LRGS/OPENDCS)
Data Management Activities Data Acquisition (LRGS/OPENDCS) Decoding Data validity checks Sensors verification Unit conversions Time conversions Time adjustments WISKI Database Create stations Insert/update metadata Create and assign timeseries Create aggregations and processing time series Input monitoring visits and apply corrections Verify data accuracy

9 Data Acquisition Alerts and Tools
Our system requires numerous alerts and tools to pre-emptively discover and resolve issues Broken decodes Lack of platform connection and/or data currency System alerts Change decoding script as required Test connection and troubleshoot a solution Dashboard to identify overall health of system

10 Data Sources

11 Realtime Data Communications
Satellite DCPs (450) Phone Loggers (115) Webpage Data Mining (36) SCADA Server (44) SFTP Servers (15) WISKI Breakdown of what data we collect. FIRES Meteorological (250) IP Connect (88) Bulk Historical Data Products

12 Incoming Data Formats RAW MODEM OUTPUT GOES BINARY
21,2013,279,1200,1393,-1.265,-.527,0,.316,1.265,1.265,1.265,1.265,1.265,1.265,1.265,1.265,1.265,1.16,.633,.105,0,0,0,0,0,0,0,-.105,-.105,-.843,-.843,-.738,-1.16,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-2.003,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.636,-3.584,-3.69 19,2013,279,1300,1393,23.52,100,12.91,12.86,5.649,73.8,7.4,9.26,73.8,5.64,59.46,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,1.775,11.44,92.7,13.1,334.5,334.5,334.5,334.5,0,2.759,0,2.385,0,1.704,0,1.602,0,334.4,334.5,334.5,334.5,5.435,5.366,5.292,5.354,0,-.009,5.626,5.52,7.36,7.28,9.12,9.21,5.581,5.461,333.5,331,333.5,331,333.5,331,333.5,331,333.4,330.8,333.4,330.8,333.4,330.8,333.4,330.8,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,2193,0,358,0,358,.088,0,358,0,1200,358,0,358,0,358 GOES BINARY A ?30-0HF016WFF00087`B1ECAjCAeCAdCAdCA`Caas0F5qa?>+rq9V)n]1j;ik$}.5:=Kf5RKPKFTD#H3d(i**+{*m& GOES 5 MINUTE DATA – HUMAN READABLE FORMAT 4555E4EE G35+1NN167EXE :HG 2 # :VB 17 # :YP 17 # TAU INCOMING TEXT FILE Bighorn ,02-Dec ,10:00:00 , , , , , , Brazeau ,02-Dec ,10:00:00 , , , , , , Nakiska ,02-Dec ,10:00:00 , , , , , , Rose_Creek ,02-Dec ,10:00:00 , , , , , , 0 Canmore ,02-Dec ,10:00:00 , , , , , , Spray ,02-Dec ,11:00:00 , , , , , , WEB PAGE DATA MINE FROM INTERNET <HTML><HEAD><TITLE>Hourly Hydromet Data for Lake Sherburne, Swift Current Creek near Babb, MT</TITLE></HEAD><BODY><PRE> SHER |13DEC03| FB | AF | OBG | OBT | RB | RD | QR | QS | Q | | 00:00 | | | | | | | | | | | 00:15 | | | | | | | | | | INTERNET TRANSFER ATE/TIME (yyyy/mm/dd) AirTemp Precip Bat vol (standard time) (C.) (mm) (volts) 2013/11/30 00:24: 2013/11/30 01:18: 2013/11/30 01:24: 2013/11/30 02:18: ENVIRONMENT CANADA STANDARD NATIONAL OUTPUT FOR MODEM

13 System History – 1984 to 2015 1980’s STREAM/COLLECT System on VAX
Rafi Morgantini and Lenard Cheng Project Managers, Alberta Environment, Systems and Computing STREAM/COLLECT system on VAX, 12th Floor Oxbridge Late 1980’s: Moved to RFC 11th floor and managed by Thai Nguyen and Ian Barnaby after the dissolution of Systems and Computing 1990’s DACQ/COLLECT System Data General DACQ Database System & COLLECT Acquisition COLLECT Acquisition acquired GOES via dialup Telnet, manual data inputs via dialup Telemark and automated tone encoder Telemark2 for ~ 45 stations and stored on DACQ system. Here’s shows an outline of our present system 2000 NewLeaf and COLLECT Red Hat UNIX DACQ decommissioned prior to Y2K and COLLECT now independent of the new database application called NewLeaf (Ingres). Eventually Data General was decommissioned and COLLECT moved to Red Hat UNIX server (Allister Chisholm).

14 System History – 1984 to 2011 COLLECT and Acquisition Modules This period constitutes a major growth of our NRT network via GOES, MSC dialup modems, BCHydro, TransAlta, Merchant, Scada, SnoTel, AutoPoll, Hydras3, and LoggerNet. This involved the deployment of VM technology that provided a scheduled poll of each data steam as required. 2001 Local Readout Ground Station (LRGS) Dialup via Telnet was eventually discontinued and the JAVA open-source software provided by USGS was deployed to interface with COLLECT. 2005 LRGS (DELL) and DCS-Toolkit (XML) The USGS consultant, Michael Maloney (ILEX Engineering) was hired to install a server and several licensed copies of the DCS-Toolkit. This provided DM with the tools necessary for writing decoding scripts for dozens of new types of messages, including pseudo-binary. Here’s shows an outline of our present system 2011 LRGS & DCS-Toolkit Upgrade (Oracle) ILEX Engineering was hired to convert XML to Oracle and provide clients to desktops.

15 System Timeline – 2014 to 2015 2014 COLLECT Replacement – Phase 1
All hardcoded decoding scripts embedded into COLLECT replaced by DECODES. This is essential, as decoding scripts on UNIX are no longer supported to provide enhancements and/or the creation of new decodes based on new platforms or firmware upgrades. 2015 COLLECT Replacement – Phase 2 LRGS is upgraded to handle EDL (electronic data logger) and other non-GOES data inputs. Also, the dialup modem polling is rewritten to utilize DIGI ConnectPort device. January, 2015 LRIT Satellite Project Here’s shows an outline of our present system April, 2015 LRGS/OPENDCS Acceptance Testing Both Phase 1 and 2 are currently being UAT tested till the end of the contract period. Once complete, we will be running LRGS version 9 and OPENDCS RC 4.0. May, 2015 LRGS/OPENDCS PROD Launch After the ICE season and before the spring runoff, a decision will be made to go live. If so, COLLECT will be engaged as a backup system.

16 Legacy System Configuration Phase 1 and 2 Identified
Here’s shows an outline of our present system

17 NRT Data Acquisition LRGS/OPENDCS Final State

18 What is LRGS? Local Readout Ground Station is a server that can be run on any operating system, as it is written in JAVA. UNIX is our preferred operating system, however this can be changed depending on our system support. Non-GOES Messages OPENDCS

19 DOMSAT Local Readout Ground Station is a server that can be run on any operating system, as it is written in JAVA. UNIX is our preferred operating system, however this can be changed depending on our system support. Non-GOES Messages OPENDCS

20 DOMSAT Receiver Decommissioning service by 2015/16
Domestic Communications satellite is a single high-speed channel that NOAA rebroadcasts to, allowing the entire DCS (Data Collection System) GOES messages for consumption by users. ERSD owns a dish a primary receiver on top of Oxbridge tower and a backup site, at EMA (Emergency Management Alberta) in Edmonton. This service will be decommissioned by the end of 2015.

21 DOMSAT Receiver Decommissioning service by 2015/16
DCP’s transmit to either GOES W or GOES E and captured at NOAA, which in turn are broadcast to DOMSAT (AMC-4) to our DOMSAT receiver. These messages are then captured on our LRGS server. LRGS

22 LRIT Receiver Installed on January, 2015, Oxbridge Place
Our first LRIT satellite service was installed on January, 2015 and will become our primary source of GOES messages once DOMSAT is decommissioned. Messages are rebroadcast to both GOES E and GOES W. A secondary site is planned for the year end by our Provincial Telecom Coordinator, Forestry Telecommunications Shop, ESRD. Non-GOES Messages OPENDCS

23 LRIT Receiver Installed on January, 2015, Oxbridge Place
The GOES messages are received at NOAA and rebroadcast back to GOES East and GOES West. Messages are then relayed back to an LRIT receiver which we now own on Oxbridge Place.

24 Satellite Antennas @ Oxbridge
Currently, our newly installed LRIT antenna is pointing at GOES East (SE direction).

25 DCS via Internet As a backup, should messages fail via our satellite receivers, a redundancy is built into the system by the internet. We current use the internet feed from Wallops Command and Data Acquisition Station (WCDAS) near Chincoteague, Virginia, and Emergency Data Distribution Network (EDDN) Sioux Falls, South Dakota. Non-GOES Messages OPENDCS

26 Wallops Command Centre
WCDAS Statistics Annual Funding at Wallops (NOAA Appropriations): Approximately $11 million Workforce: 91 (mixture of Government and contractors) Number of buildings and square footage: Seven buildings with 48,920 square feet (not including antenna shelters) Number of antennas: 16 Data rate throughput: Currently 41.7 megabits/second Date NOAA occupied site: Original permit signed on February 2, 1965 and dedicated on: June 22, 1966

27 Non-GOES Messages FTP, HTTP, SQL, Dial-up Modem
Since the completing of the COLLECT replacement project, phase 2, we now have the capacity to acquire all non-GOES messages, except for stations currently using Campbell Scientific LoggerNet software. Additional work is required to integrate LoggerNet API into LRGS (2015/16). Non-GOES Messages OPENDCS

28 Non-GOES Messages FTP, HTTP, SQL, Dial-up Modem
LoggerNet Campbell Scientific Software IP Protocol and PakBus platforms require this software. Phase 3 (2015/16) will integrate the LoggerNet API into LRGS, thereby completing the total integration of non-GOES messages. Phase 2

29 Non-GOES Messages Dial-up Modem
One major improvement allows for LRGS to function with modems. We are retiring our Sangoma cards that plugged directly into our UNIX server for a device that sits on the Ethernet. 16 US Robotics modems Sangoma Card (legacy) – prevented us from upgrading our OS and server. DIGI LTS ConnectPort 32 allows for all 16 modems to have a telnet port on the internet available to the LRGS server.

30 Why still dialup modems?
We currently still maintain 115 sites with dialup modems that are decreasing at a rate of ~ per year now for 3 years. Our peak was 145 stations. We predict that within the next 6 years, this number will drop of significantly. The cost of providing copper connection to remote stations is becoming increasingly more expensive and maintenance of existing lines, more probiotic. Providers are increasingly switching to either cellular or GOES.

31 OPENDCS We started with the raw messages that are acquired by LRGS (GOES and non-GOES) and will end this presentation with the second part, that of the OPENDCS (red box). The other outputs that we own and operate are, Web DCP Monitor, Message Browser (integrated into OPENDCS) and the DCS Toolkit currently used in production (blue boxes). Non-GOES Messages OPENDCS

32 OPENDCS The OPENDCS structure is similar to DCS Toolkit currently being used in production. The reasons that make this software valuable are: Designed in JAVA, which allows for cross platform use Utilizes DECODES format structure to decode messages Configuration XML easily shared with other NRT acquisition members OPEN means, open source and therefore will not require a separate licence for each installation RC03 denotes the release is Canadian (RC), specific to ESRD requirements All station configurations formerly contained on several flat files in COLLECT are now stored in Oracle and more easily backed up and restored when required Software is easily upgraded on workstations Decoding changes are easily applied Platform terminal scripts are more easily edited

33 OPENDCS – LRGS Status LRGS Version 9.0 – This inference displays the current good, parity and dropped messages from DOMSAT, LRIT and also, electronic data logger (EDL). It also displays the internet backup from WALLOPS and EDDN.

34 OPENDCS – DCP Messages This familiar component is used to quickly retrieve messages as well as apply the decoding scripts. In this example, pseudo-binary messages are decoded to the desired parameters.

35 OPENDCS – Decodes Editor
This is where all the DECODES are added/edited. Tabs include, Platforms, Sites, Configs, Equipment, Presentation, Routing, Sources, Network Lists and Schedule Entry. The Configs tab is open, displaying numerous DECODES applied to different platforms.

36 OPENDCS – Decodes Editor Example of a decoding script
Each script has a series of format statements that apply to the incoming messages. In this case, pseudo-binary messages are decoded to the desired parameters. This latest edition of DECODES is now color coded.

37 OPENDCS – Decodes Editor Common DECODES Format Operations

38 NRT Data Acquisition In the middle of the process.
Raw Data transmitted or EDL Acquired & Decoded (LRGS/OPENDCS) WISKI

39 Acronyms

40 Credits

41 Questions??


Download ppt "LRGS & OPENDCS NRT Data Acquisition Systems"

Similar presentations


Ads by Google