1. LRGS & OPENDCS NRT Data Acquisition Systems
Enterprise Data and Information Management Section
Informatics Branch, Cooperate Services Division
Alberta Environment and Sustainable Resource Development
March 11, 2015 WISKI Conference, Edmonton

2. Enterprise Data & Information Management Staff
Vacant
Vacant
Bob Werner
Vacant

3. NRT Data Acquisition Team
Bob
Werner
Today we are focused on the NRT/Historical Water Data Group consisting of four members with Adam Minke the newest who started focus is on this smaller group. Most of them are here today and I hope you have a chance to meet and talk to them.

4. Near-Real-Time Monitoring
Hydrometric/Lakes/Ice
Water level
Stream Flow
Storage
Meteorological
Precipitation
Air temperature
Snow depth/water
Wind speed/direction
Humidity
Water Quality
Conductivity
Dissolved oxygen pH
Turbidity
Ground Water
Temperature

5. Data Acquisition Group Responsibilities
Manage the real-time data collection systems (LRGS, OPENDCS)
Provide data support for Field Staff (GOES, problems log)
Provide 24x7 NRT support during major events (ice breakup, floods)
Provide access to hydrometric, meteorological, groundwater, & water quality data
Archive historical ESRD data
Main responsibilities are:

6. WISKI: ESRD Enterprise System
WISKI serves many users & functions within ESRD
Primary Users:
River Forecast & Engineering
Dam Operations
WISKI allows:
Modeling
Graphing
Export & Analysis
NRT & Archival
Continuing to expand…

7. Data Collection Working Together
External Partners
Monitoring
Data Management
SDO
WISKI
Support
River
Engineering
River Forecasting
Dam Operations
Irrigation Operations
Public
Field
Office
Results

8. Data Acquisition (LRGS/OPENDCS)
Data Management Activities
Data Acquisition (LRGS/OPENDCS)
Decoding
Data validity checks
Sensors verification
Unit conversions
Time conversions
Time adjustments
WISKI Database
Create stations
Insert/update metadata
Create and assign timeseries
Create aggregations and processing time series
Input monitoring visits and apply corrections
Verify data accuracy

9. Data Acquisition Alerts and Tools
Our system requires numerous alerts and tools to pre-emptively discover and resolve issues
Broken decodes
Lack of platform connection and/or data currency
System alerts
Change decoding script as required
Test connection and troubleshoot a solution
Dashboard to identify overall health of system

10. Data Sources

11. Realtime Data Communications
Satellite DCPs (450)
Phone Loggers (115)
Webpage Data Mining (36)
SCADA Server (44)
SFTP Servers (15)
WISKI
Breakdown of what data we collect.
FIRES Meteorological (250)
IP Connect (88)
Bulk Historical Data Products

12. Incoming Data Formats RAW MODEM OUTPUT GOES BINARY
21,2013,279,1200,1393,-1.265,-.527,0,.316,1.265,1.265,1.265,1.265,1.265,1.265,1.265,1.265,1.265,1.16,.633,.105,0,0,0,0,0,0,0,-.105,-.105,-.843,-.843,-.738,-1.16,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-2.003,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.636,-3.584,-3.69
19,2013,279,1300,1393,23.52,100,12.91,12.86,5.649,73.8,7.4,9.26,73.8,5.64,59.46,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,1.775,11.44,92.7,13.1,334.5,334.5,334.5,334.5,0,2.759,0,2.385,0,1.704,0,1.602,0,334.4,334.5,334.5,334.5,5.435,5.366,5.292,5.354,0,-.009,5.626,5.52,7.36,7.28,9.12,9.21,5.581,5.461,333.5,331,333.5,331,333.5,331,333.5,331,333.4,330.8,333.4,330.8,333.4,330.8,333.4,330.8,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,2193,0,358,0,358,.088,0,358,0,1200,358,0,358,0,358
GOES BINARY
A ?30-0HF016WFF00087`B1ECAjCAeCAdCAdCA`Caas0F5qa?>+rq9V)n]1j;ik$}.5:=Kf5RKPKFTD#H3d(i**+{*m&
GOES 5 MINUTE DATA – HUMAN READABLE FORMAT
4555E4EE G35+1NN167EXE :HG 2 # :VB 17 # :YP 17 #
TAU INCOMING TEXT FILE
Bighorn ,02-Dec ,10:00:00 , , , , , ,
Brazeau ,02-Dec ,10:00:00 , , , , , ,
Nakiska ,02-Dec ,10:00:00 , , , , , ,
Rose_Creek ,02-Dec ,10:00:00 , , , , , , 0
Canmore ,02-Dec ,10:00:00 , , , , , ,
Spray ,02-Dec ,11:00:00 , , , , , ,
WEB PAGE DATA MINE FROM INTERNET
<HTML><HEAD><TITLE>Hourly Hydromet Data for Lake Sherburne, Swift Current Creek near Babb, MT</TITLE></HEAD><BODY><PRE>
SHER |13DEC03| FB | AF | OBG | OBT | RB | RD | QR | QS | Q |
| 00:00 | | | | | | | | | |
| 00:15 | | | | | | | | | |
INTERNET TRANSFER
ATE/TIME (yyyy/mm/dd) AirTemp Precip Bat vol
(standard time) (C.) (mm) (volts)
2013/11/30 00:24:
2013/11/30 01:18:
2013/11/30 01:24:
2013/11/30 02:18:
ENVIRONMENT CANADA STANDARD NATIONAL OUTPUT FOR MODEM

13. System History – 1984 to 2015 1980’s STREAM/COLLECT System on VAX
Rafi Morgantini and Lenard Cheng Project Managers, Alberta Environment, Systems and Computing
STREAM/COLLECT system on VAX, 12th Floor Oxbridge
Late 1980’s: Moved to RFC 11th floor and managed by Thai Nguyen and Ian Barnaby after the dissolution of Systems and Computing
1990’s DACQ/COLLECT System Data General
DACQ Database System & COLLECT Acquisition
COLLECT Acquisition acquired GOES via dialup Telnet, manual data inputs via dialup Telemark and automated tone encoder Telemark2 for ~ 45 stations and stored on DACQ system.
Here’s shows an outline of our present system
2000 NewLeaf and COLLECT Red Hat UNIX
DACQ decommissioned prior to Y2K and COLLECT now independent of the new database application called NewLeaf (Ingres). Eventually Data General was decommissioned and COLLECT moved to Red Hat UNIX server (Allister Chisholm).

14. System History – 1984 to 2011
COLLECT and Acquisition Modules
This period constitutes a major growth of our NRT network via GOES, MSC dialup modems, BCHydro, TransAlta, Merchant, Scada, SnoTel, AutoPoll, Hydras3, and LoggerNet. This involved the deployment of VM technology that provided a scheduled poll of each data steam as required.
2001 Local Readout Ground Station (LRGS)
Dialup via Telnet was eventually discontinued and the JAVA open-source software provided by USGS was deployed to interface with COLLECT.
2005 LRGS (DELL) and DCS-Toolkit (XML)
The USGS consultant, Michael Maloney (ILEX Engineering) was hired to install a server and several licensed copies of the DCS-Toolkit. This provided DM with the tools necessary for writing decoding scripts for dozens of new types of messages, including pseudo-binary.
Here’s shows an outline of our present system
2011 LRGS & DCS-Toolkit Upgrade (Oracle)
ILEX Engineering was hired to convert XML to Oracle and provide clients to desktops.

15. System Timeline – 2014 to 2015 2014 COLLECT Replacement – Phase 1
All hardcoded decoding scripts embedded into COLLECT replaced by DECODES. This is essential, as decoding scripts on UNIX are no longer supported to provide enhancements and/or the creation of new decodes based on new platforms or firmware upgrades.
2015 COLLECT Replacement – Phase 2
LRGS is upgraded to handle EDL (electronic data logger) and other non-GOES data inputs. Also, the dialup modem polling is rewritten to utilize DIGI ConnectPort device.
January, 2015 LRIT Satellite Project
Here’s shows an outline of our present system
April, 2015 LRGS/OPENDCS Acceptance Testing
Both Phase 1 and 2 are currently being UAT tested till the end of the contract period. Once complete, we will be running LRGS version 9 and OPENDCS RC 4.0.
May, 2015 LRGS/OPENDCS PROD Launch
After the ICE season and before the spring runoff, a decision will be made to go live. If so, COLLECT will be engaged as a backup system.

16. Legacy System Configuration Phase 1 and 2 Identified
Here’s shows an outline of our present system

17. NRT Data Acquisition LRGS/OPENDCS Final State

18. What is LRGS?
Local Readout Ground Station is a server that can be run on any operating system, as it is written in JAVA. UNIX is our preferred operating system, however this can be changed depending on our system support.
Non-GOES Messages
OPENDCS

19. DOMSAT
Local Readout Ground Station is a server that can be run on any operating system, as it is written in JAVA. UNIX is our preferred operating system, however this can be changed depending on our system support.
Non-GOES Messages
OPENDCS

20. DOMSAT Receiver Decommissioning service by 2015/16
Domestic Communications satellite is a single high-speed channel that NOAA rebroadcasts to, allowing the entire DCS (Data Collection System) GOES messages for consumption by users. ERSD owns a dish a primary receiver on top of Oxbridge tower and a backup site, at EMA (Emergency Management Alberta) in Edmonton. This service will be decommissioned by the end of 2015.

21. DOMSAT Receiver Decommissioning service by 2015/16
DCP’s transmit to either GOES W or GOES E and captured at NOAA, which in turn are broadcast to DOMSAT (AMC-4) to our DOMSAT receiver. These messages are then captured on our LRGS server.
LRGS

22. LRIT Receiver Installed on January, 2015, Oxbridge Place
Our first LRIT satellite service was installed on January, 2015 and will become our primary source of GOES messages once DOMSAT is decommissioned. Messages are rebroadcast to both GOES E and GOES W. A secondary site is planned for the year end by our Provincial Telecom Coordinator, Forestry Telecommunications Shop, ESRD.
Non-GOES Messages
OPENDCS

23. LRIT Receiver Installed on January, 2015, Oxbridge Place
The GOES messages are received at NOAA and rebroadcast back to GOES East and GOES West. Messages are then relayed back to an LRIT receiver which we now own on Oxbridge Place.

24. Satellite Antennas @ Oxbridge
Currently, our newly installed LRIT antenna is pointing at GOES East (SE direction).

25. DCS via Internet
As a backup, should messages fail via our satellite receivers, a redundancy is built into the system by the internet. We current use the internet feed from Wallops Command and Data Acquisition Station (WCDAS) near Chincoteague, Virginia, and Emergency Data Distribution Network (EDDN) Sioux Falls, South Dakota.
Non-GOES Messages
OPENDCS

26. Wallops Command Centre
WCDAS Statistics Annual Funding at Wallops (NOAA Appropriations): Approximately $11 million Workforce: 91 (mixture of Government and contractors) Number of buildings and square footage: Seven buildings with 48,920 square feet (not including antenna shelters) Number of antennas: 16 Data rate throughput: Currently 41.7 megabits/second Date NOAA occupied site: Original permit signed on February 2, 1965 and dedicated on: June 22, 1966

27. Non-GOES Messages FTP, HTTP, SQL, Dial-up Modem
Since the completing of the COLLECT replacement project, phase 2, we now have the capacity to acquire all non-GOES messages, except for stations currently using Campbell Scientific LoggerNet software. Additional work is required to integrate LoggerNet API into LRGS (2015/16).
Non-GOES Messages
OPENDCS

28. Non-GOES Messages FTP, HTTP, SQL, Dial-up Modem
LoggerNet
Campbell Scientific Software
IP Protocol and PakBus platforms require this software. Phase 3 (2015/16) will integrate the LoggerNet API into LRGS, thereby completing the total integration of non-GOES messages.
Phase 2

29. Non-GOES Messages Dial-up Modem
One major improvement allows for LRGS to function with modems. We are retiring our Sangoma cards that plugged directly into our UNIX server for a device that sits on the Ethernet.
16 US Robotics modems
Sangoma Card (legacy) – prevented us from upgrading our OS and server.
DIGI LTS ConnectPort 32 allows for all 16 modems to have a telnet port on the internet available to the LRGS server.

30. Why still dialup modems?
We currently still maintain 115 sites with dialup modems that are decreasing at a rate of ~ per year now for 3 years. Our peak was 145 stations. We predict that within the next 6 years, this number will drop of significantly. The cost of providing copper connection to remote stations is becoming increasingly more expensive and maintenance of existing lines, more probiotic. Providers are increasingly switching to either cellular or GOES.

31. OPENDCS
We started with the raw messages that are acquired by LRGS (GOES and non-GOES) and will end this presentation with the second part, that of the OPENDCS (red box). The other outputs that we own and operate are, Web DCP Monitor, Message Browser (integrated into OPENDCS) and the DCS Toolkit currently used in production (blue boxes).
Non-GOES Messages
OPENDCS

32. OPENDCS
The OPENDCS structure is similar to DCS Toolkit currently being used in production. The reasons that make this software valuable are:
Designed in JAVA, which allows for cross platform use
Utilizes DECODES format structure to decode messages
Configuration XML easily shared with other NRT acquisition members
OPEN means, open source and therefore will not require a separate licence for each installation
RC03 denotes the release is Canadian (RC), specific to ESRD requirements
All station configurations formerly contained on several flat files in COLLECT are now stored in Oracle and more easily backed up and restored when required
Software is easily upgraded on workstations
Decoding changes are easily applied
Platform terminal scripts are more easily edited

33. OPENDCS – LRGS Status
LRGS Version 9.0 – This inference displays the current good, parity and dropped messages from DOMSAT, LRIT and also, electronic data logger (EDL). It also displays the internet backup from WALLOPS and EDDN.

34. OPENDCS – DCP Messages
This familiar component is used to quickly retrieve messages as well as apply the decoding scripts. In this example, pseudo-binary messages are decoded to the desired parameters.

35. OPENDCS – Decodes Editor
This is where all the DECODES are added/edited. Tabs include, Platforms, Sites, Configs, Equipment, Presentation, Routing, Sources, Network Lists and Schedule Entry. The Configs tab is open, displaying numerous DECODES applied to different platforms.

36. OPENDCS – Decodes Editor Example of a decoding script
Each script has a series of format statements that apply to the incoming messages. In this case, pseudo-binary messages are decoded to the desired parameters. This latest edition of DECODES is now color coded.

37. OPENDCS – Decodes Editor Common DECODES Format Operations

38. NRT Data Acquisition In the middle of the process.
Raw Data transmitted or EDL
Acquired & Decoded (LRGS/OPENDCS)
WISKI

39. Acronyms

40. Credits

41. Questions??