Emerging Technologies and It's Use Implementing fast-track methodologies in IoT-based technology development for the meteorological community: A multidisciplinary approach By: Bienifer Vinluan

About the presentation This presentation will try to address the following questions: What are the changing user demands? What are driving these changes? and Why should we care?

Changing user demands (cont'd) To begin, our legacy systems today still serve our our community very well, but with an information rich society and availability of mobile devices, our existing systems will not cope with this new generation of expectations. As an example, our observation system architecture looks like this:

Existing System Infrastructure Value Add to AWS Data Fully Manual Coop Observer Manual Observations Message Switch AWS One Minute Data OMD or NVP format CMSS WA NT QLD NSW VIC SA TAS HO RTDB (NEONS) Climate Aviation Forecasting International Regional Products HO & Regional Computing Systems AWS One Minute Data External Product – Web/FTP

Internet and our users This system is robust, highly available and provides the required product/output for public and commercial consumption. But it has its constraints, there's a growing expectation that data can be provided to everyone, everywhere on everything?? Our new generation of users will only recognise the internet, smart devices and use of: And we can't easily make end-to-end system changes without huge investments and project overhead…

Adopting Emerging Technology We need to look into different ways of developing and delivering those products fast, up-to-date and can be easily consumed by our users. So., we dived into the Internet of Things (IoT), in particular we looked at MQTT an IoT protocol.

MQTT protocol Extremely simple and lightweight binary messaging protocol, Designed for constrained devices and low-bandwidth, high-latency or unreliable networks; Ensures reliability and a degree of assurance of message delivery, via its inbuilt three level QoS (Quality of Service) delivery model; IBM developed with open source community support; From here we did a proof-of-concept to delivery data through a data broker that has capability to publish and subscribe..

MQTT – Data Broker System Push Sensor One Minute/ One second Data Subscribe to ASOS-CS 1 minute data Applications Publish Sensor One Minute Data Publish Calculated Data Subscribe to ASOS-CS 1 minute data HO & Regional Computing Systems MQTT WA Subscribe to Sensor One Minute Data NT QLD NSW VIC Push calculated Data, met products SA TAS Subscribe to products or ASOS-CS one minute data HO External Clients

Key objectives of the Data Broker The key aim in this stack implementation was to publish as many sources of relevant Bureau information to the broker. This information will include: 1 minute AWS messages from the Bureau’s 700 AWS; 3rd Party AWS data; Relevant model output and forecasts; Satellite graphics or images; and, Customised products.

Technology Stack Measurement devices (devices such as Raspberry Pi’s or telemetry devices using the MQTTX protocol ); A Publish /Subscribe broker that forms the core of our information exchanges; The use of IoT friendly web and HTTP methods, MQTTX and web sockets; Visualisation tools; Node.js - event driven, non-blocking server side JavaScript, optimized throughput and scalability; Web sockets - full-duplex communication; NoSQL, Time Series and Key Value data stores (use of Influx Db); Geo-hash technologies; Meteor and React – JavaScript based, enables rapid application development for web, mobile, desktop and building reusable components;

Our approach to Development When new trends come up we look, we assess, we think, we evaluate… But this traditional process has been proving to take just too long. By the time we complete any working system the technology has become obsolete…. and by then we've invested too much time/effort and $$$

Our approach to Development(cont'd) So we need to think differently…. When we started to tap into the pub/sub platform using MQTT as an IoT protocol we needed to do a very rapid prototype to prove that it can be integrated into our existing architecture… In doing this we have to form team that is Agile – we started to adopt the Agile Framework..

Our approach to development (cont'd) We formed a multidisciplinary team from various areas – IT Developers, Testers, System Administrators, Engineers and Project Management and worked on this project for 3 months. We worked under an Agile Devops environment The outcome is positive – we are able to deliver a number of features including: Data provisioning through pub/sub platform; Data visualisation; Download Features; Basic alerts and thresholds – per AWS rule.

Challenges This didn't get through without challenges such as: The need to prove that the technology is scalable and can be easily extended at a notice. The organisation is still in its early stages of being Agile ready. The ongoing cost and maintenance has not be fully considered.

Conclusion To address changing user demands in this information rich society we now live in and continue to be relevant in the community we need to have a two-fold strategy: That will allow us to tap into the new space of emerging technology. 2. To develop products and deliver them in an agile fashion. We need to be quick, fast or we will miss opportunities.

Questions? Thank you for listening…. Bienifer Vinluan b.vinluan@bom.gov.au