CM Cheng Hong Kong Observatory HKIE ITD Annual Conference 2014 BIG DATA FOR A BIG CITY - Challenges of Big Data to Weather Services

What we are doing …  Provide weather forecasting and warning services  Monitor and assess radiation levels in Hong Kong  Provide other meteorological and geophysical services – aviation weather services, marine weather services, climate change and climate projection, earthquake and tsunami monitoring, time service, etc.

What types of data we are using …  Data from manual and automatic weather stations (temperature, humidity, wind, visibility, rainfall, pressure …) in Hong Kong and around the world  Remote sensing data – upper-air data, radar, satellite, LIDAR, …  Data from radiation monitoring stations ……

Data volume is huge  Single most voluminous amount of data – satellite data  ~ 0.1 TB per day, increasing to ~0.3 TB per day in a few years

OK with these data  Data are well structured with known data format  Data come in according to prescribed time schedule  Have the right tools to receive and process these data into products

Emerging data/information from …  Discussion forum  Social media  On-line news - Less structured. No fixed time schedule. Spread very fast. A challenge to be read by machine without ‘training’. Data archival issue.