Himawari‐8 Operational Data Processing and Access Leon Majewski 25 August 2015

Data Ingest and Reception Multiple access methods and processing chains HimawariCloud NHMS access: operational since July 2015 HimawariConnect ABOM access: Due September 2015 100 Mbit link to JMA (Tokyo) HimawariCast Melbourne: Due August 2015 Darwin: Due September 2015 Methods: Direct = the direct link between ABOM and JMA – ABOM will be the only NMHS with a direct connection Himawari Cloud is the distribution point for NMHS Himawari Cast is a broadcast of lower resolution data (1 x 1km, 13 x 2km) Adds robustness (fall back methods) Latency Important to note that the 00:00Z is available to us at 00:12 and available to users at 00:16 That is, we turn it around in ~3 minutes, some data is available via CMSS in 30 seconds

Himawari-8 HimawariCast Hope JMA HimawariConnect HimawariCloud Methods: Direct = the direct link between ABOM and JMA – ABOM will be the only NMHS with a direct connection Himawari Cloud is the distribution point for NMHS Himawari Cast is a broadcast of lower resolution data (1 x 1km, 13 x 2km) Adds robustness (fall back methods) Latency Important to note that the 00:00Z is available to us at 00:12 and available to users at 00:16 That is, we turn it around in ~3 minutes, some data is available via CMSS in 30 seconds

Data Ingest and Reception Multiple access methods and processing chains HimawariCloud NHMS access: operational since July 2015 HimawariConnect ABOM access: Due September 2015 100 Mbit link to JMA (Tokyo) HimawariCast Melbourne: Due August 2015 Darwin: Due September 2015 Methods: Direct = the direct link between ABOM and JMA – ABOM will be the only NMHS with a direct connection Himawari Cloud is the distribution point for NMHS Himawari Cast is a broadcast of lower resolution data (1 x 1km, 13 x 2km) Adds robustness (fall back methods) Latency Important to note that the 00:00Z is available to us at 00:12 and available to users at 00:16 That is, we turn it around in ~3 minutes, some data is available via CMSS in 30 seconds

Data Ingest and Reception Approximate HimawariCloud data availability timeline 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 1   2 3 4 5 6 7 8 9 10 ABOM image generation ABOM product generation What does this translate to in terms of latency? Chart is an approximation of the data processing schedule Darwin observed around 00:06-00:07, data is available 7-8 minutes after obs Brisbane/Perth observed around 00:07-00:08, data is available 6-7 minutes after obs Syd/Mel/Hbt/Adel… observed around 00:08-00:09, data is available 5-6 minutes after obs Forecaster access is 6-10 minutes after scan completion 00:00Z scan completes at 00:10Z Available to download at 00:12Z-00:15Z netCDF and GeoTIFF output are generated Data is distributed to forecasters at 00:16Z-00:19Z 1 2 3 4 5 6 7 8 9    Observation  Processing  Available for download

Real time processing: Observations HSF data is converted to netCDF4 Metadata: CF, ACDD Native projection (+proj=geos) Application of calibration coefficients Scaled radiance (bands 1-6) Bidirectional reflectance factor (bands 1-6) Brightness temperature (bands 7-16) Generation of solar geometry scaled_radiance   "scaled" means multiplied by a constant factor of proportionality. The scaled_radiance is equal to the measured radiance multiplied by pi and divided by the solar irradiance averaged over the spectral band for normal incidence and an Earth-Sun distance of 1 astronomical unit. The radiant fluxes are integrated across the spectral band. Equivalently, the scaled_radiance is equal to the toa_bidirectional_reflectance_factor multiplied by the cosine of the solar zenith angle and divided by the square of the Earth-Sun distance. toa_bidirectional_reflectance_factor "toa" means top of atmosphere. The term "bidirectional" implies single directions for the incident and reflected radiances (entering and emanating, respectively, from solid angles that are differential in theory but very small in practice for satellite observed radiances at the TOA). "bidirectional_reflectance_factor" is the ratio of the reflected radiant flux exiting a surface to the reflected radiant flux from an ideal and diffuse (Lambertian) surface under identical view direction and solid angle and identical single direction illumination. The fluxes are integrated across the spectral band. The toa_bidirectional_reflectance_factor is equal to the measured radiance multiplied by pi, divided by the cosine of the solar zenith angle, and divided by the solar irradiance averaged over the spectral band for normal incidence and the Earth-Sun distance at the time of the measurement. The brightness temperature is the temperature a black body in thermal equilibrium with its surroundings would have to be to duplicate the observed intensity of a grey body object, such as the Earth, at a given wavelength.

Real time processing: Imagery Generation of imagery from netCDF4 GEOTIFF and JPEG with colour scale applied Native projection Simple latitude/longitude grid (+proj=latlong) Geospatial Data Abstraction Library Open source toolkit for geospatial data manipulation Image translation (crop, format) & re-projection Going back to the first slide, you can see many of the AHI bands line up with the NOAA GOES-R ABI Consequently many of the algorithms NOAA has developed for ABI are applicable to AHI (with a bit of tweaking) Note that

Real time processing: Products GEOCAT framework (NOAA & University of Wisconsin) Cloud mask & properties (Heidinger & Pavolonis) Fog & Low cloud (Pavolonis) Volcanic Ash (Pavolonis) Atmospheric Motion Vectors (Daniels & Le Marshall) Sea Surface Temperature (Griffin & Majewski) External processing systems (post-GEOCAT) Solar Radiation (MINES-ParisTech & Grant) Going back to the first slide, you can see many of the AHI bands line up with the NOAA GOES-R ABI Consequently many of the algorithms NOAA has developed for ABI are applicable to AHI (with a bit of tweaking) Note that

Data access: Observations Operational (Government, commercial & non-commercial) netCDF: Registered user service Service Level Agreement Research use: National Compute Infrastructure Delayed availability (< 24 hours) HSF: 30 days online, archive to MDSS netCDF: online, copy on MDSS Working with NCI to achieve efficient access Going back to the first slide, you can see many of the AHI bands line up with the NOAA GOES-R ABI Consequently many of the algorithms NOAA has developed for ABI are applicable to AHI (with a bit of tweaking) Note that

Data access: Products Operational (Government, commercial & non-commercial) Research use: National Compute Infrastructure RDSI Delayed availability (< 24 hours) netCDF: online, copy on MDSS Released as determined to be "valid" Consistent with NOAA output Routine validation metrics available Updates are expected as improvements are made Going back to the first slide, you can see many of the AHI bands line up with the NOAA GOES-R ABI Consequently many of the algorithms NOAA has developed for ABI are applicable to AHI (with a bit of tweaking) Note that

Validation Observations JMA GSICS web page JMA Navigation web page Products Cloud properties: CloudSat, heritage products AMV: Radiosondes, NWP model impact Solar radiation: Solar radiation network Sea Surface Temperature: drifting buoys Going back to the first slide, you can see many of the AHI bands line up with the NOAA GOES-R ABI Consequently many of the algorithms NOAA has developed for ABI are applicable to AHI (with a bit of tweaking) Note that

Future Developments Meteorological product development Overshooting tops Convective initiation Significant rainfall NWP data assimilation Atmospheric Motion Vectors Clear Sky Radiances Cloud properties Public viewer release (late September) End note: At the start, I mentioned HW8 was a big data challenge. The volume and velocity components are challenging, but we can do it. The real challenge is to take advantage of the variety. New products is one thing HW8 alone (temporal and spatial) HW8 products (AMVS and cloud type to see where those ice clouds are headed over the next hour?) HW8 fused with other data sets (satellite, nwp, …) But taking advantage of the products is another

Leon Majewski Leon.Majewski@bom.gov.au Thank you… Leon Majewski Leon.Majewski@bom.gov.au

Advanced Himawari Imager Advanced imagers provide us with a "Big Data" challenge Volume Imagery: 50TB/year Products: 100 TB/year Velocity 10 minute refresh Variety Band combinations New Products AHI Band Central Wavelength Spatial Resolution Bit Depth MTSAT Central Wavelength GOES-R ABI Central Wavelength [µm] [m] [bits] 1 (B) 0.47 1000 11 2 (G) 0.51 3 (R) 0.64 500 0.68 4 0.86 1.37 5 1.60 2000 1.61 6 2.30 2.24 7 3.90 14 3.70 8 6.20 6.80 6.17 9 6.90 6.93 10 7.30 12 7.33 8.60 8.40 9.60 9.61 13 10.40 10.80 10.33 11.20 15 12.40 12.00 12.30 16 13.30 Essentially a big data challenge we get lots more data due to improved resolutions we get it faster and more frequently we've got a lot more scope for products and band combinations it's a challenge to store, process, delivery, retrieve… and do it in a robust fashion: in MTSAT land, if an issue arose, one image per hour would be lost. now 6 per hour – difficult to catch up

netCDF4 data format 20150220005000-P1S-ABOM_OBS_B13- PRJ_GEOS141_2000-HIMAWARI8-AHI.nc Date: 20150220005000 Time coverage: P1S Product code: ABOM_OBS_B13 Projection code: PRJ_GEOS141_2000 Platform: HIMAWARI8 Sensor: AHI Going back to the first slide, you can see many of the AHI bands line up with the NOAA GOES-R ABI Consequently many of the algorithms NOAA has developed for ABI are applicable to AHI (with a bit of tweaking) Note that

netCDF4 data format Variable names: channel_0001_scaled_radiance Band 1 scaled radiance channel_0003_brf Band 3 TOA bidirectional reflectance factor channel_0013_brightness_temperature Band 13 TOA brightness temperature Going back to the first slide, you can see many of the AHI bands line up with the NOAA GOES-R ABI Consequently many of the algorithms NOAA has developed for ABI are applicable to AHI (with a bit of tweaking) Note that

Imagery: VIS, IR, WV, Combinations