Australian Geoscience Data Cube A Collaboration between Geoscience Australia, CSIRO and NCI GA/CSIRO Agency Report Robert Woodcock - CSIRO Simon Oliver – Geoscience Australia The Australian geoscience data cube is an innovative project which is transforming the way we analyse large gridded datasets such as earth observation data. It is allowing us to harness the power of high performance computation though the creation of high performance data which in turn allows us to deliver useful information into the hands of decision makers and the public. The Australian Geoscience Data Cube (AGDC) is being developed as a partnership between Geoscience Australia (GA), Australia’s National Computational Infrastructure (NCI), and the Commonwealth Science and Industrial Research Organisation (CSIRO) with the main aim being to support the management and quantitative analysis of massive volumes of Earth observation (EO) and other geoscientific data. CEOS WGISS41

Overview Regional Copernicus Data Access/Analysis Hub RADAR cube Australian Geoscience Data Cube Update on progress: Version 1 User Documentation - development discontinued - focus on v2 Version 2 Development The Future: Data Cubes and WGISS CEOS WGISS41

Australia’s Regional Copernicus Data Access/Analysis Hub An integrated ‘Team Australia’ approach to Support government information requirements. Support the broader objective of enhancing access to satellite Earth observation data by research, industry and civil society. Facilitate collaboration between Australians, Europeans and inhabitants of the South-East Asia-South Pacific region in exploitation of Earth observation data. Benefits for: Australia, the region, EC/ESA/EUMETSAT, and the global satellite EO community. All Sentinel Products For Supporting Consortium Partners Collaborators CEOS WGISS41

Australia’s Regional Copernicus Data Access/Analysis Hub ESA IntHub EUMETCAST Terrestrial GEANT/AARNET Partnerships (pay-per-use) Master Repository (NCI) Delivery for: Australia Region Regional Portal (a la ‘SciHub’) Original Format Data Files (HSM Storage) Agreed Standards Interoperable Services (e.g. Thredds) Analysis-Ready Data (Lustre/spinning disk) R&D for: Research/Academia Civil Society Government Industry Australia and Region Cloud and HPC (Petascale) Industry Exploitation Operational Products Deep National Archives (Government Systems) CEOS WGISS41

RADAR cube UK Catapult collaborative agreement signed between: GA / UNSW / CSIRO / UK Catapult Analysis Ready Data for RADAR: Sentinel-1 Level-1 Interferometric Wide Swath Level-1 Extra Wide Swath ERS-1/2 SAR ENVISAT ASAR ALOS-1 PALSAR CEOS WGISS41

AGDC Overview A series of data structures and tools to enable efficient analysis of large earth observation archives in HPC / Cloud and Desktop environments Simple Data Structures Spatially regular tiles Managed by a relational database Calibrated and Standardised Unique Observations Surface Reflectance Observations Quality Assured Observations Flagged for cloud, cloud shadow, saturation and other quality indicators Analysis Ready Data as input CEOS WGISS41

AGDC Programme Partners: GA, CSIRO and the NCI International collaborators are increasingly involved including the USGS, NASA and CEOS. In the midst of moving to an updated version reflecting recent advances in technology Supporting the establishment of other international data cubes, initially with Kenya and Colombia AGDC is currently supporting a range of remote sensing applications across the water, vegetation and mineral domains Providing valuable information for environmental monitoring and modelling across all Australian jurisdictions CEOS WGISS41

Landsat Analysis Ready Data preparation DataCube Solid science Taking the data to comparable quality assured measurements Adapting software systems for embarrassingly (massively) parallel processing, enabling quality assurance, building code workflows that allow processes to be iterated and improved / experiments – this is where workflows are important (I will discuss in a minute how we are tackling this) CEOS WGISS41

Using tidal models to map tidal extents Tidal Range of >10m Tidal Zone Extent Can be attributed with offsets of LAT to lowest observed tide and HAT to highest observed Tidal Zone Morphology Fraction of water observations over the time series. Can we attribute this with depths? CEOS WGISS41

Using tidal models to map tidal extents Time-series analysis - calculating median from acquisitions at low (right) and high (right) tide models Dr Stephen Sagar Instead of the median pixel value, this image uses the bottom 10% of tide heights to select pixels, producing a generic low-tide image of the entire Australian coast. This is highly sought-after for coastal studies and navigation charting. Similarly by taking the top 10% of tide heights to select pixels, we can produce the high-tide image of Australia. Once again this is very valuable for coastal studies. CEOS WGISS41

Mangrove change over 25 years Here we use the Data Cube to create yearly statistical summaries of greeness in the landscape using the Normalised Difference Vegetation Index. The image shows peak greenness for 1987, 2000 and 2014 as blue, green and red. So areas that are blue were green early in the time series while currently are not. Areas that are red are green in 2014 but not earlier. Areas that are white or shades of grey have not changed but have stayed at the same level of greeness through time. In this area it shows how mangroves have colonised some areas and retreated from others over the last 30 years. CEOS WGISS41

Terrestrial applications Vicarious calibration sites Identify climatic zones, spatial and temporal variation, and seasonal suitability for calibration activities Landsat MODIS blending Blend Landsat and MODIS scenes to produce Landsat-like data (25m resolution) with MODIS repeat cycle (~ every 4-days) GEOGLAM Rangeland and Pasture Productivity Rangeland and pasture cover information for ecosystem health and livestock resources globally CEOS WGISS41

Solutions to assist algal management in lakes and rivers An increasing global problem Systems under pressure Human and animal health impacts AU$180-240M per annum in bloom impacts Tim Malthus, Janet Anstee, Hannelie Botha, Stephen Gensemer, Eric Lehmann, Xavier Ho Fundamentally physically based approach… driven by colour Development of specific algal alert indices – here for chlorophyll – have had to have made the ranges ourselves. Information services CEOS WGISS41

Processing Pathway and Decision Support Tools From remote sensing data to visualization of algal alerts Select extents from WOFS data Source latest AGDC dataset(s) Apply pixel quality and optical filters Apply algal alert algorithm(s) Produce visualisation products Moving towards operationalisation Built around the Australian Geospatial DataCube (Landsat 8) Software framework / Product delivery system Rapid turnaround of satellite data streams to red/yellow/green alerts Able to accommodate new sensors and data streams (S2), new algorithms CEOS WGISS41

AGDC v2 Complete rewrite of AGDC v1 (v2 Deployment July 2016) Reprocessing of Landsat - improved geolocation, spectral & terrain correction MODIS refinements to ARD preparation (C005 and prep for C006) AVHRR national archive move to AGDC under investigation Hybrid NoSQL influenced database model Ingest support for multiple sensors Multidimensional storage - NetCDF4-CF1.6 Multidimensional access - XArray and Dask Analysis and Production Pipeline Provenance capture Projection support / Geographic / Sinusoidal / Albers Equal Area etc. Transition planning - Landsat Archive Reprocessing - featuring terrain corrected Landsat as an additional layer Analytics / Query Language / Quality and Spectral Semantics Test it out - github.com/data-cube/agdc-v2/tree/develop - example ingest configurations for MODIS MCD43A1->4, USGS Landsat SR, Himawari 8 BRF from Australian BoM, GA ARG25 NBAR/T products, SRTM DEM. 1. MODIS (and AVHRR) dot point(s) Refining our acquisition, management and processing workflows for MODIS with a view to Collection 6: Australian L1A archive for NRT and swath-based ocean and terrestrial products, and; USGS/LPDAAC C5 and C6 selected products for global and Australasian applications. Refactoring our national archiving, calibration and “stitching” activities for Australian-acquired AVHRR towards a low-maintenance and sustainable system and resource. CEOS WGISS41

AGDC v2 Data Ingestion Prepare data for analysis in a couple of steps: Storage configuration - Projection / resampling / grid specification Data preparation (assuming Analysis Ready Data input) - write configuration file allowing datacube to interpret input data Ingest Stack Analyse TODO - simplify data preparation - currently using python template CEOS WGISS41

AGDC v2 Projection and Sensor Support Flexible storage unit representation Ability to access data in native or lat/lon projection coordinates through API MODIS MCD43A4 tile in sinusoidal projection, Sentinel-2 L1C and Landsat storage units in Albers Equal Area South Eastern Australia - At right: Sinusoidal projection - At left: Australian Albers Equal Area Projection CEOS WGISS41

Open Data and Code AGDC Web: http://www.datacube.org.au AGDC Wiki: http://www.datacube.org.au/wiki/Main_Page Code repositories are available through GitHub: https://github.com/data-cube/agdc-v2/tree/develop CEOS WGISS41

OGC Discrete Global Grid Systems (DGGS) SWG DGGS an important part of the future versions of Datacube OGC DGGS SWG to begin elaboration of Extension Standards to the DGGS Core Standard Anticipated Extensions include: Interoperability interface protocols for OGC Web Services (e.g. WCS, WCPS, WCTiles, etc…) to facilitate DGGS-to-DGGS communication and processing 3D (and higher dimensional) DGGS Specifications Best Practice Guide CEOS WGISS41

Future: WGISS involvement AWS - S3 object storage study The Future: Data Cubes and WGISS Discrete Global Grid Systems - OGC SWG and implications for AGDC evolution CEOS WGISS41