AOGEOSS Task 11. Develop Regional GEOSS Data set

The Data Problem Increased Volume Low Capacity Slow Internet A significant growth in imagery data will increase data volumes by >10x in the next few years. Many countries lack the knowledge, infrastructure, and resources to access and use space-based data. Data Cube architecture provides a solution that saves countries time and money and reduces technical complexity. Increased Volume Low Capacity Slow Internet

Expectations for On-Demand Integration On the other side of this deluge of spatial content is a growing expectation by decision-makers for on-demand access to data from a diversity of sources for use with other content without a reliance on time-consuming and costly integration processes. The expectation amongst decision makers is for geospatial data integration on-demand. To enable this, a common analytical platform is required that can link very large multi-resolution and multi-domain datasets together to enable application of the required analytic processes.   Community safety Information for decision support Sentinel 2 and the Australian Geoscience Data Cube

Traditional remote sensing process Traditional approaches to acquiring, managing, distributing and even using this data have made it challenging to realise the true value of this data As well as the points we just made about use of satellite data to produce simulated ‘photos’ … Traditional ‘pull based’ models, where acquired data is only prepared to the point of being useful when somebody asks for it, create a significant ‘barrier to entry’ for those businesses, agencies or researchers thinking of a new initiative And even when the interest is there, the sheer amount of time and cost involved in the preparation process can be prohibitive … often resulting in Potential data being ‘thrown away’ because ‘too much’ of a particular scene was cloudy Initiatives being unnecessarily scaled back to focus on smaller areas, or smaller time periods, Only use data from one satellite because it is too hard for a particular group to justify the technical work required to make data from different satellites comparable The use of approaches such as ‘mosaicing’, where the raw data is ‘merged’ from multiple passes of the satellite is ‘merged’ to produce intermediary products that manageable and usable on commodity ICT infrastructure … but with the effect that often valuable data is ignored, for example reducing a potential 45 observations of a spot on the ground from the Landsat series down to just 1 that has been judged the ‘best’ for that year The storage of much data on tape archives, whilst ensuring data is preserved for the long haul, simply do not work in a world where information is needed ‘anywhere’ and ‘now’ And such approaches often result in the product of a product that provides a ‘point in time’ perspective, with significant challenges to creating the sorts of dynamic information now essential to businesses and governments

What are Data Cubes? Open Source Software https://github.com/data-cube Data Cube = Time-series multi-dimensional (space, time, data type) stack of spatially aligned pixels ready for analysis Proven concept by Geoscience Australia (GA) and the Australian Space Agency (CSIRO) and planned for the future USGS Landsat archive. Shift in Paradigm ... Pixels vs Scenes Analysis Ready Data (ARD) ... Dependent on processed products to reduce processing burden on users Supports an infinite number of applications, reduces data preparation time, allows time series analyses, increases interoperability of multiple datasets. Open source software approach allows free access, promotes expanded capabilities, and increases data usage. Open Source Software https://github.com/data-cube TIME

Advantages of Data Cube Free and Open ... Open source software, CEOS Agency contribution No data loss ... No scene-based screening required, all pixels used No processing ... Based on analysis-ready data products Spatial consistency ... Allows for user-defined nested grids Data interoperability ... Multiple aligned datasets in one system Efficient ... Less input-output (I/O) and smaller storage Faster ... Faster temporal and spatial analyses Flexible deployment ... Local computer, data hub, or cloud Allows automation ... Data download and ingestion can be automated Flexible architecture ... APIs allow infinite applications and users Improved capacity building .. Simple architecture and methods

Continental Scale Water Observations from Space 27 Years of data 25m N ominal Pixel Resolution Approx. 300 000 individual scenes in approx. 20 000 passes Entire 27 years of 1 312 087 ARG25 tiles => 93x1012 pixels visited 0.75 PB of data 3 hrs at NCI But the Data Cube paradigm has enabled us to do not only undertake this analysis at a regional scale for the first time, it has in fact enabled us to scale this to produce a comprehensive national product We can now analyse 15 years of data, across the country, representing every <n> square metres … in under 1 day. Not only does this mean we can now for the first time produce this analysis once … It means that we can produce it in an ongoing basis, enabling the production of updated information as new data becomes available, and enabling researchers to interact with the data iteratively to improve quality and try new ideas And it means that, in future, industry and others can build value-adding products on top of this data by leveraging the services-based approach adopted

Big Data for a Big Planet: a global network of regional data cubes? Data Cubes for: Africa, South America, Asia, China, India, Europe, North America, … Connecting the EO, Spatial and Statistical world to support global programs

Task Description: Task 11. Develop Regional GEOSS Data set Leaders: David Hudson (GA, Australia), YU Tao (RADI, China). Subtask 11.1 Create dialogue to assess data format requirements among data-rich countries Datasets to be developed based on GCI tools and DAB protocols, Subtask 11.2 Consolidate operational users' needs and dynamic features of large volume EO data to support development of AO Data Cube (AODC) data set format: released in 2018. Australia Geoscience Data Cube (AGDC) and Chinese Geoscience Data Tile DEM (CNGDT) Subtask 11.3 Advocate and support incorporation of pilot research using standardized AODC format, with the help of activities in task 1 to task 9. final assessment report will be released around the middle of 2019. Subtask 2.4 Advocate and support incorporation of data catalogues among member countries and contributes to task end of 2019.