CEOS Data Cubes A briefing for the GEO Secretariat Brian Killough CEOS Systems Engineering Office (SEO) April 1, 2016

The Data Problem A significant growth in land imagery data (e.g. Landsat, Sentinel) will increase free/open data volumes by >10x in the next few years. Many countries lack the knowledge, infrastructure, and resources to access and use space- based data. Countries have expressed a desire for support from CEOS by providing help for data access, storage, processing, and analysis tools. The new CEOS Data Cube architecture provides a solution that saves countries time and money and reduces technical complexity. Increased Data Volume Low Computing Capacity Slow Internet Low processing knowledge

What are Data Cubes? Data Cube = Time-series multi-dimensional (space, time, data type) stack of spatially aligned pixels used for efficient and effective data access and analysis. Proven concept by Geoscience Australia (GA) and the Australian Space Agency (CSIRO) and planned for the future USGS Landsat archive. Shift in Paradigm... Pixels vs Scenes Analysis Ready Data (ARD)... Dependent on processed products to reduce technical burden on users Supports an infinite number of applications, reduces data preparation time, allows time series stacking and analyses, increases interoperability of datasets. Open source software approach allows free access, promotes expanded capabilities, and increases data usage. TIME Open Source Software

The Data Cube Vision...  The CEOS Data Cube infrastructure will become a commonly used free and open source software toolset for creating local, regional or national pixel- based time-series of interoperable datasets that are spatially aligned according to user needs (spatial region, time period, data layers, grid projection).  Users will connect free/open user interface tools to the Data Cube for common analyses (cloud-free mosaics, time series change detection) or develop their own tools to query the content.  Scene-based tools will be used less frequently as users move toward a preference for time-series stacks of spatially aligned pixels (Data Cubes).  Space Agencies will routinely supply Analysis Ready Data (ARD) products that are easily ingested into Data Cubes.

Data Cube Architecture Working with CEOS Space Agencies to develop requirements and plans for sustained provision of Analysis Ready Data (ARD) products. Sentinel-1A and Sentinel-2A are the highest priority. Users Data Cubes Data Testing prototypes in Colombia and Kenya Developing and testing user interfaces for custom mosaic creation, water detection and water quality. Investigating capacity building options. Open source software, developed and sustained by CEOS. Support for diverse datasets (space, ground, climate), grid projections. Deployment via local computers, regional hubs (e.g. SERVIR), or computing cloud (e.g. Amazon). Connections to common GIS tools (ArcGIS, QGIS) Advanced Programming Interfaces (APIs) for users to create their own user interfaces

Custom Mosaic Tool Filter Selections Country: Kenya or Colombia Data Product: L7 or L8 Product Type: 11 options Non-clear pixels: RED flag Season: Continuous or Multi Dates: Month-Year Bounding Box or Lat-Long

Cloud Filtering Example Product: Landsat 7 Region: Southern Colombia Output: RGB Bands-7,4,2 (SWIR2, NIR, GREEN) Filter (RED): = Cloud, Shadow, Water, No Data January-February scenes 70% no data (mostly clouds) January-April scenes 36% no data (mostly clouds) January-June scenes 27% no data (mostly clouds) ~1 minute run time for each mosaic with output in GEOTIFF

Water Detection Tool Western Kenya, County of Baringo Lake Bogoria Nature Preserve Year 2013 Flooding is seen in the northern tip of the lake * Counts water / non-water QA flags * Landsat-7 “banding” is visible * Blue = Often water, Yellow = Infrequent * Able to assess drought/flood risk extent Future version will look like the Australian output on the left

Kenya Pilot Project The project is led by NASA-SEO and supported by the Australian Government and the Clinton Foundation (CCI and SLEEK). Two operating versions of the Kenya Data Cube exist: Amazon Cloud and a local SEO computer. Considering options for the SERVIR-Africa hub TB of Landsat data (7500+ scenes back to 2000). Pixel data from 42 path-row regions were extracted and reformatted into a cube. 68 time-series stacked tiles (1-deg square). Uses surface reflectance (SR) products from USGS. The Kenya team is currently utilizing scene-based methods to develop historic forest maps. The SEO has been facilitating data analysis and access to SR products. They are NOT quite ready for a Data Cube. Future testing of a Data Cube is planned in 2017.

Colombia Pilot Project The project is led by NASA-SEO and the Colombian Government. Supported by CSIRO and IDEAM (Institute of Hydrology, Meteorology and Environmental Studies). A mini-cube (4 Landsat path-row regions) was delivered in Oct Cube includes data since time-series stacked tiles (1-deg square). SR products directly from USGS. The Colombia government presented the Data Cube concept to the Ministry in late 2015 and was approved to implement the architecture for national scale applications in The Colombia team is very interested in user interface tools to produce custom mosaics, detect changes, and conduct time series analyses (land and water). The Colombia team is growing capacity... Colombia has already demonstrated use of the ingestor software to expand the data in their cube and they have modified the user interface to add median mosaics, PCA change detection and NDVI-based forest/non-forest maps.

Other Data Cubes  The concept of a Data Cube is not new. Many local scientists use Data Cubes to package their data for time series analyses. The “new” concept is that CEOS would like to globalize a common infrastructure.  USGS Land Change Monitoring Assessment and Projection (LCMAP) initiative is planning a CONUS Data Cube. They will use a Cassandra database and a REST user interface based on their available computing infrastructure. USGS will start with a CONUS (US only) cube and then eventually move to a global distribution of data in cube format. Their global delivery will be many years away …  Australian Geoscience Data Cube (AGDC) is focused on Australia applications and designed to utilize the National Computing Infrastructure (NCI) high performance computing facility. Their focus is only the Australia region, so their support for datasets and global issues is limited.

Capacity Building  Since the Data Cube concept is a rather new concept for the satellite and user community, there is a need to build capacity and provide training for those countries and specific users that desire to use this capability. Here is a status...  World Bank (WB) – SEO is meeting with WB in Washington on April 13. Exploring how Data Cubes may fit into their objectives. A future workshop may be held with WB for a larger group.  GFOI (Global Forest Observation Initiative) and FAO (Forestry Division) – Plans to develop an end-to-end demonstration in one country where GFOI, FAO, and CEOS bring their relevant contributions together to support improved forest mapping and increased country capacity to use satellite data. Candidates are Nepal, Indonesia and Myanmar. FAO believes Data Cubes could reduce data preparation time.  SilvaCarbon - The SEO has participated in past workshops to provide COVE tool training. By the end of 2016, Data Cubes will be ready for delivery.  CEOS Working Group on Capacity Building – Exploring approaches for Data Cube capacity building.  Boston University – Meeting planned at NASA on June 7-8 to develop a Data Cube connector for QGIS... a common GIS tool.