1. CEOS Data Cubes A new need for Capacity Building Brian Killough CEOS Systems Engineering Office (SEO) WGCapD-5 Meeting March 30 – April 1, 2016

2. The Data Problem A significant growth in land imagery data (e.g. Landsat, Sentinel) will increase free/open data volumes by >10x in the next few years. Many countries lack the knowledge, infrastructure, and resources to access and use space- based data. Countries have expressed a desire for support from CEOS by providing help for data access, storage, processing, and analysis tools. The new CEOS Data Cube architecture provides a solution that saves countries time and money and reduces technical complexity. Increased Data Volume Low Computing Capacity Slow Internet Low processing knowledge

3. What are Data Cubes? Data Cube = Time-series multi-dimensional (space, time, data type) stack of spatially aligned pixels used for efficient and effective data access and analysis. Proven concept by Geoscience Australia (GA) and the Australian Space Agency (CSIRO) and planned for the future USGS Landsat archive. Shift in Paradigm... Pixels vs Scenes Analysis Ready Data (ARD)... Dependent on processed products to reduce technical burden on users Supports an infinite number of applications, reduces data preparation time, allows time series stacking and analyses, increases interoperability of datasets. Open source software approach allows free access, promotes expanded capabilities, and increases data usage. TIME Open Source Software https://github.com/data-cube

4. The Data Cube Vision...  The CEOS Data Cube infrastructure will become a commonly used free and open source software toolset for creating local, regional or national pixel- based time-series of interoperable datasets that are spatially aligned according to user needs (spatial region, time period, data layers, grid projection).  Users will connect free/open user interface tools to the Data Cube for common analyses (cloud-free mosaics, time series change detection) or develop their own tools to query the content.  Scene-based tools will be used less frequently as users move toward a preference for time-series stacks of spatially aligned pixels (Data Cubes).  Space Agencies will routinely supply Analysis Ready Data (ARD) products that are easily ingested into Data Cubes.

5. Data Cube Architecture Working with CEOS Space Agencies to develop requirements and plans for sustained provision of Analysis Ready Data (ARD) products. Sentinel-1A and Sentinel-2A are the highest priority. Users Data Cubes Data Testing prototypes in Colombia and Kenya Developing and testing user interfaces for custom mosaic creation, water detection and water quality. Investigating capacity building options... Open source software, developed and sustained by CEOS. Support for diverse datasets (space, ground, climate), grid projections. Deployment via local computers, regional hubs (e.g. SERVIR), or computing cloud (e.g. Amazon). Connections to common GIS tools (ArcGIS, QGIS) Advanced Programming Interfaces (APIs) for users to create their own user interfaces

6. Custom Mosaic Tool Filter Selections Country: Kenya or Colombia Data Product: L7 or L8 Product Type: 11 options Non-clear pixels: RED flag Season: Continuous or Multi Dates: Month-Year Bounding Box or Lat-Long

7. Cloud Filtering Example Product: Landsat 7 Region: Southern Colombia Output: RGB Bands-7,4,2 (SWIR2, NIR, GREEN) Filter (RED): = Cloud, Shadow, Water, No Data January-February 2014 16 scenes 70% no data (mostly clouds) January-April 2014 30 scenes 36% no data (mostly clouds) January-June 2014 46 scenes 27% no data (mostly clouds) ~1 minute run time for each mosaic with output in GEOTIFF

8. Water Detection Tool Western Kenya, County of Baringo Lake Bogoria Nature Preserve Year 2013 Flooding is seen in the northern tip of the lake * Counts water / non-water QA flags * Landsat-7 “banding” is visible * Blue = Often water, Yellow = Infrequent * Able to assess drought/flood risk extent Future version will look like the Australian output on the left

9. Kenya Pilot Project The project is led by NASA-SEO and supported by the Australian Government and the Clinton Foundation (CCI and SLEEK). Two operating versions of the Kenya Data Cube exist: Amazon Cloud and a local SEO computer. Considering options for the SERVIR-Africa hub. 11.5 TB of Landsat data (7500+ scenes back to 2000). Pixel data from 42 path-row regions were extracted and reformatted into a cube. 68 time-series stacked tiles (1-deg square). Uses surface reflectance (SR) products from USGS. The Kenya team is currently utilizing scene-based methods to develop historic forest maps. The SEO has been facilitating data analysis and access to SR products. They are NOT quite ready for a Data Cube. Future testing of a Data Cube is planned in 2017.

10. Colombia Pilot Project The project is led by NASA-SEO and the Colombian Government. Supported by CSIRO and IDEAM (Institute of Hydrology, Meteorology and Environmental Studies). A mini-cube (4 Landsat path-row regions) was delivered in Oct 2015. Cube includes data since 2000. 20 time-series stacked tiles (1-deg square). SR products directly from USGS. The Colombia government presented the Data Cube concept to the Ministry in late 2015 and was approved to implement the architecture for national scale applications in 2016. The Colombia team is very interested in user interface tools to produce custom mosaics, detect changes, and conduct time series analyses (land and water). The Colombia team is growing capacity... Colombia has already demonstrated use of the ingestor software to expand the data in their cube and they have modified the user interface to add median mosaics, PCA change detection and NDVI-based forest/non-forest maps.

11. Capacity Building  Since the Data Cube concept is a rather new concept for the satellite and user community, there is a need to build capacity and provide training for those countries and specific users that desire to use this capability. Here is a status...  World Bank (WB) – SEO is meeting with WB in Washington on April 13. It is believed they desire a Data Cube workshop in Washington later this year for WB members and supported countries. This connection was made by Alex Held (CSIRO).  FAO – SEO is working with Erik Lindquist on a plan to accompany him on a future country visit to better understand the FAO workflow for developing forest maps. FAO and the SEO have agreed on several small tasks in 2016 to connect the Data Cube to the FAO workflow. FAO believes Data Cubes could reduce data preparation time.  GFOI (Global Forest Observation Initiative) – Plans to develop an end-to-end demonstration in one country where GFOI, FAO, and CEOS bring their relevant contributions together to support improved forest mapping and increased country capacity to use satellite data. Candidates are Nepal, Indonesia and Myanmar.  SilvaCarbon - The SEO has participated in past workshops to provide COVE tool training. By the end of 2016, Data Cubes will be ready for delivery.  Interest in Data Cubes expressed by Mexico and the Balkans (led by Greece).

12. How can WGCapD Support?  The architecture will be ready for delivery in late 2016. There is a need to develop plans for training and capacity building.  Is it best to link this initiative with groups connected to WGCapD (e.g. SilvaCarbon, SERVIR) or should we consider a CEOS-specific contribution?  Could we include Data Cubes into future SAR training workshops in late 2016?  Might we identify people within CEOS agencies that could become “trainers” and attend workshops to train countries on how to set up Data Cubes and use the user interface tools?  What is your strategic advice regarding relationships with banks (e.g. World Bank, Asia Development Bank) and foundations (e.g. Gates, Clinton)? This will be a topic of discussion at the upcoming CEOS SIT Meeting in Italy.