1 Space-Time Datasets in Arc Hydro II by Steve Grise (ESRI), David Maidment, Ernest To, Clark Siler (CRWR)

2 CUAHSI Observations Data Model Space-Time Datasets Sensor and laboratory databases From Robert Vertessy, CSIRO, Australia

3 Space-Time Dataset A set of records with –Time –Location –1 or more variables time variables x, y, z c b a c b a c b a c b a c b a c b a

4 Example: River Flow For surface water resources, stream gages have a fixed location with continuous measurements over time Variables related to stream flow are the most common measurements Data is typically measured regularly and continuously, but there are often gaps due to device errors or routine maintenance There are also cases of overflow or dry conditions where the values are outside of the range of measurement for the device time variables fixed x, y, z c b a c b a c b a c b a c b a c b a c b a c b a c b a c b a c b a Data gap c b a c b a An overflow condition could be recorded simply as > 500 cubic feet/second stream flow river height mean velocity

5 Example: Water Quality For water quality, sampling sites have a fixed location with intermittent measurements over time –Four times per year is typical There is a sampling “event”, and a large number of chemical species are produced through laboratory analysis of water samples Data has metadata that specifies what laboratory procedure was used Some data require a qualifier to be properly interpreted like “<“ to indicate a measurement that is below a detection limit Data are “Time stamped” with the time that the sampling event began. They are considered “instantaneous data” observed at that time. time variables fixed x, y, z c b a c b a c b a c b a turbidity nitrate conductivity c b a t1t1 t2t2 t3t3 t4t4 t5t5 water quality sample

6 Display of data that vary in latitude, longitude, depth and time (Ernest To)

7 Data Structure for a single variable These data are extracted from CUAHSI ODM, and Offset = Depth in this instance

8 Example: Water Reservoir For water reservoirs, data is recorded for the water level of the reservoir, along with all inflows and outflows A flow time series dataset describes the information required to do a water balance on the reservoir contents “Flow variables” apply over the entire time interval; “state variables” apply at instants of time at beginning and end of interval; Typically there are derived datasets –Monthly data compiled from daily data –Annual data from monthly data Data are recorded regularly through time time variables x, y, z c b a c b a c b a c b a c b a c b a c b a c b a inflow outflow storage Inflow Outflow Precip Evap Storage

9 Example: Water Rights Analysis A water resources simulation model is run for monthly time steps for ~50 years and it computes ~40 variables related to water supply reliability –Water rights diversion points, –Reservoirs, and –Other “control points” on the stream system Each model “run” generates millions of data values. The “data cube” is completely filled in because it is all computed Information products needed are graphs of variables at points, maps of feature conditions at a single time point, and maps of averages through a defined time interval of feature conditions (i.e. dataset derived “on the fly”) time variables x, y, z c b a c b a c b a c b a c b a c b a c b a c b a % of time reliability % of volume reliability flow Study area (watershed) modeled point features

10 Maps and Charts Plot a map for a time point Plot a graph for a space point SpaceTime A set of variables ……

11 Example: Climate and Weather Observations that come from weather balloons and other measuring devices have dynamic location properties For weather and climate forecast datasets, each data point represents an area with consistent atmospheric characteristics For weather observations, a large amount of data comes from fixed stations so the datasets are similar to stream gage datasets time variables x, y, z c b a c b a c b a c b a c b a c b a c b a c b a temperature air pressure relative humidity t t t t t t t t balloon trajectory forecast data

12 Example: Species Observations In this type of dataset, observers are frequently moving along a path such as a hiking trail or a boat cruise Multiple species may be observed, and even the lack of information is significant Data is often recorded using offsets from the observer location time variables species group “a” species group “b” species group “c” aa c a cc b a a a c c c b x, y, z

13 Other Datasets There are many types of Time Series Datasets –Observations –Samples –Model results –Remote sensing data/imaging Concepts are useful for many communities –Science –Business –Statistics –Planning –Health –Transportation

14 Space-Time Datasets: Implementation Concepts The general pattern can be described as –Time Series Values The data –Time Series Descriptions The metadata There are a number of ways to store and manage this information in a computer system

15 Example: Arc Hydro Version 1 Implementation Approach works well for an individual project with stream gage and other surface water data Constrained to 1 variable per time step Limited in its ability to handle location –Changes in x, y, z over time i.e., Marine and species observation datasets have an additional “cruise” or “observation” concepts linking multiple features FeatureID provided some flexibility, but did not directly support unique identity for features at different time steps In general, implementation patterns for the feature portion of the data model were not explored/explained –Different spatial representations Raster data Multidimensional data –GIS Layers and their properties were considered but not explained –Inefficient approach with multiple variables

16 Arc Hydro Version 2 Improvements 1.GIS Layer and representation focus 2.Use of Metadata 3.Improved Efficiency 4.More documented implementation patterns 5.General Time Series Dataset concepts applicable to many communities

17 Representations in GIS Time series data can be represented in different ways –Charts and graphs –Modeling simulations –Surfaces –Rasters –Vector feature classes GIS Layers provide a convenient set of representation types for different views into Time Series Datasets

18 Layers Layers represent data –Layer Properties Queries Representation types Display/styles Variable(s) Labels Layers deal with presentation of data, and they are closely linked to the data storage model

19 Vector Layers

20 Raster Layers

21 Metadata Each Time Series Dataset is a complex structure, and there are many patterns Metadata is a tool that can be used to document datasets –Facilitates search and discovery –Aids in sharing and re-use of data –Standards-based metadata/cataloging methods are available In practice, once users understand the dataset, they tend to work with the Time Series Values and rarely re-visit the metadata in applications Shift in Arc Hydro II to use of FGDC/ISO metadata to document datasets and variables –For the grey boxes in the diagram shown here

22 Improved Efficiency In Arc Hydro 1, we tried to put all time series values into a single table This implied creating rows for each variable, or adding additional columns/TSValues rows to datasets Since it was table-based, it did not include feature and raster representations, which required additional processing steps By promoting multiple datasets with a flexible approach for managing variables, data management activities will be improved, especially for larger datasets Single Time Series Table with 1 variable Time Series Datasets with multiple variables

23 Improved Efficiency For display, layers are built using Time Series Datasets Typically we “Select” or “Slice” 1 variable for presentation Layers can be built from source Values using InMemory layers, or built from Time Series Datasets Time Series Layers with variable(s) Time Series Datasets with variable(s)

24 Implementation Patterns Patterns will be explained for different types of implementations –Small/single project –Workgroup or multi-project environments –Very large datasets –Different spatial representation options –… –One key difference is that there will be multiple datasets – basically one dataset per set of time series values Different dataset names and storage strategies Documented with metadata

25 A General Spatial-Temporal Model A Space-Time Dataset is a set of records with –Time –Location –1 or more variables time variables x, y, z c b a c b a c b a c b a c b a c b a