Database management system Data analytics system: Modeling change from large-scale high-dimensional spatio-temporal array data Meng Lu and Edzer Pebesma Institute for Geoinformatics, University of Muenster, Germany Contact: Meng Lu meng.lu@uni-muenster.de Introduction The massive data that comes from earth observation satellites and other sensors provide significant information for modeling global change. At the same time, high dimensionality and large size of the data has brought challenges in data acquisition, management, effective querying and processing. In addition, the output of earth system modeling tends to be data intensive and needs methodologies for storage, validation, analysis and visualization (e.g. as maps). An important proportion of earth system observations and simulated data can be represented as multi-dimensional array data, which has received increasing attention in big data management and spatio-temporal analysis. Array based data management and analysis brings opportunities in modeling change with large-scale high-dimensional data. Examples of Array Data 1-D: Time series 2-D: Satellite images 3-D: Image time series 3-D: Sediment/nutrient in flow 4-D: Hyper-spectral remote sensing time series data x,y stand for spatial coordinates, t stands for time, z for height Time series analysis Goal Spatial-temporal change detection and quantification from multi-dimensional array data Time domain Frequency domain Trend analysis Pattern identification Seasonal variation Periodical pattern Forecasting Other cyclic variation Multi-dimensional array on t Change detection in NDVI time series Source: Verbesselt et al. (2009) Analysis in frequency domain: Spectrum density estimation Spatio-temporal analysis Research Questions How to model spatio-temporal change? • How to reduce dimensions spatially and temporally, or thematically? How to analyze array data? • How to extend existing GIS functions to work on multidimensional arrays? • How to combine data sets of different dimensionality or different resolutions? • Can map algebra be extended to an intelligible array algebra? • In what sense are space and time special, as dimensions, compared to other properties? Dimension Reduction --Practical PCA (Principle Component Analysis) Works Time Spatial rainfall variability 1) What is the spatial variability of the long- term rainfall? 1967 1968 1969 … 1 245 286 256 2 223 271 268 3 234 264 253 …. Gage ID 2-Dimensional array data for summer average rainfall amount, from year 1967 to 1999, 83 gages selected Map of rainfall gage network in a semi-arid watershed (ca. 150 square Km) (Walnut Gulch Experimental Watershed, Arizona, US) Temporal rainfall variability 2) What is the temporal variability of rainfall within the whole watershed ? reflectance PC 1 PC 2 PC 3 PC 4 loading loading loading loading Spatial distribution of PC1 loadings (top), PC2 loadings (middle) and PC3 loadings (bottom); gage as variable, time as record Tools: Database Management Systems PC 1st 2nd 3rd 4th Others Proportion of variance 80% 6% 3% 2.3% 8.7% Data analytics system year year year year Database management system + Data analytics system: The way to go? PC 1 PC 2 PC 3 PC 4 Variance explained by each PC, gage as variable, time as record Interpretation: Each of the circle represent the loadings of PC for each gage. The size of the circle indicates the magnitude of the PC loadings (how much the variable contributes to the variance). The blue and black indicated negative and positive of PC loadings. The first PC explains more than 80% of the total variance. The rather uniform distribution of PC1 loadings suggests that the spatial variability of climate across the watershed is small. PC2 contributes to 6% of total variance, but is more interesting. It forms a loading pattern that varies from west to east, which could be interpreted as weather variability. PC3 also shows spatial weather pattern, which is likely to show the variability from south to north. Noting that the PC3 explains less variance than PC2, the signal in west-east direction is dominating the spatial rainfall variability related to weather. A programming language for statistics and graphics Open source Difficult to scale R Main memory SciDB Array-based Database management and analytics system Open source Partially open source Scalable commercial version scalable AQL (Array Query Language) AFL (Array Functional Language) mixture of SQL syntax and trees of algebraic operators RaSQL (Raster SQL) based on SQL-92 and implemented array algebra Chunks maps onto a disk block. Tiles stored as BLOBs (binary large objects) in RDBMS R and Python interface No R or Python interface General features License Scalability Query Language Storage Interaction with .. predicted predicted predicted predicted Gage ID Gage ID Gage ID Gage ID Loadings of PC 1 to PC 4 (up) and predictions using different loadings from PC1 to PC4 (bottom). Time as variable, gage as record Interpretation: the loadings of PC1 to PC4 represent the variability of each year that contribute to the PC. Note that when using gages as variables (shown on right), the first PC represents the mean rainfall of the period. The data has been centered with the removal of the long-term climate information. The weather variability in each gage could be observed from each of the PCs. The bottom plot shows the predictions (scores) for each record, from which the variability of rainfall that each gage received could be observed.