Tony Rees and Glenelg Smith Divisional Data Centre + Remote Sensing Facility CSIRO Marine Research, Australia Application of c-squares spatial indexing to an archive of remotely sensed data

The starting point … Archive of c.60,000 satellite AVHRR scenes, current, accumulating at another 15+ per day Region covered varies for each scene Metadata (satellite, orbit, time parameters) and computed polygon (e.g. 7 points per side) available for each scene Determining which scenes include any given point, on- the-fly (to answer user’s request) is a standard requirement, but not trivial – available methods may be slow, or inaccurate, or both...

example in more detail... Example – may need to query which out of these four scenes includes point “x” (in reality, stop/start points of scenes also vary, also other scenes cross at different angle/s) x

Options for spatial indexing Bounding polygons Require a GIS system (or similar) for spatial queries (polygon overlays); can be slow to test many thousands of scenes Can be problems with footprints which cross the date line, or include a pole Bounding rectangles Computationally cheap, rapid to query, no GIS required Typically a poor fit to actual data footprints (typically these are not square, or even nearly so) – many “false positives” Can be problems with footprints which cross the date line, or include a pole Gridded (tile based) representation Potentially simple to query, no GIS required May be poor fit to actual data footprints if tile size is large (smaller tiles best, however index gets larger – tradeoff required here) Compromise approach – slightly inexact compared with polygon overlays, but should be better performance for real-time queries (most computation has been done in advance) Problems with footprints which cross the date line, or include a pole, can be pre- handled

C-squares basics Based on a tiled (gridded) representation of the earth’s surface, at choice of 10 x 10, 5 x 5, 1 x 1, 0.5 x 0.5 degree tiles, etc. (0.5 x 0.5 degree tiles are used in this example) Every tile has a unique, hierarchical alphanumeric ID (incorporates the ID of all parent, grandparent, etc. in every child tile ID) Dataset (=scene) extents are represented by a list of all the tiles included in, or intersected by the footprint Spatial search comprises looking for one or more tile IDs in the set associated with any dataset (= simple text search). (more details – see )

C-squares nomenclature – ten degree squares (follows WMO 10 x 10º square nomenclature)

Adaptive square size – 10 x 10, 5 x 5, 1 x 1, 0.5 x 0.5 degrees (etc.) – produces quadtree-like efficiencies (“compaction”) (this example shows detail to 1- degree square level) 10 degree square: 3213:*** 5 degree square: 3214:1** 1 degree square: 3315:100 - can reduce storage requirement by up to 90%, cf. constant- resolution encoding

A real-world example NOAA Jun :23 10 x 10 degree squares (28) (base level of hierarchy, cannot compact) 5 x 5 degree squares (99) = 36 after “compaction” 1 x 1 degree squares (1,982) = 515 after comp. 0.5 x 0.5 degree squares (7,691) = 704 after comp. 0.5 x 0.5 degree squares - detail

Metadata table concept – 2 “search” tables only Scene details (1 row per scene) Scene <> c-square pairs ( rows per scene)

Clickable map interface

Example search result

SQL for spatial search (example for 0.5 degree search square) select distinct A.scene_id, B.satellite, B.scene_date_time, B.image_location from satdata_csq_all A, scene_info B where ( (sqrsize = 0.5 and (A.csquare = search_csq -- e.g. 3414:100:1 (0.5 degree square) or A.csquare = substr(search_csq,1,8)|| ': * ' -- 1 level of compression or A.csquare = substr(search_csq,1,6)|| ' ** : * ' -- 2 levels of compression or A.csquare = substr(search_csq,1,4)||': *** : * ') -- 3 levels of compression ) -- (other supported search square size options inserted here) ) and (startdate is null or B.scene_date_time >= startdate) and (enddate is null or B.scene_date_time <= enddate) and (sat = 'any' or B.satellite = sat) and A.scene_id = B.scene_id order by B.scene_date_time, B.satellite;

Summary of steps involved (1) Encoding – using custom polygon-to-c-squares algorithm (first attempt: 3+ mins per scene; second attempt: 30 seconds per scene to encode) (2) Storage – first attempt: all scene/c-square pairs in one large table, second attempt: partitioned by 10 x 10º “bins” (100 smaller tables, for faster searching) (3) Querying – using standard SQL and text matches (4) (optional) Visualising – generating footprints, using web call to multi-purpose c-squares mapper More information: c-squares website - CMR Satellite Data spatial search: