1. IRI/LDEO Climate Data Library
International Research Institute for Climate and Society Columbia University

2. Overview multidimensional Specialized Data Tools Maproom
Generalized Data Tools
Data Viewer
Data Language
Dataset
Variable
ivar
multidimensional
So here is the IRI/LDEO Data Library shot at this, connecting the space of data and data manipulations. At the bottom we have the compute engine/data organization, which is what maps the data/manipulation space into URLs, i.e. the WWW.
Built on top of that are some general data tools, i.e. they can be applied to any dataset and adapt accordingly. There is a data language, making it possible to specify sophisticated analy ses. And there is a data viewer, making it possible to quickly graph data in a number of standard ways.
And there are also more specialized tools, designed for particular audiences to view specific things. We have a Maproom (soon to be or already map rooms) which contains continuously updated views of aspects of the climate system, as well as specialized tools aimed at particular audiences that let a use extract views/data with a few clicks.
There is a tradeoff here: the general tools are great, but require a user to navigate a vast set of datasets and a vast set of possible manipulations, which not everybody is up to. The specialized tools are a way to make sophisticated c alculations easy to access.
IRI Data Collection
URL/URI for data, calculations, figs, etc

3. Data Flow based Analysis with explicit semantics
Results
analysis
data
data
Semantic Web
One can even include the reverse linkages, so that the final result is connected to the earlier results, enchancing the documentation of the results way beyond what is seen in a paper or presentation. This has the potential to greatly improve the reproducibility of the analysis, since anyone that the is really interested can ultimately see all the steps. Also, the analysis can easily be rerun, if, for example, the data is extended in time or otherwise modified.
Data
Data
analysis

4. “geolocated by lat/lon” multidimensional “geolocation by
IRI Data Collection
Economics
Public Health
“geolocated by
entity”
Ocean/Atm
“geolocated by lat/lon”
multidimensional
GIS
“geolocation by
vector object or
projection metadata”
spectral harmonics
equal-area grids
GRIB grid codes
climate divisions
IRI Data
Collection
Dataset
Variable
ivar
multidimensional
Data Cultures “Broadly Speaking”
We started with Oceans/Atm – multidimensional geolocated by lat/lon – with exceptions that tend to get handled in non-standard ways. GRIB in some ways is the 600 lb gorilla, since it is very similar in style to the WCS standard in that metadata carries the geolocation, but, of course, it is a difficult if-not- impossible standard to completely code for.
Economics/Public Health geolocation by entity, mostly tables
GIS geolocation by vector object or by projection metadata -- mostly a 2D mindset in the tools, which makes time analysis of data difficult.
IRI Data Collection – nested datasets, multidimensional variables with independent variables
All variables have attributes which can affect the way the data is processed and/or displayed
We use dimensions for a lot –lon,lat,height,time forecast time, lead time, eigenvalue number, member number, country, district, category
Dataflow, delayed execution architecture, which means one can usefully define many dimensions even when one tends to only evaluate a few realizations at a time.

5. GRIB netCDF images binary Database Tables queries OpenDAP
IRI Data Collection
GRIB
netCDF
images
binary
spreadsheets
shapefiles
Database
Tables
queries
Servers
OpenDAP
THREDDS
images w/proj
IRI Data
Collection
Dataset
Variable
ivar
Having got all that data into the data library, we can process it in a uniform way.
The data structure leads directly to calculations and “virtual variables”, i.e. many
of the Data Library entries are actually calculations done on other entities, e.g.
PressureLevel data zonal velocities computed from hybrid level divergence and
vorticity, or sea surface temperature anomaly computed from sea surface temperature
and sea surface temperature climatology
User Interface – the data collection structure and metadata is used to generate a
web interface to the data – provides navigation through the datasets, a viewer that
slices and dices, many manipulations and calculations.
We also generate output Data Files in many different formats, as well as tables of
various kinds, many of which are useful to one kind of user or another, i.e. different data cultures
have different preferred formats. Atm/Ocean like netcdf and straight binary, GIS prefers sets of
images, Public Health prefers tables.
We also act as a data server using OpenDAP and THREDDS, again mostly useful for Ocean/Atm
We have perhaps implemented OpenGIS Web Map Server v1.3 – a bit of a mistake, since v1.3
is the next version rather than the currently widely used one. Time will correct this, with any luck.
It is important to note that everything following from the structure and attributes in the IRI Data Collection, no additional configuration is done to control the conversions to different fromats or to serve the data with different protocols. Not all data can be served in all formats.
OpenDAP/THREDDS is particularly important because it can express any dataset and/or any analysis, so that I can transfer calculations between servers. At least, it will once I code transmission of SimpleFeatures with OpenDAP.

6. descriptive and navigational pages
IRI Data Collection
GRIB
netCDF
images
binary
spreadsheets
shapefiles
Database
Tables
queries
Servers
OpenDAP
THREDDS
images w/proj
IRI Data
Collection
Dataset
Variable
ivar
Calculations
“virtual variables”
images
graphics
descriptive and navigational pages
Having got all that data into the data library, we can process it in a uniform way.
The data structure leads directly to calculations and “virtual variables”, i.e. many
of the Data Library entries are actually calculations done on other entities, e.g.
PressureLevel data zonal velocities computed from hybrid level divergence and
vorticity, or sea surface temperature anomaly computed from sea surface temperature
and sea surface temperature climatology
User Interface – the data collection structure and metadata is used to generate a
web interface to the data – provides navigation through the datasets, a viewer that
slices and dices, many manipulations and calculations.
We also generate output Data Files in many different formats, as well as tables of
various kinds, many of which are useful to one kind of user or another, i.e. different data cultures
have different preferred formats. Atm/Ocean like netcdf and straight binary, GIS prefers sets of
images, Public Health prefers tables.
We also act as a data server using OpenDAP and THREDDS, again mostly useful for Ocean/Atm
We have perhaps implemented OpenGIS Web Map Server v1.3 – a bit of a mistake, since v1.3
is the next version rather than the currently widely used one. Time will correct this, with any luck.
It is important to note that everything following from the structure and attributes in the IRI Data Collection, no additional configuration is done to control the conversions to different fromats or to serve the data with different protocols. Not all data can be served in all formats.
OpenDAP/THREDDS is particularly important because it can express any dataset and/or any analysis, so that I can transfer calculations between servers. At least, it will once I code transmission of SimpleFeatures with OpenDAP.
Clients
OpenDAP
THREDDS
Data Files
netcdf
binary
images
Tables
OpenGIS
WMS/WCS

7. Data Page IRI General Data Tools
While that is all there, that is not what the user sees (bad advertising for us, b ut life is easier for the user). This is a page representing a dataset, in this case weekly sst, ssta and error fields. It has links some analyses (as well as the full language interface – expert mode). It has some basic informatnion – note the last_modified and expiries times – as well as links to pages of virtual files in various commonly used formats (with coachs for commonly used programs).

8. Data Viewer IRI General Data Tools
Here is the Data .iewer. As I said earlier, it can be applied to any dataset or variable, and what you see v aries according to the structure and metdata of the dataset. In this case, it is applied to a dataset, so one of the controls is which variable to look at. One can enter alternative dates (or a range of dates for an animation) change which dimensions are plotted/selected, many of the standard controls one might want in a data viewer. Click/drag to zoom got added a year ago, turned out to be more important that I thought, shame on me.
Anyway, clearly it is simply creating the appropriate url for the graphic and letting the underlying compute engine generate the figure from the data.
One feature that makes it very much a web interface, is the cut-and-paste link option on the bottom. It takes you to….

9. IRI Map Room

10. Malaria Early Warning System
IRI Map Room
Malaria Early Warning System
• Front page illustrates most recent dekadal rainfall estimates
(FEWS RFE)
• Administrative and epidemiological overlays available
• Change dates to view different time periods
• Click and drag box across map to zoom

11. MEWS Time Series Analyses
IRI Map Room
MEWS Time Series Analyses
STEP 1: Select
size of domain
for analysis
Administrative District OR
Box – 11km, 33km,
55km, 111km
STEP 2: Select
location for which
analysis will be
created

12. MEWS Time Series Analyses
IRI Map Room
MEWS Time Series Analyses

13. IRI Maproom Maproom link Global Regional Applications Health
Food Security
Fires
IFRC

14. Finding the Data and Maps
Semantic Web metadata/navigation/provenance

15. Faceted Search http://iridl.ldeo.columbia.edu/ontologies/query2.pl?...
Again as an example, here is a user interface built on that ontological structure

16. Metadata Browser http://iridl.ldeo.columbia.edu/ontologies/browse.pl
Browses through the metadata the search interface is based on. Includes several alternative term-sets for the search interface, and a number of ontologies collected from the Internet.

17. Generalizing the IRI plugin
Currently entities are a view plus (optionally) a set of local data views
Local view is click-on-map plus some parameter choices
Would be good to generalize those parameter choices
Parameter choices should only appear when appropriate

18. Generalizing the IRI plugin
Currently entities are selected from a cascading menu
IRI should serve that menu
Menu should be generated in a meaningful way
Do we need to consider some other way?

19. Possible Basis for Cascading Menu in SERVIR
Dataset Hierarchy
Dataset Faceted Search
Maproom Faceted Search
Maproom Hierarchy

20. Generalizing the IRI plugin
Making better use of WMS
Should separate out the layers
Transparent Background on Images
Could include local info references within WMS