1. ESSE Environmental Scenario Search Engine for the Data Services Grid Mikhail Zhizhin, Geophysical Center Russian Academy of Sciences jjn@wdcb.ru Eric Kihn, National Geophysical Data Center NOAA Eric.A.Kihn@noaa.gov

2. www.wdcb.ru Geophysical Center Russian Academy of Sciences World Data Centers for Solid Earth and Solar- Terrestrial Physics Environmental data archives – paper, tapes, files, databases, e-journals… International network for geophysical data exchange with the US, Japan, China, … Computer center, Linux cluster, fiber optics Part of the European GRID infrastructure EGEE, Russian GRID Virtual Organization e-Earth

3. 50 years ago – International Geophysical Year – IGY1957 Total data volume ~ 1 Gb Exchange ~ 1 Mb/year

4. Yesterday – databases, Internet, web – Y2K Total data volume ~ 1 Tb Exchange ~ 1 Gb/year

5. Tomorrow – Electronic Geophysical Year – EGY2007 Total data volume ~ 1 Pb Exchange ~ 1 Tb/year

6. Boulder Moscow Kamchatka Nagoya SydneyGrahamstown SPIDR – Space Physics Interactive Data Resource SPIDR 3 SPIDR 2 http://spidr.ngdc.noaa.gov Beijing

7. Cross-disciplinary data exchange Users need data from different disciplines Rapid growth of the data volume and data demand requires new tools for the data management and the data mining

8. “Metcalfe’s law” for databases The utility of N independent data sets seems to increase super-linearly One can find N(N-1) ≈ N 2 relations between data sources, that is their utility grows ≈ N 2 It is more efficient ot use several data sources than one archive

9. Sources of data inflation? 1.New versions 2.Derived data products 3.Reanalysis Products of Level 1 (NASA terminology) take 10% of the Level 0 volume, but the number of the Level 1 products is increasing. If the volume of the Level 0 data grows as N, then the volume of Level 1 data is growing as N 2.

10. Observations + Model = Reanalysis 1.Direct observations, including raw and processed data, e.g. meteorological station or satellite. 2.Numerical model “knows” physics, uses direct observations as boundary values, e.g. Global Circulation Model. Input data volume (irregular grid) is less than the output volume (regular grid). 3.Reanalysis – accumulated output of the numerical model runs based on the direct observations for a long time period, say 50 years.

11. D-day reanalysis – morning (after ECMWF) June 6 th, 1944, midnightJune 6 th, 1944, 6 AM

12. D-day reanalysis – evening (after ECMWF) June 6 th, 1944, 6 PMJune 6 th, 1944, 12 AM

13. Data inflation after reanalysis Modern global atmospheric circulation model (GCM) at 2.5 o (latitude) x 2.5 o (longitude) x 20 (levels) = 10 6 gridpoints. GCM outputs "high-frequency" data every six hours of simulation time, so ~ 1 Gb of data per simulation day. By contrast, the world-wide daily meteorological observational data collected over the Global Telecommunications System, is ~ 200 Mb. As an extreme, to run the GCM for 50 years of simulation time will provide 40 Tb of data.

14. Input: ground and satellite data from SPIDR Space weather numerical models Output: high-resolution representation of the near-Earth space Space Weather Reanalysis

15. ESSE solutions Do not use data files, use distributed databases Optimize data model for the typical data request Virtualize data sources using grid (web) services Metadata schema describes parameters, grids, formulas for virtual parameters (e.g., wind speed from U- and V-wind) Search for events in the environment by the “scenario” in natural language terms Translate the scenario into the parallel request to the databases using fuzzy logic

16. ESSE architecture Fuzzy logic engine performs searching and statistical analysis of the distribution of the identified events Parallel mining of several distributed data sources, possibly from different subject areas Both the fuzzy logic engine and data sources implemented as Grid (web) services Interfaces and data structures can be obtained from the definitions of the web-services (WSDL) Web services and prototype user interface are installed on two mirror servers: −Boulder, US −Moscow, Russia

17. Parallel database cluster (NCEP reanalysis)

18. ESSE “time series” data model Indexed lat-lon grids of time series in BLOBs

19. What is fuzzy logic? Fuzzy logic uses set membership values between and including 0 and 1, allowing for partial membership in a set. Fuzzy logic is convenient for representing human linguistic terms and imprecise concepts (“slightly”, “quite”, “very”). Fuzzy membership functions

20. What good is fuzzy logic for ESSE? Fuzzy engine allows to build queries in human linguistic terms: (VERY LARGE “wind speed") AND (AVERAGE "surface temperature") AND (“relative humidity“ ABOUT 60%) You can use the same terms for different value ranges: AVERAGE TEMPERATURE for Africa is not the same as for Syberia. Results are given as a list of “most likely” events. Each event is assigned a value, representing its “likeliness”.

21. “High” Wind “Average” Temperature “About” 60% Humidity

22. Prototype workflow and UI Prototype UI implemented as a web-application Discover data sources by keyword-based metadata search Use predefined weather events (e.g. “ice storm”, “flood”) Define the event as a combination of fuzzy conditions on a set of environmental parameters (e.g. “high temperature and low relative humidity”) Review statistics for the detected events Visualize the selected event as time series plots or contour maps Download the event data in self-describing format (NetCDF or HDF) to the user’s workstation

23. Setting spatial locations Select a set of "probes" (representing spatial locations of interest, e.g. New York) where the desired event may occur.

24. Defining fuzzy search criteria Select several parameters for the event from a list. Set the fuzzy constraints on the parameters for the event (e.g. “very high temperature”, “very high humidity”).

25. Working with scenarios The user may search for a desired scenario by describing several subsequent events

26. Search Results “Score” represents the “likeliness” of each event in a numerical form. The results page provides links to visualization and data export pages.

27. Visualizing event as time series

28. Visualizing event in 5D

29. Visualizing event from satellites

30. What do we get at the end? Using the “time machine”, we can see the weather on the D-day, or the Rita hurricane, or the typical September day in San Diego. Statistics to estimate risk from natural disasters, global climate change, realistic weather in movies, computer games, simulators When Tim Berners-Lee uses semantic web to find a photo of the Eiffel Tower on a sunny summer day, ESSE can provide a list of sunny days to be merged with the list of images named with “eiffel”