Technical and social requirements for a putative AR5 distributed database of simulation and other data Bryan Lawrence (obviously et.al.)

Outline IPCC beyond working group one. Technical Requirements for AR5 Social Requirements for AR5 Concluding remarks

Background Working Group I assesses the scientific aspects of the climate system and climate change. Working Group II assesses the vulnerability of socio-economic and natural systems to climate change, negative and positive consequences of climate change, and options for adapting to it. Working Group III assesses options for limiting greenhouse gas emissions and otherwise mitigating climate change. National Green Gas Inventory Programme DDC: Managed by the Task Group on Data and Scenario Support for Impact and Climate Analysis (TGICA): The mandate of the TGICA is to facilitate wide availability of climate change related data and scenarios to enable research and sharing of information across the three IPCC working groups.

Requirements WG I result from investigations of model climate sensitivity and model behavior WG II and III result from the investigation of questions of mitigation of and adaptation to climate change based on the results of WG I climate models. – Access to monthly mean and daily / sub-daily complete time series of model variables (GCM DM) –Model results on regular grids esp. ocean and regional modeling (GCM DM) – Climate model data products for adaptation and mitigation studies like climate indices (e.g. list from CLIVAR expert team, URL: – Visualization and user support (IPCC DDC)

Volumes Rough estimate of data volume –20 Models –10 experimental designs –10 realisations –10 TB IPCC relevant data per realisation Total: 20 PB plus/minus a factor of at least two Cannot be stored at one site (if nothing else, bandwidth becomes a showstopper)

Distributed Solution A distributed archive will be necessary, but some elements may be centralized and/or duplicated to avoid unnecessary distributed querying. Data catalogue Monthly means of standard variables Decadal and longer means. Implies that at very the least common interface descriptions are required, and very probably common software elements.

Technical Requirements For a PB-class addition to existing holdings, we can be fairly sure that data providers will not change their underlying architectures, and the data will be used for multiple projects! Archives will need to be interoperable in terms of interfaces Consumers will need to be able to Download data from multiple sources in the same form, and discover data of the same type in the same ways. Initiate and to perform data processing on multiple data sources in order to Reduce the amount of data to be transferred Calculate end-user data products Re-format query results to community specific standards

Technical Requirements (cont) Contents conform to CF; A major role for CMOR –Common data format still an issue! –CF-compliant NetCDF will be the default. Model and realisation differences important. Given multiple audiences, supplementary information about data and simulations will need to be more extensive than that provided for AR4. –Role for METAFOR/CURATOR/NMM/NumSim etc Standard parameters will be important, and possibly a “standard grid” Interoperability into the GIS community more important as WG2 and WG3 become first order citizens in the analysis. Observations and georeferencing more important as scales become more georelevant (i.e. going beyond “global-means”)

Access Control Access will need to be restricted: - protect data producers IPR - protect servers from denial of service - support load balancing It will be desirable to have “single-sign on” including common authorization, authentication (and accounting).

Social requirements - Money “Show me the money” All other things being equal, the ability to bring more funding to the problem (and more people) ought to result in a better solution, and this is best achieved by using multiple funding sources from within multiple nations. BUT It is extremely unlikely that a parent body will be established which can inherit the funding and deploy it according to a “master-plan” …so there are some significant problems to be overcome.

Social Requirements - Plans No centralised funding, but there does need to be a “master-plan”, which means: a) It needs to be developed! b) It needs to reflect that it will be delivered by multiple components, funded nationally (or internally within relevant bodies), and that those components will form two classes: I. Components deployed at multiple sites II. Components deployed locally which either conform to an agreed interface spec, or migrate local information/data into agreed formats.

Social Requirements - Rewards Funding bodies need to see results from their endeavours; and if they’re funding the development of software, multiple sites had better be deploying their software (but maybe not all). Those who have “technical development” programmes get their own rewards by others deploying their software.

Some Realities  It is highly unlikely that one technical solution (i.e. a solution funded from one camp) can be deployed which will cover all the requirements, and even if it were so, this would mean that other funding had not been utilized (or it had been spent, but not achieved anything useful for the master plan).  Technical solutions appropriate in one place may not be appropriate in another place (e.g. NASA constraints on software deployed on “their” websites etc).

Some Bryan Assertions We need to deploy something in two years! We can’t engineer anything from the ground up. We need to work with pieces of what we’ve got now. The price of failure would significantly impact on AR5. GO-ESSP is the right organisation to make this happen, but we haven’t got a great track record in doing things together! Some folk need to meet more often to make this work.

Summary All those point imply that the master plan should identify components which can be developed independently (preferably from multiple institutions and multiple countries) which together would achieve the technical (and scientific) requirements. The most important next step after formulating the requirements is to develop a methodology to architect a solution that meets the requirements, and DO IT QUICKLY, and not get hung up on DETAIL.