Interoperable Information System of Systems for HTAP Rudolf B. Husar and Rich Scheffe With Erin Robinson Presented at HTAP Workshop, WMO, Geneva, January 26, 2006

Intercontinental Transport Events have Influenced AQ Regulations ..and now Policy? –> Air Quality Management Web-Based Collab. in Virtual Workgroups –> Science, the Process Near-Real-Time Data Access & Integration –>Technologies and Tools US-Daily Avg. PM25 Dust: Seasonal + Events

HTAP Characteristics Stakeholders are autonomous countries, organizations Diverse perspectives on the issues, solutions Pollution sources and impacts are hemispheric Information integration is key to success Observations and models are ready for integration

The HTAP - GEOSS GEOSS is a good system model for HTAP Stakeholders are autonomous countries, organizations Diverse perspectives on the issues, solutions Recognition of the need to understand, act Pollution sources and impacts are hemispheric Observations and models are ready for integration Information integration is key to success Hence GEOSS is a good system model for HTAP HTAP is a suitable use case for GEOSS Some Characteristics of System of Systems (SoS) Autonomous constituents managed independently Independent evolution of each constituent Displays emergent (novel, unpredictable) behavior

Summary There is an ‘alignment of stars’ for integrating heterogeneous HTAP data System of Systems architecture is suitable for integrating HTAP data Standard data access is a key interoperability protocol Heterogeneous data can be non-intrusively standardized by mediators Service-based software architecture delivers tailored products to diverse uses Federated data and shared web-based tools are in use DataFed already includes over 100 datasets (emissions, ground, satellite) The system has been applied to EPA regulatory development However, DataFed is just one of the many mediator nodes, but these need to be connected Much more data would need to be federated HTAP model-data comparison would be an attractive use case WMO-GAW could be an effective neutral facilitator for global data integration using GEOSS (system of systems) principles

GEOSS Architecture Framework - DSS for HTAP Models HTAP Reports: Model Estimates, Obs. Evidence Knowledge into the Minds of AQ Analysts HTAP Knowledge into the Minds of Policy Makers HTAP Knowledge into the Minds of Policy Analysts Observations Decisions Decision Support System The decision support system for hemispheric transport assessment performs both model to model comparison as well as model to observation comparison. The reports of analyses are submitted to policy analysts in the hemispheric transport Task Force. The policy makers decide whether and in what way hemispheric pollutant transport should be included in International Long-Range Transport Protocols. GEOSS Architecture Framework

System of Systems: Interoperability Connecting Machines and People Interoperability Stack: Key concept of the Web System components have to be interoperable at each layer Amplify Individuals Connect Minds Open Architecture Data Standards Service Based IP – Internet Protocol

How do you agree on Interoperability?

Are WCS/WMS Space-Time-Parameter queries Convergence Protocols What few things must be the same so that everything else can be different? Are WCS/WMS Space-Time-Parameter queries GetCapabilities GetData Capabilities, ‘Profile’ Data Where? When? What? Which Format? Server Back End Std. Interface Client Front End WCS/WMS App Query GetData Standards Where? BBOX OGC, ISO When? Time What? Temperature CF Format netCDF, HDF.. CF, EOS, OGC T1 T2 Convergence Protocols

Lets agree on Space-Time-Parameter Data Access Query Protocol

From ‘Stovepipe’ to Federated Architectures Current info systems are project/program oriented and provide end-to-end solutions Info Users Data Providers Info System AIRNow Public Each project/program can be augmented by Federation data and services Part of the data resources of any project can be shared for re-use through DataFed Model Compliance Manager Scientist Science DAACs Data processing and rendering can then be performed through web services Through the Federation, the data are homogenized into multi-dimensional cubes Traditional air quality information systems provide end-to-end solutions through dedicated ‘Stovepipe’ processing streams However, many formerly closed data systems are now opening up and they share their data – through DataFed or other mediators In a federated approach data are homogenized and augmented with services and tools streams… The federated resources in turn can be fed back into individual project processing streams The transition from the stovepipe to the networked, service oriented architecture can be gradual and non-disruptive.

Web Services and Workflow for Loose Coupling Obs. & Models Decision Support System Control Std. Interface Std. Interface Data Reports Datasets Data Views Integrated Data Reports Web Service Interaction Service Chaining & Workflow Service Broker Emergent behavior in a network that is only created by the combination, not by the individual members. Emergent behavior: novel and coherent (logically connected, purposeful, meaningful) structures, patterns and properties arising from self-organization (mashers are self-organizers) in complex systems Wikipedia Emergent behavior is created by mashing Mashing occurs when open on both ends of network.- diffusion Input side – grow by assimilation Output side – Harvest (i.e. DataFed harvests datasets) Find Publish Bind Service User Service Provider

Collaborative Reporting and Dynamic Delivery Obs. & Models Decision Support System Control Std. Interface Std. Interface Data Reports Datasets Data Views Integrated Data Reports Analysis Reports: Information supplied by many Needs continuous program feedback Report needs many authors Wiki technologies are for collaborative writing Co Writing - Wiki Dynamic Delivery: Much of the content is dynamic Animated presentations are compelling Movies and screencasts are for dynamic delivery Screencast

Model-Data Comparison Workflow Software VIEWS Chemical Data Ft. Collins, CO Std I/O Model- Data Comparison Workflow AeroCom Chemical Models Paris, FR Std I/O Workflow Flow Program Lego-like assembly of component WMS, WCS OGC Services READ T

Sample of Federated Datasets

HTAP Model-Obs “Test/Benchmark”ed A virtual data center design to meet the needs of the TF HTAP model intercomparison and evaluation Archives of key observations and model data Tools for model intercomaprison

Tools.. Analyst Console User Layout

Imagine…More Shared Global Obs & Models 2007 Global Data & Models 2007++ More Global Data & Models

The Network Effect: Less Cost, More Benefits through Data Reuse Programs ask/get Data Orgs Develop Programs Public sets up Orgs Data Data Program Data Organization Program Data Re-Use Network Effect Public Program Organization Data Data Program Data Pay only once Richer content Less Prog. Cost More Knowledge Less Soc. Cost More Soc. Benefit VIEWS AeroCom Federated Information System Observations, Models, Services CAPITA Other Federations Obs Models Obs & Services Services Obs & Model Models & Services

Agile Information System Design Principles Guiding Idea: System of Systems GEOSS Domain of Interaction WMO/GAW & Stakeholders Infrastructure: Internet, Middleware, Web 2.0 Methods, Tools: Standards, Use Cases Possible DataFed Roles: Guiding Idea: Refine, solidify, evangelize the System of Systems idea Methods and Tools: Develop, promote, implement, verify standards; Cooperatively develop use cases Infrastructure: Maintain the DataFed middleware for distributed data access; Supply Agency champions with SoS ‘sales’ material Based on P. Senge et. al, 1994: Architecture of Learning Organizations(Link)

Organizations have different missions R. Scheffe Apparent divergences? Organizations have different missions NPS Protect ecosystems, AQ WQ NASA Explore fundamental Earth System Properties USDA Protect/optimize Ag and forest resources EPA (Protect human health & envi. Improve air, water, ecosystem Not necessarily. The various organizations have very different missions from protecting human health to understand Earth’s environment to the exploration of the Earth System. In the past, these different missions have lead to divergences. CDC To promote health and quality of life …. NOAA To understand and predict changes in the Earth’s environment and …

Stars aligned? Confluence again.. Alignment R. Scheffe Observation technologies {e.g., satellites} Computational power Collaboration (culture) empire building? Alignment Science, talent {embodied in AQ models and young geniuses} Budgets, agency collaboration resource/program accountability We know that we are in the midst of a sensing revolution, there are major improvements in data sharing technologies and that cultural changes are taking place toward increased collaboration. So, are the stars aligned to better integration? Information technologies {e.g., data sharing protocols} Accountability, ↓ regulatory assessments {e.g., NAS, CASAC}

Summary There is an ‘alignment of stars’ for integrating heterogeneous HTAP data System of Systems architecture is suitable for integrating HTAP data Standard data access is a key interoperability protocol Heterogeneous data can be non-intrusively standardized by mediators Service-based software architecture delivers tailored products to diverse uses Federated data and shared web-based tools are in use DataFed already includes over 100 datasets (emissions, ground, satellite) The system has been applied to EPA regulatory development However, DataFed is just one of the many mediator nodes, but these need to be connected Much more data would need to be federated HTAP model-data comparison would be an attractive use case WMO-GAW could be an effective neutral facilitator for global data integration using GEOSS (system of systems) principles