DRAFT April 28, 2005 ESIP AQ Cluster, Current Air Quality Information ‘Ecosystem’ (Draft for Feedback) AQ information includes emissions, ambient & satellite data and model outputs The distributed data are produced and provided by agencies, mostly through portals Providers have different access protocols, formats, and information usage conditions This lack of interoperability causes the under-utilization of the rich data resources

DRAFT April 28, 2005 ESIP AQ Cluster, Future Integrated AQ information System (Draft for Feedback) DataMart VIEWS NEISGEI AIRNow AQMod DAACs ASOS NEI Emission IDEA GASP Missions WeaMod Forecast GloMod FireInv Data Federation Distributed, Virtual, Uniform AQ Forecasting AQ Compliance Status and Trends Network Assess. Data Processing Filtering, Aggregation, Fusion Info Products Reports, Websites Data are maintained by custodians and exposed through ‘portals’ Mediators uniformly ‘wrap’ data and provide processing services Analysts program the services to create application-specific products Responsibility is shared among data providers and mediator/ integrators ESIPFed can provide the infrastructure and tools for the AQ info system Mediators

DRAFT April 28, 2005 ESIP AQ Cluster, Air Quality Information Providers AQ information includes emissions, ambient (surface) and satellite data and model outputs The information is provided by multiple Agencies, have different form and is AQ data usage requires considerable processing and integrating Emission Ambient Satellite Model Form | Content NOAA GASP NASA DAACs NASA IDEA NASA Missions EPA NEI EPA NEISGEI NOAA FireInv State/Local Emission NOAA ASOS RPO VIEWS EPA AIRNow EPA-AQS DataMart NOAA WeaMod EPA AQModel NASA GloModel NOAA Forecast

DRAFT April 28, 2005 ESIP AQ Cluster, DataFed WS Output Data Types UrlGranuleType TimePointType TimeDimensionType MapVectorType MapTrajectoryType MapTimePointType MapPointType MapLocationTableType MapImageLatLonType MapGridType ImageType HtmlType DotNetTableType DataSetType

DRAFT April 28, 2005 ESIP AQ Cluster, Interagency Working Group for Earth Obs. (IWGEO) Global Earth Observing System of Systems (GEOSS) T. Karl, NOAA, NCDC

Integrated Observing Systems OBSERVING SYSTEM TIMELINE 21 st Century Atmospheric Observations Data Systems Technology Development Innovations Breakthrough Efficiencies Cost Mass Productions Space Observations Ocean Observations Innovations Breakthrough Efficiencies Cost Mass Productions 6 T. Karl, NOAA, NCDC

DRAFT April 28, 2005 ESIP AQ Cluster, Tools for Users Pare down large file sizes of high resolution data and products. (re-) Group different data sets to create needed products – such as initialization files for model development, analysis, and intercomparison. Subset the data: –in parameter space –in physical space –in temporal space

DRAFT April 28, 2005 ESIP AQ Cluster, Collaborations: How do we get there? Data transport is being actively pursued: OPeNDAP, SOAP,... Earth System Partners need to be able to find and use various data sets, wherever they may be, whatever format... THREDDS can provide dynamic access and generate catalogs GCMD is a major resource for metadata management for the entire GeoSciences community- this activity must evolve! Ontology projects such as SWEET in conjunction with THREDDS and GCMD can provide individual data sources, data variables and metadata management for the community. G. Rutledge: Emerging Tools for Distributed Data Access and Collaborations G. Rutledge: Emerging Tools for Distributed Data Access and Collaborations Data systems based on the integration of independently developed system elements offer many more opportunities than more traditional centrally developed ones. P. Cornillon

DRAFT April 28, 2005 ESIP AQ Cluster, Tools & Data Information Systems Professional Productivity Information Management Shared Services... Becoming More Intelligent And Distributed Web Services Networks GIS is Evolving to a Web Services Environment

DRAFT April 28, 2005 ESIP AQ Cluster, GIS Networks Will Allow Us to Connect and Integrate Distributed GIS Resources... Making Virtual Collaborations Possible Maps Models GeoDataSets Peer-to-Peer GIS Metadata Data Models

DRAFT April 28, 2005 ESIP AQ Cluster, PervasiveComputing Terabyte/Second Communications Faster Hardware Faster Hardware Distributed Computing Distributed Computing Mobile/Wireless Mobile/Wireless Services Oriented Architecture Services Oriented Architecture Large Data Repositories Large Data Repositories GIS Software GIS Software Capacity In 10 Years 100x Computing 100x Computing 1000x Storage 1000x Storage 5000x Networks 5000x Networks Enabling Technology

DRAFT April 28, 2005 ESIP AQ Cluster, Internet Web Services GIS Portals Support Data Dissemination SelectFormat Data Conversion Zoom to Extent... Clip/Zip/Ship TIGER DXF VPF S57 GML XMC MIF Geomedia SDTS DLG DWG DGN CAD... Supporting Interoperability IMS Server IMS Server Support Many Formats Many Standard Formats And

DRAFT April 28, 2005 ESIP AQ Cluster, Interoperability Is Important Conversion Direct Read (API) DBMS Integration... Focus Is On Simple and Practical Approaches That Work Practical Approaches That Work Web Services GISServer There Are Many Standards... XML/SOAP

DRAFT April 28, 2005 ESIP AQ Cluster, Direct Read & Use Dynamic Read/Conversion/Use Custom Format Converters MIF GML M.S. MIF Standards And Direct Proprietary Interfaces Interoperability Technology Is A Fundamental Part Of GIS Products... Supporting Complex Data Transformation

DRAFT April 28, 2005 ESIP AQ Cluster, For GIS Networks to Work Either Everyone Uses Same Software, Data Formats, and Data Models... They Use Interoperability Procedures... Geoprocessing Models Can Transform Data Automatically... OR Format Conversion Schema Reorganization (ETL) Scale Projection Changes Generalization Merge GeoprocessingModels Interoperability Is Important... Enhancing Collaboration

DRAFT April 28, 2005 ESIP AQ Cluster, Geoprocessing On Servers GIS... Distributed Workflow & Process Models Distributing Spatial Analysis And Modeling NowFuture GIS Browser Desktop Data Sets

DRAFT April 28, 2005 ESIP AQ Cluster, Managing Multi Dimensional Geographic Data Sets And Simulation Modeling Data ModelingData Modeling Tools for ManipulationTools for Manipulation –Query –Change Analysis –Iterative Processing –Visualization –Animation –Charting With Particular Focus on Time FutureT1... Iterative/Recursive Modeling Simulation / Time Looping New Folder\ELNINO_Final.aviNew Folder\ELNINO_Final.avia New Folder\ELNINO_Final.avi

DRAFT April 28, 2005 ESIP AQ Cluster, GIS Will Maintain Distributed Geographic Knowledge Relationships Will be via “Messaging” (Sending/Receiving Web Services Messages) Geodatabases Will be Distributed and Federated

DRAFT April 28, 2005 ESIP AQ Cluster, Serving Globes Over the Web... Serving 3D Virtual Geography Globe Web Server

DRAFT April 28, 2005 ESIP AQ Cluster, Personal GIS Server... Users Will Share And Serve Their Knowledge Supporting Map Services Metadata Catalog (Searching & Harvesting) Download –Data –Models –Data Models Easy to Use Simple to Install Geodatabase Web Service GISDesktop PersonalServer Metadata Models Maps GeodataSets DataModels Will Allow Peer to Peer Collaboration

DRAFT April 28, 2005 ESIP AQ Cluster, T T T TUpdateMessages National State Local ReplicatedReplicated Periodically UpdatedPeriodically Updated History/ArchivingHistory/Archiving Geodatabases Will Support Distributed Data Management T = Transactions

22 Infusion Confusion Solutions: Putting Technology to Work Earth Science Data System Working Group on Technology Infusion Karen Moe, NASA/ESTO Rob Raskin, NASA/JPL

DRAFT April 28, 2005 ESIP AQ Cluster, One of four groups established by the REASoN CAN –Standards & Interfaces –Metrics Planning & Reporting –Reuse Frameworks –Technology Infusion Outgrowth of SEEDS –Strategic Evolution of ESE Data Systems –Explored ways to support NASA ES strategy More PI production processing Measurement-oriented systems REASoN = Research, Applications, and Education Solutions Network CAN = Cooperative Agreement Notice ESDSWG = Earth Science Data System Working Groups What is the Technology Infusion Working Group? SEEDS REASo N CAN ESE Strategic Plan Projects Data Life Cycle ESDSWG Standards Metrics Reuse Infusion New in 2005

DRAFT April 28, 2005 ESIP AQ Cluster, Effective Technology Infusion Enterprise Context Constrained budgets Broad data service provider community Pragmatic Infusion Approaches Information sharing Demonstration testbeds Emerging Technologies Technology investments Web and grid computing Linux clusters Organizational Goals Lower system costs Increase community participation Increase flexibility & responsiveness Internal Opportunities Drivers External Why is Technology Infusion Important? Drivers and Opportunities

DRAFT April 28, 2005 ESIP AQ Cluster, Why is Technology Infusion Important? Meeting ESE Goals Requires Tech Infusion Science and application needs –Faster & better models –Near-real-time data –Easier data fusion Science data system needs –Enable open distributed architecture for PI processing –Fill capability gaps in current systems –Support evolution New ResearchNew Applications New System Capabilities System Capability Vision Technology Infusion Technology Identification / Development Science & App Needs

DRAFT April 28, 2005 ESIP AQ Cluster, Capability Needs Technology Projections Technology Roadmaps Technology Development Technology Infusion Operational Systems Identified Gaps Solicitation Formulation Peer Review & Competitive Selection Capability Vision Technology Infusion is Part of a Larger System Evolution Process Think globally, act locally –How can we improve technology infusion across the community? –How can you successfully infuse technology in your own projects?

DRAFT April 28, 2005 ESIP AQ Cluster, What Capabilities are Needed? CapabilityBenefit Assisted information discovery Identify needed data quickly and easily Seamless information access Enable access to any data from anywhere Assisted knowledge building Provide research and operations assistance Interactive analysis environments Reduce research algorithm implementation from months to hours Super-scalable analysis portals Provide computing power and data storage on demand Interoperable information services Increase synergy within the ESE community through service chaining Integrated modeling frameworks Enable linked and ensemble models for improved predictive capability Responsive information logistics Ensure research priorities are met and enable new uses of ESE data Verifiable information quality Provide confidence in products and enable community data providers

DRAFT April 28, 2005 ESIP AQ Cluster, THREEDS - Topics Traditional Unidata Approach –Mainly meteorological data –Subscription system pushes data to user sites –Unidata Program Center provides data analysis tools for use on data at user sites THREDDS Enhancements –Broader menu of Earth system data –Local client access from remote servers –Less arcane, more general and accessible tools –Integration of data and analysis tools into educational modules and digital libraries THREEDS The THREDDS (Thematic Realtime Environmental Distributed Data Services) project is developing middleware to bridge the gap between data providers and data users. The goal is to simplify the discovery and use of scientific data and to allow scientific publications and educational materials to reference scientific data. The mission of THREDDS is for students, educators and researchers to publish, contribute, find, and interact with data relating to the Earth system in a convenient, effective, and integrated fashion. Just as the World Wide Web and digital-library technologies have simplified the process of publishing and accessing multimedia documents, THREDDS is building infrastructure needed for publishing and accessing scientific data in a similarly convenient fashion.

DRAFT April 28, 2005 ESIP AQ Cluster, THREDDS THematic Real-time Environmental Distributed Data Services Connecting people, documents and dataPeopleDocuments Data

DRAFT April 28, 2005 ESIP AQ Cluster, Summary Universities have used Unidata tools to acquire, analyze, and display real-time atmospheric data for nearly 20 years THREDDS – along with related client/server access and display technologies-- makes an even broader menu of Earth system data to a more diverse community of users THREDDS technologies enable the creation of compound educational modules and scientific publications with embedded pointers to datasets and tools.