Workshop Resources

Transitioning to ISO Ted Habermann, NOAA/NESDIS/NGDC Documentation: It’s not just discovery... 50% change in global average Why? i checked my archives, and here is what i found out: it appears that the current 3rd generation algorithm was implemented into operations around Oct-Nov 2002 time frame. cannot say more precisely, as all correspondence i am looking at, talks about this indirectly. (maybe it's what's referred to as the Phase II algorithm.) At the same time, we had implemented quite a few other changes fixing data bugs and formats: view angle problem, increased digitization in all channel's reflectances and AODs, etc. The jump is deemed due to introducing 3rd generation algorithm, which replaced the 2nd generation. The new numbers (~0.08) look more realistic than the previous ones (~0.05 or so). The changes seen in the data is close to the expected effect of this change. The 3rd gen alg takes into account the exact spectral response, whereas the 2nd gen is generic ("one size fits all"). hopefully this settles the issue..

Why Documentation?

Metadata Types and Sharing Discovery Use / Mashup Understanding Discovery Portal Community Metadata Collections User More documentation is required for understanding data than discovering or using it.

Designated Communities - Users Data preservation is communicating with the future

Custom Documentation Custom Documentation FGDC Mandatory Identification Metadata FGDC Mandatory Identification Metadata FGDC Discovery Distributor FGDC Discovery Distributor FGDC Complete Entity and Attribute Data Quality Lineage Spatial Organization Spatial Reference FGDC Complete Entity and Attribute Data Quality Lineage Spatial Organization Spatial Reference FGDC RSE Identifier Documentation Mission Platform Instrument Aggregation Algorithm Processing FGDC RSE Identifier Documentation Mission Platform Instrument Aggregation Algorithm Processing ISO Code Lists Many Identifiers OnLine Resources Responsible Parties Data Quality Reports Spatial/Temporal Extents Metadata Levels / Scope ISO Code Lists Many Identifiers OnLine Resources Responsible Parties Data Quality Reports Spatial/Temporal Extents Metadata Levels / Scope ISO Quality Coverage Usability Platform Instrument Processing Algorithm ISO Quality Coverage Usability Platform Instrument Processing Algorithm FGDC to ISO International Standard Unified Modeling Language eXtensible Markup Language FGDC to ISO International Standard Unified Modeling Language eXtensible Markup Language Basic Content Extended Content ISO Services ISO Feature Types ISO Services ISO Feature Types Evolution: Documentation Standards FGDC BIO Taxonomy Geographic Extent FGDC BIO Taxonomy Geographic Extent

FGDC Complete Entity and Attribute Data Quality Lineage Spatial Organization Spatial Reference FGDC Complete Entity and Attribute Data Quality Lineage Spatial Organization Spatial Reference FGDC to ISO International Standard Unified Modeling Language eXtensible Markup Language FGDC to ISO International Standard Unified Modeling Language eXtensible Markup Language Evolution: Documentation Standards A small step in content but a huge step in tools 1. UML - Unified Modeling Language 2. XML - eXtensible Markup Language 3. Wiki - Community

Technology Adoption Moore described the “chasm” in the adoption life cycle. He proposes that many new technologies do not make it across the chasm between visionaries and early adapters. They fall into the chasm. The technology S-curve with the chasm might look like: TIME

Technological Disruption Era of Ferment Selection Tipping Point Technology Adoption The rate of progress increases because community energy is focused. Developments are supportive and cumulative

What Does a Tipping Point Look Like? GOES-R: NOAA and NASA AMS Metadata Policy Working Group Recommendations: The AMS should adopt ISO and SensorML as the metadata standards for the Nationwide Network of Networks. In coordination with other organizations within the broader atmospheric and oceanic sciences community, the AMS should provide opportunities for its members to become better educated on the nomenclature, structure, and implementation of metadata and the standards recommended below. The AMS should establish a permanent committee to serve as a body of experts … to provide advice to organizations working to implement a Nationwide Network of Networks and to coordinate with the Federal Committee for Integrated Observing Systems (CIOS). AMS Metadata Policy Working Group Recommendations: The AMS should adopt ISO and SensorML as the metadata standards for the Nationwide Network of Networks. In coordination with other organizations within the broader atmospheric and oceanic sciences community, the AMS should provide opportunities for its members to become better educated on the nomenclature, structure, and implementation of metadata and the standards recommended below. The AMS should establish a permanent committee to serve as a body of experts … to provide advice to organizations working to implement a Nationwide Network of Networks and to coordinate with the Federal Committee for Integrated Observing Systems (CIOS). Federal Coordinator for Meteorology Metadata Joint Action Committee The Joint Action Group recommends that the Federal departments and agencies adopt ISO and SensorML as weather observation metadata standards and participate in the process of implementing those standards across the national weather enterprise. Federal Coordinator for Meteorology Metadata Joint Action Committee The Joint Action Group recommends that the Federal departments and agencies adopt ISO and SensorML as weather observation metadata standards and participate in the process of implementing those standards across the national weather enterprise. Chapter VI - Other Post Award Requirements and Considerations 4. Dissemination and Sharing of Research Results b. Investigators are expected to share with other researchers, at no more than incremental cost and within a reasonable time, the primary data, samples, physical collections and other supporting materials created or gathered in the course of work under NSF grants. Grantees are expected to encourage and facilitate such sharing.

Shared Needs The groups that created these documents are leading the transition to the ISO Standards and recognizing the need for guidance and tools. Do they each head off in their own direction… Or come together to communicate and collaborate? Do they each head off in their own direction… Or come together to communicate and collaborate? USGS NASA NSF DOE

A Documentation Consortium DOC NSF NASA USGS DOE A consortium is an association of two or more organizations to participate in a common activity or pooling their resources for achieving a common goal: to enhance the exchange, understanding and use of Earth-systems data across disciplinary and international boundaries. Initially to coordinate and facilitate the transition to effective use of ISO Documentation Standards. A consortium is an association of two or more organizations to participate in a common activity or pooling their resources for achieving a common goal: to enhance the exchange, understanding and use of Earth-systems data across disciplinary and international boundaries. Initially to coordinate and facilitate the transition to effective use of ISO Documentation Standards. Standards Training Tool Adoption & Development

The ISO Standards are broad

ISO Standards

MD_ or MI_Metadata MD_ReferenceSystem MD_DataQuality MD_SpatialRepresentation MD_DataIdentification MD_MaintenanceInformation MD_Distribution MD_ApplicationSchemaInformation MD_PortrayalCatalogueReference MD_Constraints MD_MetadataExtensionInformation MD_ContentInformation ISO Metadata Objects MD_Usage MD_Keywords MD_AggregateInformation MD_BrowseGraphic MD_Format MI_AcquisitionInformation EX_Extent

Addresses many important needs Do you need different documentation for different parts of your data? Do you need different documentation for different temporal and spatial subsets? Do you have datasets with multiple sources? Do you need to reference On-Line Resources? Do you need to describe a series of related granules? Do you need to describe many kinds of aggregations? Does data quality vary within the dataset? Do you need to track processing for multiple data sources? Do you need to track compliance with standards? Do you need to use spatial features to describe quality, like grids of quality flags? Do you need to explain why you did things to the data? Do you have datasets in multiple geographic locations? Do you need to describe instruments used to make observations? Do you need to unambiguously identify things using your own namespace? Do you want to manage metadata using a relational or XML database? Do you want to serve metadata using a REST web service? Do you need to identify people in different roles? Do you need to keep track of user problems? Do you need to explain why you did things to the data? Do you need to track requirements and plans? Do you need to share data with international and national partners? Do you need to describe data formats and structures? Do you need to track data transformations and processing? What might the entity and attributes look like for station level metadata?

Documentation Repository ISO 19115, , and extensions Documentation Repository ISO 19115, , and extensions THREDDS netCDF DIF, FGDC, Data.Gov DIF, FGDC, Data.Gov SensorML WCS, WMS, WFS, SOS ISO KML Multiple Dialects and Connections

The ISO Standards are broad What Goes Where?

Spiral 1: Initial Content Spiral 1: Initial Content Metadata Content Independent of standard Metadata Content Independent of standard Standard Guidance / Implementation Standard Guidance / Implementation Check Back With Users Check Back With Users Spiral 2-N: Scientific Questions New Requirements New Use Cases Check Back With Data Collectors/ Providers Check Back With Data Collectors/ Providers Spiral Development / Training

Identification Id Title Abstract Resource Date Topic Category Theme Keyword Metadata Contact Science Contact Identification Id Title Abstract Resource Date Topic Category Theme Keyword Metadata Contact Science Contact Extent Geospatial Bounding Box Temporal Start/End Vertical Min/Max Place Keywords Extent Geospatial Bounding Box Temporal Start/End Vertical Min/Max Place Keywords Distribution Distributor Contact Online Resource Distribution Format Data Center Keywords Browse Graphic Distribution Distributor Contact Online Resource Distribution Format Data Center Keywords Browse Graphic Text Searches Purpose Extent Description Lineage Statement Project Keywords Text Searches Purpose Extent Description Lineage Statement Project Keywords Quality/Lineage Sources Process Steps Quality Reports / Coverages Quality/Lineage Sources Process Steps Quality Reports / Coverages Acquisition Information Instrument Platform Instrument Keywords Platform Keywords Acquisition Information Instrument Platform Instrument Keywords Platform Keywords Content Information Attribute Type Attribute Names Attribute Definitions Attribute Units Content Information Attribute Type Attribute Names Attribute Definitions Attribute Units Discovery Understanding Connection OnlineResource: Linkage (URL) Name Description Function Connection OnlineResource: Linkage (URL) Name Description Function Potential Spirals

SpiralNone0-33%34-66%67-99%All Identification Connection Extent Distribution Description Content Lineage Acquisition Information Metadata Evaluation for Dataset X Date: 2010/10/24 Spirals and Rubrics

Initial Rubric for FGDC > ISO Translation

Identifying Bounding Extents

Connections are Critical

Rubric Reflects the Improvements

Migrating Standards Upstream Science Question Data Collection ProcessingDistribution ArchiveDiscovery Analysis # Users Disciplinary Experts Non-Experts STANDARDSSTANDARDS REALTIMEREALTIME

Documentation Repository ISO 19115, , and extensions Documentation Repository ISO 19115, , and extensions THREDDS netCDF DIF, FGDC, Data.Gov DIF, FGDC, Data.Gov SensorML WCS, WMS, WFS, SOS ISO KML Multiple Dialects and Connections

Multiple Dialects: THREDDS Data Server THREDDS Catalog THREDDS Catalog file1.nc file2.nc file3.nc file4.nc file1.nc file.nc file2.nc file.nc THREDDS Catalog THREDDS Catalog THREDDS Catalog THREDDS Catalog Extract Data OPeNDAP WMS WCS

Multiple Dialects: Web Accessible Folders

Documentation in Three Dialects Documentation Repository Documentation Repository Spiral Tracker NcML

Documentation Dialects in THREDDS

File System Persistence vs. Transport - OPeNDAP File System Client DAP

File System Rich Inventory Relational Databases Client XML Persistence vs. Transport - Documentation

Database Objects and References Citation ID Title Date Friend_ID Location_ID Citation ID Title Date Friend_ID Location_ID Person ID Name Person ID Name OnlineResource ID Name URL OnlineResource ID Name URL

XML Attributes: Objects and References ISO XML consists of tags, elements (with or without content), and attributes. An attribute is a name/value pair that exists within a start-tag or empty-element tag. Attributes provide additional information about an element which is not part of the data. Attribute values must contain either single or double quotes. This example shows a step element with one attribute, number with a value of “3”: Connect A to B. Many of the XML attributes used in the ISO Standards fall into two groups: identifiers and references: Identifiers: id and uuid References: uuidref and xlink:href Objects that start with upper case letters have identifiers (id and uuid) Roles that start with lower case letters have references (uuidref and xlink:href) object: CI_ResponsibleParty id="JaneDoe" object: CI_ResponsibleParty id="JohnDoe" role: friend xlink:href=#JohnDoerole: friend xlink:href=#JaneDoe

Database and XML Keys Citation ID Title Date Friend_ID Location_ID Citation ID Title Date Friend_ID Location_ID Person ID Name Person ID Name OnlineResource ID Name URL OnlineResource ID Name URL XML …

Positive deviance says that if you want to create change, you must scale it down to the lowest level of granularity and look for people within the social system who are already manifesting the desired future state. Leadership Model: Positive Deviance Take only the arrows that are already pointing toward the way you want to go, and ignore the others. Identify and differentiate those people who are headed in the right direction. Give them visibility and resources. Bring them together. Aggregate them. Barbara Waugh

GEO-IDE Wiki: Home page: ISO Pages: ISO WAF with Rubric Links: Spirals: Workbooks: ftp://ftp.ncddc.noaa.gov/pub/Metadata/wkbk ISO Standards: Links

Questions? Questions / Comments / Suggestions: