ESDIS METADATA ARCHITECTURE STUDY DISCUSSION Presented by Kevin Murphy and the Metadata Architecture Study Team Members 2013 ESIP Summer Meeting | Chapel Hill, NC

Outline Background MAS Phase I MAS Phase II Community Feedback Open Discussion 2 July 11, ESIP Summer Meeting | Chapel Hill, NC

BACKGROUND 3 July 11, ESIP Summer Meeting | Chapel Hill, NC

EOSDIS Mission The Earth Science Data and Information System (ESDIS) project provides access to data through the development and operation of the science systems of the Earth Observing System (EOS) Data and Information System (EOSDIS). July 11, ESIP Summer Meeting | Chapel Hill, NC 4

Metadata Architecture Study (MAS) ESDIS initiated a study of EOSDIS metadata capabilities/processes in the spring of 2012 to evaluate opportunities for improved data access and use, reduce efforts by data providers, and improve metadata integrity. Goals: Unify several disparate metadata formats (DIF, ECHO, EMS). Provide consistent results, performance, and access to users regardless of the mechanism through which they access EOSDIS metadata. Improve the consistency and quality of EOSDIS metadata holdings. Provide a platform capable of expanding EOSDIS metadata systems to incorporate services, tools, documents, etc. 5 July 11, ESIP Summer Meeting | Chapel Hill, NC

Ground Rules Metadata Harmony Study EOSDIS metadata are currently a widely used resource which must remain available with at least the same level of service during system definition, development and implementation. Enhancements to metadata systems must focus on improving user interactions with the system (End Users, International Users, Data Providers), sustaining engineering of the system, and the integrity of metadata. 6 July 11, ESIP Summer Meeting | Chapel Hill, NC

METADATA ARCHITECTURE STUDY (PHASE I) 7 July 11, ESIP Summer Meeting | Chapel Hill, NC

MAS I Research Study Objective: Identify opportunities for improved data access and use, reduce efforts by data providers, and improve metadata integrity. Of interest were the overlapping roles and interfaces between the GCMD and ECHO. The five-member study team included representatives from ESDIS, GCMD, ECHO, EMS, and the DAACs. Results - Eight recommendations in four categories: 8 Common Approaches to Metadata (2) Improved Communications (4) Keywords (1) Looking Forward (1) July 11, ESIP Summer Meeting | Chapel Hill, NC

MAS I Recommendations Move to a single, shared NASA Earth Science Metadata repository for submission of collection level metadata. ESDIS should define a consistent dataset identifier policy to be used across DAACs and central data systems (ECHO, GCMD, EMS). Consider a regular communication forum to address and clarify ESDIS requirements. Schedule periodic technical telecons between GCMD and the DAACs to provide insight into new or planned capabilities. GCMD should implement a mechanism to notify DAACs of DIF and SERF structure modifications. Both ECHO and GCMD should develop a DAAC-accessible mechanism for tracking implementation plans and status for requested changes. GCMD should roll out the new Keyword Management System and its RESTful APIs to the DAACs as soon as practical. ESDIS should continue to monitor emerging metadata needs and related technologies, and work with the community to define conventions for these new metadata. 9 July 11, ESIP Summer Meeting | Chapel Hill, NC

METADATA ARCHITECTURE STUDY (PHASE II) 10 July 11, ESIP Summer Meeting | Chapel Hill, NC

MAS II Architecture Development Objectives: Develop a Unified Metadata Model (UMM) that unifies the EOSDIS metadata assets into one cohesive model. Good position for future ISO compatibility. Create a Concept of Operations document that provides an overview of how the Unified Metadata Model (UMM) and Common Metadata Repository (CMR) will be used. 11 Unified Metadata Model (UMM) Common Metadata Repository (CMR) Metadata Harmony July 11, ESIP Summer Meeting | Chapel Hill, NC

UNIFIED METADATA MODEL (UMM) July 11, ESIP Summer Meeting | Chapel Hill, NC 12

Overview Unification Process What was in scope, what was not in scope Vocabulary UMM Core Vocabularies Extendable Vocabularies Knowledge Base System Documentation July 11, ESIP Summer Meeting | Chapel Hill, NC 13

Unification Process Initial focus is to unify the following models: Current conversions are “lossy”. By unifying these models ESDIS can latter focus on creating one mapping to ISO utilizing the upcoming NASA “flavor”. The ISO model was mapped where possible, and will be in scope for a future phase. 14 Metadata July 11, ESIP Summer Meeting | Chapel Hill, NC

Unifying the Models DIF ECHO Data Set Citation Personnel Instrument Platform Temporal Coverage Paleo-Temporal Coverage Spatial Coverage Location Data Resolution Project Quality Access Constraints Use Constraints Distribution Data Set Language Data Set Progress Related URL DIF Revision History CitationForExternalPublication ProcessingLevelId Price SpatialKeywords TemporalKeywords Temporal Contact ScienceKeywords Platform Instrument Sensor Campaigns TwoDCoordinateSystem OnlineAccessURL OnlineResource AssociatedDIFs Spatial ArchiveCenter AdditionalAttributes AssociatedBrowseImages SpatialInfo CollectionProgress RestrictionComments 15 July 11, ESIP Summer Meeting | Chapel Hill, NC EMS Not Shown

Unified Fields 16 July 11, ESIP Summer Meeting | Chapel Hill, NC

Vocabularies UMM Core Vocabularies Extendable Vocabularies July 11, ESIP Summer Meeting | Chapel Hill, NC 17

Benefits of Core Vocabularies Provides a classification system for metadata. Prevents zero-hit queries. Improves metadata accuracy and completeness. Improves the usability of earth science metadata. 18 July 11, ESIP Summer Meeting | Chapel Hill, NC

Core Vocabularies - Unique Keyword Identifiers Controlled vocabularies are referenced in the UMM using keyword identifiers (UUID’s) to SKOS concepts in the GCMD Keyword Management System Service. Updates to keywords referenced in metadata happen automatically since keywords are only “referenced” using these identifiers. Additions and/or changes to the vocabularies can be made through the existing governance process. 19 July 11, ESIP Summer Meeting | Chapel Hill, NC

UMM Core Vocabulary Types Science Keywords Platforms Instruments Sensors Projects Organizations Organization Types Locations Temporal Resolution Range Keywords Horizontal Resolution Range Keywords Vertical Resolution Range Keywords Chronostratigraphic Units Dataset Languages Related URL Types and Protocols ISO Topic Categories DataSet Progress DataSetLanguage Temporal Coverage Types Processing Level Identifiers Units for Resolutions Units for Geographic Coordinates 20 July 11, ESIP Summer Meeting | Chapel Hill, NC

Extendable Vocabularies AncillaryKeyword Class Provides a way to reference controlled vocabularies external to the CMR. This UMM class has a resource attribute associated with it, allowing the metadata author to reference an external RDF concept with its own namespace identified through a URI. Example: Wind Speed The example above links to a CF standard_name keyword provided by the Marine Metadata Interoperability Project (MMI). 21 July 11, ESIP Summer Meeting | Chapel Hill, NC

July 11, ESIP Summer Meeting | Chapel Hill, NC 22 Knowledge Based System (Many Interconnections)

UMM Documents (Finalized by Early August) UMM Overview Document High level UML diagrams of key parts of the model and descriptions of how metadata can be represented in the model. UMM Crosswalk Study Document Analysis of each model element, similarities, differences, and provides recommendations. UMM Crosswalk Spreadsheet Detailed mapping between DIF, ECHO, EMS. Maps UMM to ISO (where possible). UMM Schema created from the model UMM is not intended to be a format, but can be useful to describe in this way. Online Javadoc style navigation of the UMM Schema 23 July 11, ESIP Summer Meeting | Chapel Hill, NC

COMMON METADATA REPOSITORY (CMR) July 11, ESIP Summer Meeting | Chapel Hill, NC 24

2013 ESIP Summer Meeting | Chapel Hill, NC URS GIBS EMS EOSDIS Discovery Tools DAACs Near Real Time (LANCE) ??? CMR July 11,

2013 ESIP Summer Meeting | Chapel Hill, NCJuly 11,

2013 ESIP Summer Meeting | Chapel Hill, NCJuly 11,

Metadata Concept Support 2013 ESIP Summer Meeting | Chapel Hill, NCJuly 11,

2013 ESIP Summer Meeting | Chapel Hill, NCJuly 11, Extensibility

2013 ESIP Summer Meeting | Chapel Hill, NC ECHO 10GCMD DIFEMS ISO19115 UMM Ingest Adapter VisualizationDocumentationServicesParameters July 11,

Improving Metadata Quality with Assessment and Scoring 2013 ESIP Summer Meeting | Chapel Hill, NCJuly 11,

2013 ESIP Summer Meeting | Chapel Hill, NCJuly 11,

2013 ESIP Summer Meeting | Chapel Hill, NCJuly 11,

How Will This Affect Me? July 11, ESIP Summer Meeting | Chapel Hill, NC 34

Metadata Providers You won’t have to reconcile multiple systems Most systems will continue to work ECHO FTP ingest going away Current formats will continue to function You will need to be engaged in the adaptation process You will need to support controlled vocabularies You will have a voice through the Technical Committee July 11, ESIP Summer Meeting | Chapel Hill, NC 35 Streamlined Processes, More Engagement

Client Developers 36 More Metadata, Better Quality Richer, higher quality metadata will be available Existing APIs will be supported Standards-compliant APIs will be available Metadata will be available in an ISO19115 compliant format July 11, ESIP Summer Meeting | Chapel Hill, NC

End Users 37 More Metadata, Better Quality Richer, higher quality metadata will be available Unified view of EOSDIS holdings Easier access through a variety of clients, data casting, etc Metadata will be available in an ISO19115 compliant format July 11, ESIP Summer Meeting | Chapel Hill, NC

COMMUNITY FEEDBACK 38 July 11, ESIP Summer Meeting | Chapel Hill, NC

Early Reviewers NASA Standards Interest Group (SIG) Review Stakeholder Review/Interviews 39 Review Groups July 11, ESIP Summer Meeting | Chapel Hill, NC April 2013May – June 2013 June – July 2013

Community Feedback Process Reviewers submitted feedback and received comments using the EOSDIS Issue Tracking Software – JIRA 334 “Issues” created: 206 Resolved/Closed, 68 for Future Study 40 July 11, ESIP Summer Meeting | Chapel Hill, NC

Summary of Feedback Received Provide consistent product identifiers (i.e. UUID’s) Address ISO and NASA flavor of ISO Include more required fields Provide common values for project, platform, instrument Offer more controlled vocabulary…such as processing level Contain more structured fields...such as resolution units Provide types and descriptions for any URL Include provenance tracking 41 July 11, ESIP Summer Meeting | Chapel Hill, NC

Next Steps 42 July 11, ESIP Summer Meeting | Chapel Hill, NC Wrap-Up/Document Delivery (Early August) 1 Requirements, Prototyping, and Design 2 Implementation and Testing 3 Data Cleanup/Transition 4

MAS II Team Lead: Kevin Murphy ECHO Representatives Katie Baynes John Farley Dan Pilone EMS Representative Lalit Wanchoo GCMD Representatives Thomas Cherry Tom Northcutt Scott Ritz 43 July 11, ESIP Summer Meeting | Chapel Hill, NC

OPEN DISCUSSION 44 July 11, ESIP Summer Meeting | Chapel Hill, NC