IOOS Biological Data Services Enrollment/Publication Process Hassan Moustahfid (NOAA,US IOOS) Philip Goldstein (USGS, OBIS-USA) IOOS DMAC RAs Workshop.

Slides:



Advertisements
Similar presentations
GEOSS ADC Architecture Workshop Clearinghouse, Catalogues, Registries Doug Nebert U.S. Geological Survey February 5, 2008.
Advertisements

Data Management in Support of the Fish & Wildlife Program Summary.
Geospatial One-Stop A Federal Gateway to Federal, State & Local Geographic Data
Integrating NOAA’s Unified Access Framework in GEOSS: Making Earth Observation data easier to access and use Matt Austin NOAA Technology Planning and Integration.
FGDC & ISO: What is the Current Status and Considerations when Moving Forward? Viv Hutchison USGS Core Science Systems November 10, 2010 Salem, OR.
Caro-COOPS Data Management: Metadata. Cast-Net addresses the need for improved connectivity among coastal observing systems by creating a regional framework.
1 ISO – Metadata Next Generation International consensus being built on structured metadata within a broader Geomatics Standard under ISO Technical.
BIS TDWG Conference 28 October 2013, Florence Documenting data quality in a global network: the challenge for GBIF Éamonn Ó Tuama, Andrea Hahn, Markus.
NOAA Metadata Update Ted Habermann. NOAA EDMC Documentation Directive This Procedural Directive establishes 1) a metadata content standard (International.
System Design/Implementation and Support for Build 2 PDS Management Council Face-to-Face Mountain View, CA Nov 30 - Dec 1, 2011 Sean Hardman.
SERNEC Image/Metadata Database Goals and Components Steve Baskauf
Metadata (for the data users downstream) RFC GIS Workshop July 2007 NOAA/NESDIS/NGDC Documentation.
From Data Integration to Integrated Solutions Tom Shyka, GMRI Paul Currier, NH DES Water Quality Workshop MACOORA - NERACOOS -SECOORA IOOS, USGS, EPA,
II Course on GBIF Node Management Arusha, Tanzania 31 st October and 1 st November 2008 Tim ROBERTSON Systems Architect GBIF Secretariat Data Publishing.
MEDIN Data Guidelines. Data Guidelines Documents with tables and Excel versions of tables which are organised on a thematic basis which consider the actual.
Themes in OBIS-USA, for Discussion in Arctic ATN Workshop Philip Goldstein March 25, 2013 O CEAN B IOGEOGRAPHIC I NFORMATION S YSTEM.
Metadata Guides for Smarties Marine Metadata Initiative URL:
VOCABULARIES A data management presentation. Data management best practices Inventory of resources/datasets – Database level or series of datasets/collections.
Controlled Vocabularies (Term Lists). Controlled Vocabs Literally - A list of terms to choose from Aim is to promote the use of common vocabularies so.
Introduction to OBIS-USA Biological Data, Applications, & Relationships March 14, 2011.
Geospatial One-Stop FGDC and GOS: Working as One to Build the NSDI Rob Dollison Geospatial One-Stop Program Office.
Training course on biodiversity data publishing and fitness-for-use in the GBIF Network, 2011 edition How Darwin Core Archives have changed the landscape.
North American Profile: Partnership across borders. Sharon Shin, Metadata Coordinator, Federal Geographic Data Committee Raphael Sussman; Manager, Lands.
Sept 19,  Provides a common set of terminology and definitions  A framework for describing resources and processes  Enables computer based interoperability.
U.S. Department of the Interior U.S. Geological Survey Management of Oceanographic time-series data at the Woods Hole Coastal and Marine Science Center.
Metadata and Geographical Information Systems Adrian Moss KINDS project, Manchester Metropolitan University, UK
Standards and tools for publishing biodiversity data Yu-Huang Wang June 25, 2012.
Vers national spatial data infrastructure training program Value of Metadata Introduction to Metadata An overview of the value of metadata to.
Extensible Markup Language (XML) Extensible Markup Language (XML) is a simple, very flexible text format derived from SGML (ISO 8879).ISO 8879 XML is a.
FGDC and GOS Metadata: Foundations to Build the NSDI Sharon Shin FGDC Secretariat / Geospatial One-Stop.
GLOBAL BIODIVERSITY INFORMATION FACILITY Éamonn Ó Tuama Senior Programme Officer, IDA 21 June Metadata publishing with the IPT.
Creating documentation and metadata: Recording provenance and context Jeff Arnfield National Climatic Data Center Version a1.0 Review Date.
Definition of an Observation In general, an observation represents the measurement of some attribute, of some thing, at a particular time and place. Observations.
U.S. Integrated Ocean Observing System (IOOS ® ) IOOS ® Biological Observations Data Project A Multi-Agency Effort to Enable Access to Biological Observations.
Geospatial One-Stop FGDC and GOS: Working as One to Build the NSDI Sharon Shin Federal Geographic Data Committee Geospatial One-Stop Metadata Coordinator.
2008 EPA and Partners Metadata Training Program: 2008 CAP Project Geospatial Metadata: Introduction Module 1: Introduction & Overview of the FGDC CSDGM.
Canadian IDN Status Report Brian McLeod, Martine Rocheleau Cameron Wilson and Christine Therriault CCRS, Natural Resources Canada CEOS WGISS Plenary and.
An introduction to data exchange protocols in TDWG Renato De Giovanni TDWG 2008.
Laura Russell Programmer VertNet Buenos Aires (Argentina) 28 September 2011 Training course on biodiversity data publishing and.
Documenting UAF Data Ted Habermann NOAA/NESDIS/National Geophysical Data Center.
Digitization – Basics and Beyond workshop Interoperability of cultural and academic resources New services for digitized collections Muriel Foulonneau.
Hellenic Centre for Marine Research (HCMR) MedOBIS - Ocean Biogeographic Information System for the Eastern Mediterranean and Black Sea.
Metadata Training for SEFSC Science Staff Part Two.
Writing Metadata Working Towards Best Practices for SEFSC.
The New GBIF Data Portal Web Services and Tools Donald Hobern GBIF Deputy Director for Informatics October 2006.
Merging Metadata Standards: FGDC CSDGM and ISO and Sharon Shin Federal Geographic Data Committee Metadata Coordinator
The Research Data Archive at NCAR: A System Designed to Handle Diverse Datasets Bob Dattore and Steven Worley National Center for Atmospheric Research.
Improving Access to Biological Observations data Dr. Hassan Moustahfid NOAA US IOOS Program Office Silver Spring, MD NMFS FMAC F2F Brief
Data Services Task Team WGISS-22 meeting Annapolis, the US, September 12th 2006 Shinobu Kawahito, JAXA/RESTEC.
Laura Russell VertNet Meherzad Romer NatureServe Canada John Wieczorek
The Earth Information Exchange. Portal Structure Portal Functions/Capabilities Portal Content ESIP Portal and Geospatial One-Stop ESIP Portal and NOAA.
Open Access data at VLIZ Experience in retrieving data from EMODnet “Data ingestion, archiving, citation and DOI” June 26, 2014.
IPT + Darwin Core OBIS XML Schema OBIS Database Schema Explained Mike Flavell OBIS Data Manager OBIS Nodes Training Course, Oostende, Belgium, 6 May 2014.
OBIS IODE PO OBIS INCOIS OBIS- SEAMAP Separate files OBIS Nodes Data providers Separate files GBIFLifeWatchGEOSSEOL,…CBDFAOISA Fail-over mirrorGeo-load.
An Introduction to the MEDIN Discovery Metadata Standard MEDIN Workshop BGS, Edinburgh, June 2015.
An Introduction to the MEDIN Discovery Metadata Standard MEDIN Workshop NOC, Liverpool, Sept 2015.
OBIS Data Scenarios: Using Darwin Core to bring data into OBIS Philip Goldstein O CEAN B IOGEOGRAPHIC I NFORMATION S YSTEM May 5, 2014.
IOOS Biological Data Services Three Steps to Enrollment Philip Goldstein (University of Colorado, OBIS-USA) Hassan Moustahfid (NOAA US IOOS) May 28, 2014.
International OBIS: A Process for Enhancing Data Content using Ratified Darwin Core Philip Goldstein, Mike Flavell O CEAN B IOGEOGRAPHIC I NFORMATION S.
Geospatial metadata Prof. Wenwen Li School of Geographical Sciences and Urban Planning 5644 Coor Hall
3.2) Data sharing and dissemination Data Sharing between OBIS-SEAMAP, OBIS and GBIF.
GBIF Implementation Plan Highlights
Flanders Marine Institute (VLIZ)
Training course on biodiversity data publishing and fitness-for-use in the GBIF Network, 2011 edition How Darwin Core Archives have changed the landscape.
Active Data Management in Space 20m DG
GLOBAL BIODIVERSITY INFORMATION FACILITY
Data Management: Documentation & Metadata
Overview EMODnet Biology Portal Standards used Web services available
Robert Dattore and Steven Worley
Presentation transcript:

IOOS Biological Data Services Enrollment/Publication Process Hassan Moustahfid (NOAA,US IOOS) Philip Goldstein (USGS, OBIS-USA) IOOS DMAC RAs Workshop Sep

SERVICE REGISTRY Metadata Harvester DATA CATALOG DATA VIEWER US IOOS PORTAL Backend Data storage IOOS Regional Data Nodes Bio. Core Var. Data (ERDDAP or other accepted IOOS Service) Backend Data storage Federal Agencies (NOAA NMFS, NOS, BOEM, NAVY..etc) Bio. Core Var. Data ERDDAP or other Accepted IOOS service) OBIS-USA DATA.GOV Other data portals DATA USERS NODC Enrollment process Formatting to IOOS DMAC Standards. Biological Observations Data Architecture

O CEAN B IOGEOGRAPHIC I NFORMATION S YSTEM Address the Topic: What is Enrollment? … and answer in terms of the following … Results of Enrollment Flavors of Enrollment Enrollment Steps Enrollment Techniques and Tools Enroll data in the standard; publish data via services IOOS Core Biological Variables: Fish Species, Fish Abundance, Zooplankton Species, Zooplankton Abundance, Phytoplankton Species Applicable to any aquatic taxa Current and Future Introduction to this Presentation:

Dataset is on IOOS RNs web/database/webservices Dataset is on IOOS DMAC (Registry/Catalog/Viewer) Archive en route to NODC sync with OBIS-USA Metadata published and distributed Dataset included at Other Portal such as Data.gov Delivered Results of Enrollment – What is Accomplished When Enrollment is Complete? Enrollment Checklist What is complete when enrollment is complete … ✔ Data via IOOS RN and/or partner ✔ Referred by IOOS DMAC ✔ Complementary access via OBIS-USA and NODC ✔ NetCDF version of data ✔ Metadata Clearinghouse ✔ Data.gov

Value of Enrolling IOOS BDS Data: the Wealth of Data AssetDataset Richness, Documentation, Quality and Standards Established ValueDataset publicly accessible and usable; Application opportunities multiplied SecurityLife cycle from origin  application  archive has been anchored InvestmentEnhanced relationship of data originator with IOOS and partners; also enhance data origination practices? CommunityData originator’s voice accessible to future development of standards and applications

The Flavors are: 1.Assisted Enrollment 2.Self-Enrollment 3.Train-the-enroller O CEAN B IOGEOGRAPHIC I NFORMATION S YSTEM The “flavors” are in relation to … Data Originator / Data Holder Associated IOOS Regional Node Flavors of Enrollment

Assisted Enrollment IOOS RN assists data partner as opportunity or need presents itself O CEAN B IOGEOGRAPHIC I NFORMATION S YSTEM Self Enrollment Partner works independently to the extent they desire Self-Enrollment is not intended to be100%: IOOS stands by to help; IOOS always does a quality check Self Enrollment capability is influenced by: Knowledge/skill resources available Community practices, repeatbility Repeat enrollment by originator / RN Flavors of Enrollment (continued)

Service config.Metadata to ISO XML Generate Machine Metadata Capture metadata that can be queried or calculated from dataset contents. 4 O CEAN B IOGEOGRAPHIC I NFORMATION S YSTEM Map to IOOS BDS Create mapping document; identify questions at field- and dataset-level; review and resolve with data originator. 3 Gather Human Metadata Descriptive metadata for all purposes: citation, abstract, georef, taxon ID, sampling and calc methods. 2 Occurrence data Format Get data into taxon-location-time format; from data originators’ various original technologies. 1 Publish Metadata Sync NODCSync OBIS- USA Create ISO XML metadata following NGDC guidance. Use ERDDAP datasets.xml with captured metadata. Run Scripts to update Service Config files e.g. ERDDAP datasets.xml file ISO accessible to NGDC FGDC accessible by Clearinghouse and verified; FGDC to GCMD; FGDC to ISO (potential). Make new dataset (data and metadata) available to NODC; verify transfer and accession information. Make data available to OBIS-USA; verify transfer; How? Enrollment Process Steps 1 thru 9

Service config.Metadata to ISO XML Generate Machine Metadata Capture metadata that can be queried or calculated from dataset contents. 4 Map to IOOS BDS Create mapping document; identify questions at field- and dataset-level; review and resolve with data originator. 3 Gather Human Metadata Descriptive metadata for all purposes: citation, abstract, georef, taxon ID, sampling and calc methods. 2 Occurrence data Format Get data into taxon-location-time format; from data originators’ various original technologies. 1 Publish Metadata Sync NODCSync OBIS- USA Create ISO XML metadata following NGDC guidance. Use ERDDAP datasets.xml with captured metadata. Run Scripts to update Service Config files e.g. ERDDAP datasets.xml file ISO accessible to NGDC FGDC accessible by Clearinghouse and verified; FGDC to GCMD; FGDC to ISO (potential). Make new dataset (data and metadata) available to NODC; verify transfer and accession information. Make data available to OBIS-USA; verify transfer; Interaction: Highlighted steps indicate substantial involvement of data originator Here is where to balance cost and detail, and build enthusiasm Here is the excellent opportunity to build community Non-highlight steps are IOOS-internal Enrollment Process Steps Where main emphasis is content expertise

Map to IOOS BDS Create mapping document; identify questions at field- and dataset-level; review and resolve with data originator. Gather Human Metadata Descriptive metadata for all purposes: citation, abstract, georef, taxon ID, sampling and calc methods. Occurrence Data Format Get data into taxon-location-time format; from data originators’ various original technologies. Internal to IOOS Technology / Organization Involving Data Partner Enrollment Involving Data Partner, 1-2-3

Occurrence Data Format Get data into taxon-location-time format; from data originators’ various original technologies. 1 Data Sheet 10% Matrix KTE Relational Database 90% GIS KTE KTE “known to exist”: Occurrence data in this form are known to exist, though not yet encountered in IOOS BDS enrollment experience to date. Step 1: Occurrence Data; Various Original Formats

Gather Human Metadata Descriptive metadata for all purposes: citation, abstract, georef, taxon ID, sampling and calc methods. 2 Some metadata can be generated from data (referred to as machine metadata); this is an internal technical task. In contrast, Human metadata refers to essential descriptive details, vital to effective use of the data, that must be developed through human expertise and interaction. Human metadata is a key area to capture expertise from data originators; Data originator requried; Enroller assists in every way feasible. Step 2: Gather Human Metadata

Quality dialog with data originator is required Connected to the IOOS BDS data standards process Important Contents of Metadata: Dataset citation & institutional context Abstract Thematic keywords Georeference method, coordinate uncertainty Taxonomic identification details Sampling protocol / effort / conditions Derived data calculation methods Dataset history Related resources These features enhance discovery, attribution, and use in applications; also influence future data capture. Human Metadata Development Captures Valuable Knowledge

O CEAN B IOGEOGRAPHIC I NFORMATION S YSTEM Map to IOOS BDS Create mapping document; identify questions at field- and dataset-level; review and resolve with data originator. 3 Demonstrate an example of Enrollment Journal - CAGES Louisiana data Step 3: Map Data Contents to IOOS BDS Match partner’s data contents with IOOS BDS terms Document exceptions, questions, any kind of follow-up Use document sharing to communicate and resolve items between partner and enroller Assist in metadata capture and development

Make new dataset (data and metadata) available to NODC; verify transfer and accession information using the same process already in place between IOOS, RAs and NODC to archive observing data. Two important aspects of this coordination with NODC are: 1.Make sure that NODC has the same version, that is the same snapshot in the lifecycle of the data contents, that IOOS is serving 2.Where NODC needs metadata to accession an archive, and IOOS needs metadata to serve Biological Data Services, to the extent that these two uses of metadata are based on the same contents, IOOS and NODC can coordinate so as not to duplicate efforts. Sync NODC Make new dataset (data and metadata) available to NODC; verify transfer and accession information. Step 8: Sync with NODC

Step 9 – Sync with OBIS-USA Sync OBIS- USA Make data available to OBIS-USA; verify transfer; 9 OBIS-USA to carry the biogeographic component of dataset: less application-specific detail but discoverable in the national marine biogeographic dataset. OBIS-USA refers users to IOOS for live service- accessible detail and applications. Practical integration with OBIS-USA enabled by: 1.Common data and metadata content and format standards 2.Common enrollment practices.

IPT and Darwin Core Archive (DwC-A) O CEAN B IOGEOGRAPHIC I NFORMATION S YSTEM What is a Darwin Core Archive? Created by IPT, Administered by IPT, Made Accessible by IPT, Consumed by IPT Zipped together, DwC-A consists of three files: Biological Observation Records (columns-and- rows) Column List (xml) EML Metadata (xml) Good alignement with “20 Questions”

IPT and IOOS BDS Get IOOS BDS in GBIF RDF DwC/GBIF protocols. O CEAN B IOGEOGRAPHIC I NFORMATION S YSTEM IPT Production DwC/GBIF protocols. IPT Production seems a desirable step for widespread use and official versioning. IOOS BDS definitions in Xsd, RDF Definitions established xsd format created Get IOOS BDS Into IPT Sandbox DwC/GBIF protocols. Once in the sandbox, terminology is usable by IOOS / IPT Staying aligned with GBIF, global DwC, and IPT practices is a good idea throughout. GBIF/TDWG protocols TDWG Ratification not necessary; communities (such as marine, such as US) can operate with their own definitions. Ratification would serve if IOOS has global features (e.g., CF Integration) IODE Acceptance DwC/TDWG Ratification

Darwin Core Archive – Future Possibilities O CEAN B IOGEOGRAPHIC I NFORMATION S YSTEM Traditional DwC-A Possibilities: Biological Observation Records (columns-and- rows) Column List (xml) EML Metadata (xml) Good alignment with “20 Questions” approach Beyond EML (FGDC, ISO) Bundled NetCDF Format OAIS Archival Vocabulary-based variables with field-level metadata; physical and biological data for collecting event and biological observation. Web Service Integration

Service config.Metadata to ISO XML Generate Machine Metadata Capture metadata that can be queried or calculated from dataset contents. 4 O CEAN B IOGEOGRAPHIC I NFORMATION S YSTEM Map to IOOS BDS Create mapping document; identify questions at field- and dataset-level; review and resolve with data originator. 3 Gather Human Metadata Descriptive metadata for all purposes: citation, abstract, georef, taxon ID, sampling and calc methods. 2 Occurrence data Format Get data into taxon-location-time format; from data originators’ various original technologies. 1 Publish Metadata Sync NODCSync OBIS- USA Create ISO XML metadata following NGDC guidance. Use ERDDAP datasets.xml with captured metadata. Run Scripts to update Service Config files e.g. ERDDAP datasets.xml file ISO accessible to NGDC FGDC accessible by Clearinghouse and verified; FGDC to GCMD; FGDC to ISO (potential). Make new dataset (data and metadata) available to NODC; verify transfer and accession information. Make data available to OBIS-USA; verify transfer; How? Enrollment Process Steps 1 thru 9

Value of Enrolling IOOS BDS Data: the Wealth of Data AssetDataset Richness, Documentation, Quality and Standards Established ValueDataset publicly accessible and usable; Application opportunities multiplied SecurityLife cycle from origin  application  archive has been anchored InvestmentEnhanced relationship of data originator with IOOS and partners; also enhance data origination practices? CommunityData originator’s voice accessible to future development of standards and applications