Data Quality for Long-Term Datasets

Slides:

Advertisements

Similar presentations

GEOSS AIP Phase 2 Kickoff Workshop September, Boulder Colorado, USA AP ISO 1.0 Jürgen Walther Office of the Interministerial Committee for Geo Information.

Advertisements

Evolution of Metadata Standards: New Features in ISO Ted Habermann NOAA National Data Centers September, 2008

SSA Query Interface M. Dolensky, ESO Data Access Layer Working Group Interoperability Workshop, Pune, India 27-Sep-2004.

The OAIS experience at the British Library Deborah Woodyard Digital Preservation Coordinator ERPANET OAIS Training Seminar, Nov 2002.

TIPR: Repository Exchange Package Use Cases and Best Practices Joseph Pawletko and Priscilla Caplan IS&T Archiving 2011.

An Introduction June 17, 2013 Open Archival Information System (OAIS)

Merging Metadata Standards: FGDC CSDGM and ISO Sharon Shin Federal Geographic Data Committee Metadata Coordinator

ISO & OAI-PMH By Neal Harmeyer, Amy Hatfield, and Brandon Beatty PURDUE UNIVERSITY RESEARCH REPOSITORY.

1 ISO – Metadata Next Generation International consensus being built on structured metadata within a broader Geomatics Standard under ISO Technical.

PAWN: A Novel Ingestion Workflow Technology for Digital Preservation

PAWN: A Novel Ingestion Workflow Technology for Digital Preservation Mike Smorul, Joseph JaJa, Yang Wang, and Fritz McCall.

Introduction to Geospatial Metadata – ISO 191** Metadata National Coastal Data Development Center A division of the National Oceanographic Data Center.

Evolution of Metadata Standards: New Features in ISO Ted Habermann NOAA National Data Centers December, 2007

Oy Metadata Content j of Metadata. Discovery Access Understanding Levels of Metadata joy of Metadata Metadata Standards Why standards Which standards.

The Data Attribution Abdul Saboor PhD Research Student Model Base Development and Software Quality Assurance Research Group Freie.

US GPO AIP Independence Test CS 496A – Senior Design Team members: Antonio Castillo, Johnny Ng, Aram Weintraub, Tin-Shuk Wong Faculty advisor: Dr. Russ.

Interoperable Documentation Ted Habermann, NOAA/NESDIS/NGDC NCAR Earth Observing Laboratory, June 2010 Links: GEO-IDE Wiki:

1 Next Generation of Operational Earth Observations From the National Polar-Orbiting Operational Environmental Satellite System (NPOESS): Program Overview.

UWG 2013 Meeting PO.DAAC Web Services Demo. What are PO.DAAC Web Services?

Documenting Data Quality Ted Habermann, NOAA/NESDIS/NGDC Documentation: It’s not just discovery... 50% change in global average Why? i checked my 2002.

Page 1 LAITS Laboratory for Advanced Information Technology and Standards ISO & Status Liping Di Laboratory for Advanced Information Technology.

OAIS Rathachai Chawuthai Information Management CSIM / AIT Issued document 1.0.

Archival Information Packages for NASA HDF-EOS Data R. Duerr, Kent Yang, Azhar Sikander.

Metadata Implementation Ted Habermann NOAA National Geophysical Data Center Documentation: It’s not just discovery... 50% change in global average Why?

Creating Archive Information Packages for Data Sets: Early Experiments with Digital Library Standards Ruth Duerr, NSIDC MiQun Yang, THG Azhar Sikander,

Implementor’s Panel: BL’s eJournal Archiving solution using METS, MODS and PREMIS Markus Enders, British Library DC2008, Berlin.

The FCLA Digital Archive Joint Meeting of CSUL Committees, 2005.

Documenting Data Quality Ted Habermann, NOAA National Geophysical Data Center.

Workshop Resources.

GeoMAPP: Using Metadata to Help Preserve Geospatial Content Matt Peters, Utah’s Automated Geographic Reference Center Glen McAninch, Kentucky Department.

Metadata for Data Understandability Ted Habermann NOAA National Geophysical Data Center AGU, Spring 2008.

JPSS Common Ground System Document Versioning - Update Milt Panas May 10, 2011 JPSS CGS Form J /22/2010.

Transitioning from FGDC CSDGM Metadata to ISO 191** Metadata

Metadata – use data discovery e.g. a library catalog data assessment determine the fitness-for-purpose of a data set data retrieval e.g., format.

Meteorological Assimilation Data Ingest System (MADIS) and ISO Data Quality Ted Habermann NOAA National Data Centers MADIS observations on April 29, 2004.

OAIS Rathachai Chawuthai Information Management CSIM / AIT Issued document 1.0.

Documenting UAF Data Ted Habermann NOAA/NESDIS/National Geophysical Data Center.

C. Huc/CNES, D. Boucon/CNES-SILOGIC Producer-Archive Interface Specification.

Funded by: © AHDS Preservation in Institutional Repositories Preliminary conclusions of the SHERPA DP project Gareth Knight Digital Preservation Officer.

Creating Good Documentation NOAA National Geophysical Data Center

TSS Database Inventory. CIRA has… Received and imported the 2002 and 2018 modeling data Decided to initially store only IMPROVE site-specific data Decided.

ECS Metadata Considerations for Preservation SiriJodha S. Khalsa National Snow and Ice Data Center.

Preservation metadata and the Cedars project Michael Day UKOLN: UK Office for Library and Information Networking University of Bath

Merging Metadata Standards: FGDC CSDGM and ISO and Sharon Shin Federal Geographic Data Committee Metadata Coordinator

Standard Metadata in Scientific Data Formats September 19, 2007 Flash at:

Getting Familiar with Metadata Laurie Porth Rocky Mountain Research Station Audience: Scientists/researchers who have heard of metadata and now need to.

A look to the past for the future- The North American Profile Sharon Shin Metadata Coordinator Federal Geographic Data Committee.

June 21, 2011EDMC Workshop in Silver Spring, MDDan Kowal Submission Agreements: The role they play in supporting the relationship between the Data Producer/Provider.

Lifecycle Metadata for Digital Objects November 15, 2004 Preservation Metadata.

The OAIS Reference Model and Trustworthy Repositories Josh Lubell Manufacturing Engineering Laboratory NIST

The OAIS model SEEDS meeting May 5 th, 2015, Lausanne Bojana Tasic.

CLASS Metadata and Remote Sensing Extensions CLASS Data Provider’s Conference September 2005 Anna Milan, Ted.Habermann,

Physical Oceanography Distributed Active Archive Center THUANG June 9-13, 20089th GHRSST-PP Science Team Meeting GHRSST GDAC and EOSDIS PO.DAAC.

1 CAA 2009 Cross Cal 9, Jesus College, Cambridge, UK, March 2009 Caveats, Versions, Quality and Documentation Specification Chris Perry.

OAIS (archive) Producer Management Consumer. Representation Information Data Object Information Object Interpreted using its Yields.

May 2011DLM Forum, Budapest1 The First OAIS-compliant Ingest of Digital Records Zoltán Lux The National Archives of Hungary web:

OAIS (archive) OAIS (archive) Producer Management Consumer.

R2R ↔ NODC Steve Rutz NODC Observing Systems Team Leader May 12, 2011 Presented by L. Pikula, IODE OceanTeacher Course Data Management for Information.

Geospatial metadata Prof. Wenwen Li School of Geographical Sciences and Urban Planning 5644 Coor Hall

Data Management and Archival Storage Bojana Tasić FORS SEEDS Workshop I Belgrade, October.

DP Knowhow: Open Archival Information Systems (OAIS) in ISO APA/C-DAC International Conference on Digital Preservation and the Development of Trusted.

Preservation Planning

Metadata Issues in Long-term Management of Data and Metadata

Ingest and Dissemination with DAITSS

OAIS Producer (archive) Consumer Management

ACRIN Imaging Core Laboratory

Documenting Data Quality Ted Habermann, NOAA/NESDIS/NGDC

Application of Dublin Core and XML/RDF standards in the KIKERES

Open Archival Information System

Robin Dale RLG OAIS Functionality Robin Dale RLG

Presentation transcript:

Data Quality for Long-Term Datasets

NESDIS Metadata Station History Satellite Granule NBII & Other Extensions Network Management & Health Remote Sensing FGDC Classic ISO 191*

NESDIS Metadata Station History Satellite Granule NBII & Other Extensions Network Management & Health FGDC Classic ISO 191* Remote Sensing Static Dynamic

ISO Quality Scope DQ_Scope + level: MD_ScopeCode + extent [0..1] : EX_Extent + levelDescription [0..*] : MD_ScopeDescription MD_ScopeCode + attribute + attributeType + collectionHardware + collectionSession + dataset + series + nonGeographicDataset + dimensionGroup + feature + featureType + propertyType + fieldSession + software + service + model + tile MD_ScopeDescription + attributes : Set + features : Set + featureInstances : Set + attributeInstances : Set + dataset : CharacterString + other : CharacterString EX_Extent describes the spatial and temporal extent of the quality section.

OAIS Principles: Ingest Functions Quality Assurance Receive Submission Generate AIP Generate Descriptive Information Co-ordinate Updates ADMINISTRATIONADMINISTRATION Producer SIP Receipt confirmation Resubmit request SIP QA Results Data Formatting and Doc Stds. SIP, AIP for AuditAudit Report Data Management Archival Storage SIP AIP Descriptive Info Descriptive Info AIP Database Update Request Report Request Report Storage Request, AIP Storage Confirmation (Updated) SIP Quality Assurance Receive Submission

Granule Metadata - Present

Rich Inventory

Aerosol Algorithm Change

Metadata Hi Ted, We did some digging and this is the result. This conclusion is the most pertinent info we could find from logs or archives. Here it is: i checked my archives, and here is what i found out: it appears that the current 3rd generation aerosol algorithm was implemented into operations around Oct-Nov 2002 time frame. cannot say more precisely, as all correspondence i am looking at, talks about this indirectly. (maybe it's what Steve refers to as the Phase II aerosol-SST algorithm.) At the same time, Steve had implemented quite a few other changes fixing data bugs and formats: view angle problem in AEROBS, increased digitization in all channel's reflectances and AODs, etc. The jump in AOD1 is deemed due to introducing 3rd generation algorithm, which replaced the 2nd generation. The new numbers (~0.08) look more realistic than the previous ones (~0.05 or so). The changes seen in the data is close to the expected effect of this change. the 3rd gen alg takes into account the exact spectral response of N16 AVHRR, whereas the 2nd gen was using a generic set of LUTs for all AVHRRs ("one size fits all"). hopefully this settles the issue..