CWIC/IDN synchronization Lingjun Kang Michael Morahan

Outline CWIC/IDN sync overview – Workflow – Module interaction Tests supporting synchronization Integration of sync/dataset testing/audit – Sync-test-sync model – Sync-audit-sync model Syn in IDN Future plan – Contact info synchronization

CWIC/IDN sync overview sync  test IDN mapping table IDN CSW/GCMD/OS 2. Harvest existing dataset entries (‘prod’ tag in IDN) 1. Harvest new dataset entries (tagged as ‘test’ in IDN) sync  test  audit  sync Tag datasets with ‘prod’ (passing testing) Tag datasets with ‘test’ (not passing dataset)

Tests supporting synchronization CWIC/IDN synchronization is achieved by tagging dataset entries in IDN based on testing results: – Dataset passing tests  Be published to ‘CWIC’ accessible list in IDN – Dataset failing tests  Be removed from ‘CWIC’ accessible list in IDN – Dataset returning 0 granule in tests  Be removed from ‘CWIC’ accessible list in IDN

Tests supporting synchronization Accessibility testing – Objective Verify if all granules under a dataset are searchable from data provider metadata inventory – Methods Query dataset against corresponding metadata inventory

Tests supporting synchronization Query and check granule number for each dataset – Testing each dataset with requests containing spatial/temporal parameters being harvested form IDN – Check granule number

Tests supporting synchronization Query dataset holding from data provider API (if available, e.g. NASA ECHO RESTFUL API) – Testing is performed on single HTTP GET transaction for all datasets – Less system workload is imposed in data provider inventory – Not skew the metrics of both CWIC and data provider inventory

sync/testing/audit integration Sync  Test  Sync – For new dataset entries from IDN Harvest dataset entries (dataset ID) and corresponding valids (spatial/temporal) Create testing suite (construct request based on valid/other request options) Run the testing suite Tag IDN based on testing results (automatic or manual)

sync/testing/audit integration Sync  Audit  Sync – For existing dataset entries in both IDN and CWIC (daily) Update dataset valids (if valid is changed in IDN, e.g. NRT dataset) Create testing suite (construct request based on valid/other request options) Run the testing suite Tag IDN based on testing results (automatic) – If fails/0 granule  tag with ‘test’ – If succeed  audit granule number

CWIC sync/testing/audit integration Support of module integration Sync moduleTest/audit module IDN Tag module Sync valids from IDN Read valids and test Update testing status Read testing status and tag Update tagging status

CWIC sync/testing/audit integration Testing summary example Testing status Tagging status: tag ‘test’ in IDN mapping table

CWIC sync/testing/audit integration Testing summary example Testing status Tagging status: tag ‘test’ in IDN mapping table

IDN Test (DEV) CSW server Process: CWIC Mapping Table Synchronization Benefits: tagging a record without modifying the DIF entry. Add Entry_ID, Data Provider ShortName, and either the Entry_Title or ECHO Dataset ID to the mapping table for the new entry.

IDN Production CSW server Process: CWIC Mapping Table Synchronization Update cwic_status tag to "value=prod” by CWIC or IDN.

IDN CWIC Portal (OpenSearch) Process: CWIC Mapping Table Synchronization Add Entry_ID, Data Provider ShortName, and either the Entry_Title or ECHO Dataset ID to the mapping table for the new entry. CWIC OpenSearch Portal links: Production: openSearch.jsp?Portal=cwic Test: openSearch.jsp?Portal=cwictest

Future plan Synchronization of dataset level info (TBD) – Contact info – Data center info (hierarchy structure of ownership) – Other?

Questions and Discussion