PDSMC November 29-30, PDS Imaging Node PDS4 Migration Lisa Gaddis (USGS) Sue LaVoie (JPL) November 30, 2012 PDS Management Council Meeting UCLA

PDSMC November 29-30, Migrate only past missions & newly awarded missions (InSight) –Justification: for active missions, community analysis is ongoing with existing software & tools –Prioritize based on need for class development/follow-on missions and popularity –Retain PDS3 labels & metadata within PDS4 structure Migrate mission datasets requiring no (or simple) data manipulation –Retain copy of any data requiring manipulation Migrate metadata only of mission datasets requiring data manipulation –Justification: complex data migration is too risky –Retain PDS3 data format & label –Create PDS4 labels and metadata to enable complex data product searches across all holdings –Decisions/Action required by MC & Data Modelers Policy allowing waiving of data migration for complex legacy data sets Dictionary terms/keywords enabling description of legacy formats Modify IMG systems to support both PDS3 & PDS4 data sets & formats PDS4 Migration Strategy

PDSMC November 29-30, Process –Identify data set for migration –Create/update Discipline Classes, provide input to PDS-wide Classes –Map PDS3 to PDS4 labels and ancillary files/data –Create schema –Modify existing software tools & scripts to automatically create labels, collection and bundles –Generate Sample Bundle(s) –Verify –Iterate –Informal IMG internal review & testing –Load Bundle into IMG Registry –Identify next data set –….. –Repeat iterative process until Data Model and Classes are “final”, refining all along the way. Then perform the full data set migration on priority data sets PDS4 Migration Approach

PDSMC November 29-30, PDS4 Migration - Top Priority Data Sets

PDSMC November 29-30, PDS4 Migration – Waive Data Format Change

PDSMC November 29-30, Clementine 750nm Basemap (Chris) –Sample Bundle validated, reviewed by EN, updated and redelivered for ingestion Folders in bundle: data (6 map projected single band images & ISIS detached labels), document (.txt,.doc,.htm,.pdf versions of “volinfo” suite), miscellaneous (delivery & transfer manifests), xml_schema –Cartography Class - Incorporating Federal Geographic Data Committee (FDGC) model. Still needs fine tuning –Tools used: oXygen XML Editor, IMG Generate Tool, EN Validate Tool, ad hoc scripts Mars Pathfinder (MPF) IMP EDR (Elizabeth) –Sample Bundle validated and delivered to EN for review and feedback Folders in bundle: data (200 files), browse (browse for each data file), context (instrument, instrumet host, investigation, resources, etc.), document (EDR SIS), geometry (gazetteer), xml_schema –Imaging Instrument Parameter Class – in good shape for this data set & Clementine data set –Tools used: oXygen XML Editor, IMG Generate Tool, EN Validate Tool, ad hoc scripts Migration Progress

PDSMC November 29-30, Clementine UVVIS 5-band Mosaic (Chris) –Multispectral data set –Preliminary Bundle layout defined Mars Pathfinder (MPF) RVRCAM (Elizabeth) –Tables and Engineering data sets Phoenix OM, RAC, SSI EDR (Amy) –Precursor for InSight (launch 2016; Geo is lead node) Landed mission products and metadata Phoenix, MER, MSL heritage Work closely with Geo throughout –Class work: Camera Model, Landed Geometry, Mars InSitu, likely more Migration – Next Up

PDSMC November 29-30, Accomplishments –Installed all PDS2010 software on IMG operational machine, including Registry, Registry UI, Harvest, and Validate Used Validate tool on test bundles prior to delivery to EN for review –Continued development of IMG Generate Tool (label generation) to support ongoing data migration activities –Created internal IMG Wiki to track PDS4 migration details, progress, lessons learned, open items, versions of dictionaries and schemas used, etc. Plans –Harvest test & completed Bundles into IMG Registry –Work with Engineering Node to communicate with central Registry System & Tools Status

PDSMC November 29-30, Back-up

PDSMC November 29-30, Imaging Node: PDS4 Data Migration IMG manages a lot of complex data with a wide variety of formats –203 data sets, ~13 M products –Many of these are not PDS4-compliant 90% of IMG data sets require –Assessment & revision of classes of keywords Discipline Classes PDS-wide Classes –Manipulation of data to achieve compliance 25% of IMG data sets But, we assume that migration of all data sets is not required

PDSMC November 29-30, IMG Data Migration: Data Manipulation Data Manipulation –25% of IMG data sets 52 of 203 data sets –Potentially very time consuming and costly –Process: Engage experts Software development & testing V&V Peer Review? Release –Issues: Retain PDS3 version – forever?

PDSMC November 29-30, IMG Data Migration: Classes Class development will require –IMG for Discipline-specific classes –Cross-node participation for many others Classes needed for IMG - prelim analysis (not all included on chart) –Calibration – PDS-wide –Compression – PDS-wide –Geometry Camera Model – IMG Surface Geometry – PDS-wide SPICE Geometry – PDS-wide –Cartography – IMG –Imaging Instrument Parameter – IMG –Telemetry Parameters – IMG –Command Parameters – IMG –Processing History – IMG –Imaging Instrument Compression – IMG –Mosaic – IMG –Anaglyph (?) - IMG –Spectral Qube (?) - IMG

PDSMC November 29-30, Past Missions

PDSMC November 29-30, Active Missions