Implementation Review1 Moving Archive Data to the EMC Storage Array March 14, 2003 Faith Abney

Implementation Review2

3 Motivation for Migration EMC has higher availability and throughput  Less downtime  Data gets to users quicker More online processing space available on EMC  Current processing areas not large enough to process all retrieval requests when they are submitted  Requests are routinely queued and held until processing space is available

Implementation Review4 Portion of EMC array to be used for faster access to Archive data Currently all data is on MO platters in jukeboxes  Limited by number of platters mounted  Slow read/write access  Greatly affected by hardware downtime Moving data to the EMC array will eliminate these bottlenecks which are due to data being in a jukebox

Implementation Review5 Portion of EMC array to be used for faster access to Archive data Calling this area the “Data Depot” to avoid confusion with other staging areas DADS will check this area first (DADS 10.1)  if files are available will use them  if not, it will get them from MO Ingest software will be updated to write appropriate data to the depot after write to MO (DADS 10.1)  Data to depot will come from read of MO platter, verifying the write to MO

Implementation Review6 Data going to the Depot All POD files (raw science data) for instruments with OTFR – 1.5 TB  ACS, NICMOS, STIS, WFPC2 All legacy data -.5TB  GHRS, FOS, FOC, WFPC, HSP, AST Calibration reference files -.15TB FUSE data -.3 TB Jitter files, Metadata - <.1TB Mission scheduling data < 50 GB  SMS, MTL, MSC, ORB

Implementation Review7 Data not going to the Depot Some data is rarely requested and does not need faster access:  Calibrated data for which OTFR is available  Raw Engineering data  Log and Ancillary data

Implementation Review8 Methodology: Requirements Any movement of Data from MO platter to EMC will compete with both Ingest and Distribution for Jukebox access.  Migration should not prevent operations from meeting performance goals.  Migration of data off jukebox can proceed as background task  Task becomes high priority if jukebox failure rate begins to impact performance goals in operations.  Migration should allow for selection of highly requested data classes to be segregated and moved first.  POD files (raw science) for OTFR prime candidate to reduce overall retrieval times.

Implementation Review9 Methodology: Options MMM server within NSA  Used to populate MO from older Sony media  Required manual intervention for simultaneous efficient operations and transfer performance  Well understood operationally Bulk copy using primary platters  Same copy mechanism used for other mirror archives (CADC, ECF)  Not a routine operational procedure, manual intervention DADS 10.* beta test environment  V280R loading uncertain  Rapid Test and Build cycle for beta test environment Copy from safe store platters using DADS 9.* test environment.  Highly manual (requires platter loads)  No operational impact (impacts testing schedule only)

Implementation Review10 Methodology: Prescription Perform trade studies to evaluate the quickest and least intrusive method. Develop contingencies to accelerate transfer should jukebox failures become more frequent. Initiate transfer and develop a good understanding of maximum transfer rate before predicting a completion date.  Lessons learned from MO migration  Note that DADS 10.1 delivery makes transfer progress non linear!

Implementation Review11 Schedule DADS 10.1 – scheduled for Spring – contains:  Software to populate Depot on ingest  Software to be able to read from the Depot Tools to migrate existing data to Depot already written  Migration can start as soon as EMC Depot is connected to operational system  Migration completion estimate will be prepared after we complete acceptance testing and get actual performance numbers using the EMC and our tools