1. Presented at the THIC Meeting at the National Center for Atmospheric Research, 1850 Table Mesa Drive, Boulder CO 80305-5602 July 19-20, 2005 Evolution of Archive Technologies at the National Snow and Ice Data Center Ruth Duerr National Snow and Ice Data Center 1540 30th St, Boulder, CO, 80309-0449 Phone 303-735-0136 FAX: 303-492-2468 E-mail: rduerr@nsidc.colorado.edu

2. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 Outline A brief history of NSIDC Current holdings Current systems Thoughts on the future

3. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 Outline A brief history of NSIDC Current holdings Current systems Thoughts on the future

4. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 A brief history of NSIDC World Data Center for Glaciology  1957 - U.S. National Committee for the IGY awarded the operation of WDC-A for Glaciology to the American Geographical Society under the direction of Dr. William O. Field  1970 - WDC for Glaciology transferred to the U.S. Geological Survey in Tacoma, Washington under the direction of Dr. Mark F. Meier  1976 - WDC for Glaciology transfers to the NOAA Environmental Data and Information Service; an agreement between the University of Colorado and NOAA placed the WDC at CU-Boulder, Colorado under the direction of Roger Barry

5. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 A brief history of NSIDC National Snow and Ice Data Center  1982 - NOAA grants NSIDC its name  1983 - NSIDC receives a grant from NASA to archive Nimbus 7 passive microwave data  1990 - NSIDC receives funding from NSF for the Arctic System Science (ARCSS) Data Coordination Center (ADCC)  1993 - NSIDC receives first NASA Distributed Active Archive Center (DAAC) contract

6. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 A brief history of NSIDC 1996 - Antarctic Data Coordination Center (ADCC) established with NSF support 1999 - Antarctic Glaciological Data Center (AGDC) established with NSF support 2002 - Frozen Ground Data Center (FGDC) established with International Arctic Research Center (IARC) support

7. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 Outline A brief history of NSIDC Current holdings Current systems Thoughts on the future

8. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 Current Holdings Information Center  ~44,000 monographs, reports, serials, reprints, etc.

9. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 Current Holdings (continued) Analog Archives ~ 10,000 glacier photos ~ 7,000 sea ice charts ~ 1,440 maps TBD cu ft of manuscripts and other records

10. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 Data Holdings (continued) Digital Archives ~ 440 publicly advertised data sets 4.6 million granules in ECS system > 3.5 million files in non-ECS systems Archive Types ~ 8 TB on-line ~ 80 TB near-line >5 TB off-line Off-site backups for primary data without recovery agreements

11. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 NSIDC Near-Line Archive Size

12. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 NSIDC Distribution Statistics

13. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 Outline A brief history of NSIDC Current holdings Current systems Thoughts on the future

14. UCB GE Campus Lan Abilene via Front Range GigaPop Cisco GE switch ECS Router Ebnet Router Firewall LASP GES DAAC (MODIS) AMSR-E SIPS I-SIPS PO.DAAC AMSR L1A EDOS M&O LAN (Includes M&O Intranet Server and Backup Server) Ingest & Distribution ftp/file server tape production Disk Storage (FC and SATA @ RAID-5) Rimage CD/DVD AMSR L1A PDR/Met ECS Production LAN UCB non-ECS LAN Web Services Guide Documents SOTC, All about… Catalog/DIF generator Client Interfaces - EDG,GISMO,PSQ, SNOWI - ECHO Client (WSRD) Archive Services MAID vtl w/ AMASS (transition from tape archive in progress) Off-site archive Raytheon (Denver) Infrastructure Services Email Calendar Manager Center-wide Intranet Backup Services SNIPS IDS Science Processing Dev /Production SSM/I, NISE, NRTSI - Std Processing Env. GISMO/PSQ/WSRD -backend services AMSR-E PDR/.Met ICESat/GLAS Remote SCF Server Visualization Subsetting Services (Includes storage for Archived GLAS products ) HEG Server HDFEOS-to-GEOTIFF ECS Subsetter Spatial, temporal and parameter subsetting of granules (HDF-EOS Only) Data Management Data Dictionary V0Gateway - Inv. Search/Results Order Requests - subsetting services ECHO transfers Data Server Manage the archive - STK Powderhorn w/ AMASS and ACSLS Insert data into archive Search and Retrieve data from the archive Ingest Polling Server ASMR, AMSR-E NISE MODIS ICESat Data Pool On-line storage of most recently ingested data (StorNext SAN) Data accessible via WEB GUI or ftp Infrastructure Backup services What’s Up SNIPS MSS and CSS Email Gateway Order Manager Manages orders from V0Gateway, MTMWG, and Spatial Sub. Server Transfers orders to PDS for media requests NSIDC Architecture VJT 7/05

15. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 NSIDC Data Catalog Contains metadata about each published data set  GCMD, FGDC compliant metadata Used to drive web page creation Modification underway to include OAIS/PREMIS compliant metadata for all data sets

16. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 EDG CLIENT MTMGW Subscriptions QA SCFs SCFs SCIENCE USERS GENERAL USERS ECS NSIDC DAAC NISE NSIDC PDR Server EDOS AMSR L1A [1.3] GBAD PDS [<1] AMSR-E PDS [1.4] NISE Data AMSR-E SIPS L2+ Products [3.6] Total Data Distribution Aqua AMSR-E Data Products [7] Aqua MODIS Data Products [16] Terra MODIS Data Products [40] ADEOS II AMSR Data Products [1] GLAS Data Products [7] [<1] GDAAC LDAAC LaTIS NISE Data [total <1] MODIS SCFs MODIS L2/L3 Snow/Ice Products QA Updates [2/T, 1/A]] [<1/T, <1/A] MODAPS MODIS L2/L3 Snow/Ice Products [120/T, 16/A] [ ] : Archive Volumes at L+1yr in GB/day T: Terra A: Aqua NASDA AMSR-E L0 Science & GBAD PDSs 8mm back-up 1st 90 days [1.4] Emergency back-up following L+90 GLAS L0 [5.8] GLAS SIPS GLAS L1+ Products [23] Data/QA Updates EDOS LP DAAC ASTER Anc [<1] GDAACAQUA Ancillary from EMOS [<1] ECHO Metadata and Browse NSIDC DAAC ECS Interfaces

17. NSIDC/UCB Interfaces

18. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 COPAN MAID - Early Experiences Using AMASS to provide a file system front end to a COPAN 200t system configured as a L700 with 7 - 9940 drives  COPAN to act as a drop-in replacement for the STK 9710 Migration of data underway The big surprise was performance Minor surprise is the ever shrinking total archive size

19. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 Outline A brief history of NSIDC Current holdings Current systems Thoughts on the future

20. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005 Thoughts on the Future NSIDC DAAC is pushing for an entirely on- line archive  What technologies will allow this? (SATA RAID, COPAN, or ???)  What happens to media?  How do we ensure preservation of these data over time?