1 U.S. Department of the Interior U.S. Geological Survey Contractor for the USGS at the EROS Data Center EDC CR1 Storage Architecture August 2003 Ken Gacke Systems Engineer (605)

2 Contractor for the USGS at the EROS Data Center Storage Architecture Decisions Evaluated and recommended through engineering white papers and weighted decision matrices  Requirements Factors Reliability – Data Preservation Performance – Data Access Cost – $/GB, Engineering Support, O&M Scalability – Data Growth, Multi-mission, etc. Compatibility with current Architecture Program/Project selects best solution

3 Contractor for the USGS at the EROS Data Center Unix Storage in Terabytes – July 2002

4 Contractor for the USGS at the EROS Data Center Unix Storage in Terabytes – Jan 2003

5 Contractor for the USGS at the EROS Data Center Storage Technologies Online Storage Characteristics  Immediate Data Access  Server Limitations Number of I/O slots System Bandwidth  Cost is Linear High Performance RAID -- $30/GB using 146GB drives Low Cost RAID -- $5/GB using ATA or IDE Drives Non RAID – Less than $5/GB using 146GB drives  Facility Costs Disk drives are always powered up Increased cooling requirements  Life cycle of 3 to 4 years

6 Contractor for the USGS at the EROS Data Center Storage Technologies Online Storage  Direct Attach Storage (DAS) Storage directly attached to server  Network Attach Storage (NAS) TCP/IP access to storage typically with CIFS and NFS access  Storage Area Network (SAN) Dedicated high speed network connecting storage devices Storage devices disassociated from server

7 Contractor for the USGS at the EROS Data Center Storage Technologies Direct Attach Online Storage  Disk is direct attached to single server  System Configuration SCSI or Fibre Channel  RAID Fibre Channel devices are typically SAN ready Just a Bunch of Disk (JBOD) Redundant Array Independent Disk (RAID)  High Performance on the local server  Manageability Simple Configuration Resource reallocation requires physical move of controllers and disk

8 Contractor for the USGS at the EROS Data Center Storage Technologies Direct Attach Online Storage  Advantages High performance on local server Good for image processing and database applications  Disadvantages Data sharing limited to slower network performance Difficult to reallocate resources to other servers

9 Contractor for the USGS at the EROS Data Center Storage Technologies Direct Attached Host A File System Host B File System Host C File System 100Mb Network (FTP/NFS) 100MB FC

10 Contractor for the USGS at the EROS Data Center Storage Technologies NAS Online Storage  Disk attached on server accessible over TCP/IP Network  System Configuration Fibre Channel RAID Configurations Switched Network Environment  Performance Network Switches and/or dedicated network topologies  Reliability NAS Server performs a single function thereby reducing faults RAID, Mirror, Snapshot capabilities  Easy to Manage

11 Contractor for the USGS at the EROS Data Center Storage Technologies Network Attach Online Storage  Advantages Easy to share files among servers  Network Storage support NFS and CIFS  Servers can use existing network infrastructure Good for small file sharing such as office automation Availability of fault protection such as snapshot and mirroring  Disadvantages Slower performance due to TCP/IP overhead Increases network load Backup/Restore to tape may be difficult and/or slow Does not integrate with nearline storage

12 Contractor for the USGS at the EROS Data Center Storage Technologies Network Attached 1Gb Network (NFS/CIFS) Host AHost BHost C File System Share Files NAS Server

13 Contractor for the USGS at the EROS Data Center DAS/NAS Sequential I/O Throughput

14 Contractor for the USGS at the EROS Data Center DAS/NAS Random I/O Throughput

15 Contractor for the USGS at the EROS Data Center Storage Technologies SAN Online Storage  Disk attached within Fabric Network  System Configuration Fibre Channel RAID Configurations  Scalable High Performance  High Reliability with redundant paths  Manageability Configuration becomes more complex Logical reallocation of resources

16 Contractor for the USGS at the EROS Data Center Storage Technologies Host A Host B Host C SAN Configuration 100Mb Network Fibre Switch

17 Contractor for the USGS at the EROS Data Center Storage Technologies Host A Host B Host C Redundancy SAN Configuration 100Mb Network Fibre Switch (DMF) Fibre Switch

18 Contractor for the USGS at the EROS Data Center Storage Technologies SAN Online Storage Architecture  Disk Farm Multiple servers share large disk farm Server mounts unique file systems  Clustered File Systems Multiple servers share a single file system Software Required – Vendor solutions include  SGI CXFS  ADIC StorNext File System  Tivoli SANErgy

19 Contractor for the USGS at the EROS Data Center Storage Technologies Host A Host B Host C Disk Farm SAN Configuration 100Mb Network Fibre Switch Logical reallocation of disk

20 Contractor for the USGS at the EROS Data Center Storage Technologies Host A Host B Host C Cluster SAN Configuration 100Mb Network Fibre Switch CXFS Clustered File System

21 Contractor for the USGS at the EROS Data Center Storage Technologies SAN Risks  Cost is higher than DAS/NAS  Technology Maturity Solutions are typically vendor specific Application software dependencies  Infrastructure Support Complexity of Architecture Management of SAN Resources Sharing of storage resources across multiple Programs/Projects

22 Contractor for the USGS at the EROS Data Center Storage Technologies SAN Benefits  Administration flexibility Logically move disk space among servers Large capacity drives can be sliced into smaller file systems Scales better than direct attach Integrate within nearline configuration  Data Reliability Storage disassociated from the server Fault Tolerant with Redundant Paths  Increase Resource Utilization Reduce the number of FTP network transfers Logically allocate space among servers

23 Contractor for the USGS at the EROS Data Center Storage Technologies Host A Host B Host C SAN with Nearline Configuration 1Gb Network Fibre Switch CXFS DMF/CXFS Clustered File System Tape Library

24 Contractor for the USGS at the EROS Data Center Online/Nearline Cost Comparison Use of Existing Infrastructure (CR1 Silo)

25 Contractor for the USGS at the EROS Data Center Storage Technologies Bulk RAID Storage Considerations  Manageability Server connectivity constraints Many “islands” of storage  Multiple storage management utilities  Multiple vendor maintenance contracts  Data Reliability Loss of online file system requires full restore from backup  On average, could restore one to two terabyte per day  Performance Multiple user access will reduce performance  Life Cycle Disk storage life cycle shorter then tape technologies

26 Contractor for the USGS at the EROS Data Center Storage Technologies SAN Nearline Storage  Data Access Data stored on infinite file system  Immediate access to data residing on disk cache  Delayed access for data retrieved from tape Access via LAN using FTP/NFS Access via SAN Clustered File System  SGI DMF/CXFS Server  SGI, SUN, Linux, NT clients

27 Contractor for the USGS at the EROS Data Center Storage Technologies SAN Cluster Proposal  Mass Storage System & Product Distribution System (PDS)  Limit Exposure to Risk Servers are homogeneous Implement with Single dataset Data is file orientated Data currently being FTP  Anticipated Benefits Improved performance Reduce total disk capacity requirements Experience for future storage solutions

28 Contractor for the USGS at the EROS Data Center Proposed SAN Configuration DMF Server Product Distribution CXFS SAN Storage 8x9840 SCSI Tape Drives 1Gb Fibre 2Gb Fibre Sun ServerNT/Linux Server Future

29 Contractor for the USGS at the EROS Data Center Current DMF/SAN Configuration DMF Server Product Distribution CXFS SAN Storage Tape Drives 8x9840 2x9940 1Gb Fibre 2Gb Fibre Disk Cache /dmf/edc 68GB /dmf/doqq 547GB /dmf/guo 50GB /dmf/pds 223GB /dmf/pdsc 547GB

30 Contractor for the USGS at the EROS Data Center CR1 Mass Storage System

31 Contractor for the USGS at the EROS Data Center CR1 Mass Storage System

32 Contractor for the USGS at the EROS Data Center CR1 Mass Storage System

33 Contractor for the USGS at the EROS Data Center CR1 Mass Storage

34 Contractor for the USGS at the EROS Data Center CR1 Mass Storage

35 Contractor for the USGS at the EROS Data Center CR1 Mass Storage Description 1996 – 3490, pre DOQQ 1999 – D-3, DOQQ 2002 – 9840, DOQQ 2003 – 9840/9940, UA/AVHRR Av 12.1MB/sec

36 Contractor for the USGS at the EROS Data Center CR1 DMF FY04 Budget

37 Contractor for the USGS at the EROS Data Center Storage Technologies Multi Tiered Storage Vision  Online Supported Configurations  DAS – Local processing such as image processing  NAS – Data sharing such as office automation  SAN – Production processing such as product generation Data accessed frequently  Nearline Integrated within SAN Scalable for large datasets and less frequently accessed data Multiple Copies and/or Offsite Storage

38 Contractor for the USGS at the EROS Data Center Storage Technologies SAN – Final Thoughts  SAN Technology Maturity SAN solution should be from a single vendor  Program/Project SAN solution benefits + Decrease storage requirements + Increase performance + Increase reliability + Increase flexibility of resource allocations - Increase cost (hardware/software) - Increase configuration complexity

39 Contractor for the USGS at the EROS Data Center EDC Architecture Success Unix Servers – SGI/SUN/Linux  Oracle, Image Processing, Data Distribution  Synergy with NASA Storage Architecture  JBOD Storage – Transient data  RAID Storage – Databases, Datasets  Hierarchical Storage – Large Datasets