Peter Stefan, NIIF 29 June, 2007, Amsterdam, The Netherlands NIIF Storage Services Collaboration on Storage Services

 Objective of our storage development  Brief NIIF introduction  Driving force of storage development: users need it!  Storage building blocks:  FC,  AoE,  „Design and develop your own” box.  The two storage node solutions at NIIF  How can users access storage volumes?  Storage node integration  Conclusions Agenda NIIF –

 NIIF is the Hungarian NREN, connecting about 450 academic institutions and users.  Our service portfolio is as follows:  backbone data network (HBONE),  videoconferencing services,  web, mail, AAI, FTP, DNS services,  supercomputers, grids, and  data storage facilities.  Financed mostly by the Hungarian Ministry of Economics and Transportation. Who we are? NIIF –

Who we are? NIIF –

 There is a continuous need for large volume, moderate speed and high availability storage solutions.  Key usage areas that we know about are:  library archives,  on-line network monitoring (netflow),  videoconferencing,  grids,  off-site back up of large-volume servers. Why large storage volumes? NIIF –

 Fibre Channel (FC) SAN/NAS solutions dominate the market.  Experience with StorageTek (15000 USD/TB), T3, CX500 (6000 USD/TB), SFS (45000 USD/TB).  Generic experience:  FC works well  high price/{capacity,performance} figures   rather large operational costs (disk replacement, support)   not optimal choice for disk-based back ups.  Bulding blocks - FC NIIF –

 Interesting idea: use only Ethernet as storage interconnection!  ATA-over-Ethernet protocol by Coraid Inc.  Key idea: ATA commands are wrapped into Ethernet frames; the AoE driver strips Ethernet frames, then extracts ATA commands and conveys them as block devices. Bulding blocks - AoE NIIF –

 Two conceptually different AoE implementations:  Parallel ATA (older, 5 MB/s),  Serial ATA (newer, 60 MB/s).  Generic experience with the SATA boxes:  moderate performance figures in certain cases   very good price/performance value (1200 USD/TB), and prices keep dropping  low operational costs (standard disks, very simple architecture)  to build a PByte-scale system further virtualization layer is required  Bulding blocks - AoE NIIF –

 Use a multi-disk PC chassis with Linux!  SATA or SAS disks,  standard RAID cards,  PCI-express bus system,  iSCSI and AoE protocols,  universal storage box: GE, Infiniband, 10 GE.  Generic experience with ours:  modular  can be driven up to the hardware boundaries  our record is: 700 MB/s using RAID0 and 450 MB/s using RAID10  no vendor support  „Design and develop your own box” NIIF –

 The purpose is to provide a single, large disk volume for grid users.  Furbrished from 7 PATA boxes, 2 Cisco Catalyst switches, and 2 64-bit servers.  Involves 7x10x400 MBs of disks. Sum: 28 TBs.  Two storage processors are in master-slave layout.  Volumes are organized in ten vertical RAID6 arrays concatenated by LVS. This provides box redundancy.  Available via grid middleware and SCP-only.  Weak points: low IO (50-55 MB/s) , lack of switch-level redundancy , not scalable . Node solution 1 - PATA storage NIIF –

Node solution 1 - PATA storage NIIF –

 The purpose is to provide a single, large disk volume that can be split to smaller, yet scalable units for generic usage.  Made up of 4 SATA boxes, 2 Cisco Catalyst GE switches, and 2 servers.  GE is used as storage interconnection.  Involves 4x15x500 GBs of disks. Sum: 30 TBs.  Two Linux storage processors with HA.  Volumes are organized in 15 vertical RAID5 arrays concatenated by LVS.  RAID + volume management is performed on storage proeccors. Node solution 2 - SATA storage NIIF –

 Full box-, switch-, and storage processor-level redundancy.  Available via iSCSI for local users.  Has virtualization capability, i.e. any boxes and any storage protocols can be used (FC, AoE, iSCSI, Infiniband, Tape).  Can be used for building up hierarchical storage systems.  Not-yet FC-equivalent in terms of e.g. cache coherency .  IO is about MB/s. Node solution 2 - SATA storage NIIF –

Solution #2 - SATA storage NIIF –

Solution #2 - SATA storage NIIF –

 Storage nodes can provide block-level, file-level, and application-level access to the large disk volumes.  Block-level:  AoE,  iSCSI (particularly kind to NRENs).  File-level:  NFS, SMBFS,  Cluster file systems (GFS, Lustre).  Application-level:  SSH, SCP,  storage-capable grid middleware. How to give access to users? NIIF –

 Current status: 2 nodes, lot of plans for the future.  Create multiple redundant storage nodes at different parts of our network, i.e. at regional centers, or at large customers.  Integrate them at application level, by using storage management software, like SRM, or Grid Underground (GUG) storage management modules.  The target system is a geographically distributed storage system providing distributed, replicated, yet safely protected data sets, and standard interfaces.  First inter-node service: distributed and encrypted backup service. Storage node integration NIIF –

 During our storage development activities we revealed that building cost-efficient, reasonable- speed and reliable storage nodes is possible based on cheap building blocks ands free software.  To handle complexity we believe that application- integrated hierarchical storage systems are needed.  NRENs can play integrating role due to their positions.  QoS, monitoring and, in general, strict operational disciplines are necessary compared to, say, grid systems.  There is still a lot to develop: management, advanced monitoring, cache coherency. Conclusions NIIF –

? Thanks & Questions