A quick introduction to SANs and Panasas ActivStor My initial experience with SANs was learning just enough to get the first NGS cluster operational; always had nagging doubt that I was missing a trick or two bought SANs for dummies and found, no, there really isn’t that much more to them. Original motivation for much of our SAN purchases was to support a RH GFS project to provide high performance access to storage for SCARF cluster users. GFS was troublesome to say the least, so we went back to good old ext3 + fileserver access model. Over time, more of the SAN-based storage is being used by other projects, such as VMWare, Oracle database + other NFS based file servers. Storage Networking Kevin Haines, eScience Centre, STFC, RAL

WikiPedia defines a SAN: A storage area network (SAN) is an architecture to attach remote computer storage devices (such as disk arrays, tape libraries, and optical jukeboxes) to servers in such a way that the devices appear as locally attached to the operating system. Contrast to Direct Attached Storage, usually found inside or just outside of the server, and Network Attached Storage using file based protocols such as NFS.

SAN Topology It’s like a LAN... Host/HBA 1 Storage 1 Host/HBA 2 SAN Switch

SAN Topology It’s like a LAN... it has a switch... Host/HBA 1 Storage 1 Host/HBA 2 Storage 2 Qlogic SANBox 5200 16 ports, two pairs of stacking ports (10Gb/s?) for expansion Host/HBA 3 SAN Switch

SAN Topology It’s like a LAN... it has a switch, and network cards. Host/HBA 1 Storage 1 Host/HBA 2 Storage 2 Called HBAs – host bus adaptors Host/HBA 3 SAN Switch

SAN Topology It’s like a LAN... it has a switch, and network cards. Storage 1 SAN Switch Storage 2 Host/HBA 1 Host/HBA 2 Host/HBA 3 Storage Arrays, may have one or two integral HBAs.

SAN Topology It’s like a LAN...but a little different. Fibre Connectors 2/4/8Gb/s Host/HBA 1 Storage 1 Host/HBA 2 Storage 2 Fibre connectors, not copper 2/4/8 Gb/s WWNs instead of MAC addresses Initiators and Targets HBAs don’t talk to each other (usually?) FC Protocol not Ethernet/IP SCSI over FCP Ultimate result is to present a SCSI device to the Host OS Host/HBA 3 SAN Switch Initiators Targets WWNs

Controlling Access Zoning implemented by the switch Host/HBA 1 Storage 1 Host/HBA 2 Storage 2 Ensure that HBAs only ‘see’ the storage we want them to. Not necessary, but a good precaution: NGS site incident, no zoning or LUN masking used, wrong device formatted wiping out home file system Fine on its own, as long as we’re dealing with entire storage arrays/devices as the targets, but if storage is partitioned for use by different HBAs, then Zoning on its own is not adequate. Host/HBA 3 SAN Switch Zone 1 = HBA1 + Storage1 Zone 2 = HBA2 + HBA3 + Storage 2

Controlling Access LUN masking on the storage array Host/HBA 1 LD1 HBA2 If partition your storage array, then LUN masking, implemented on storage array, Usually a list of HBAs WWNs that are allowed to read and write to the Logical Drive sometimes, can be a binary masking that includes/excludes a range of WWNs (limited use) LUN masking can be used on its own, or in combination with Zoning, but using both gives an extra level of protection from yourself. LD2 Host/HBA 3 HBA2 LD3 HBA3 SAN Switch Zone 1 = HBA1 + Storage1 Zone 2 = HBA2 + HBA3 + Storage 2

Controlling Access LUN masking on the storage array Host/HBA 1 LD1 HBA2 LD2 Host/HBA 3 HBA2 LD3 HBA3 SAN Switch Zone 1 = HBA1 + Storage1 Zone 2 = HBA2 + HBA3 + Storage 2

Increasing Resilience Multipath HOST HBA 1 SAN Switch 1 Storage 1 HBA 2 SAN Switch 2 Multipath adds resilience to failures in the HBA, SAN switch, and storage array channels, at cost of money and complexity. Two HBAs in the host, two SAN switches interconnected, and two channels to the storage array, there are 4 possible paths to the same storage in this diagram Host ‘sees’ 4 instances of the same disk, requires extra software to manage this complexity. In RHL this is devicemapper-multipath (0.4.5). Use dm-managed devices instead of /dev/sdb /dev/sdd etc. No personal experience of implementing this, nor of requiring it. SAN has been very reliable system, ~2-3 system failures in 5 years, and only 1 component failure.

Questions about SANs?

Panasas ActivStor Our goal (again) to provide high performance and resilient storage solution to the SCARF users. Wanted something easy to manage, scalable and affordable Bad experience with GFS, even if fixed - can’t afford to expand to > 600 SAN ports – HBAs, switches cabling etc. Evaluated Isilon & Panasas, tenders from both and selected Panasas.

Panasas ActivStor Sold in 4U shelf format, with 11 ‘blades’ per shelf, network switch (or two), redundant power and battery backup. All hot swappable. 2 types of blades. Director blades provide NFS access and meta-data + volume management. Up to 3 DBs per shelf. Storage blades provide 2 disks (up two 2TB per blade) and 2xGb/s ethernet (failover). Can have 11 SBs in shelf, but must be managed by a DB somewhere else.

Panasas ActivStor Director Blades Sold in 4U shelf format, with 11 ‘blades’ per shelf, network switch (or two), redundant power and battery backup. All hot swappable. 2 types of blades. Director blades provide NFS access and meta-data + volume management. Up to 3 DBs per shelf. Storage blades provide 2 disks (up two 2TB per blade) and 2xGb/s ethernet (failover). Can have 11 SBs in shelf, but must be managed by a DB somewhere else. Director Blades Meta-data and volume management services NFS access gateway

Panasas ActivStor Storage Blades Two disks (up to 2TB per blade) Sold in 4U shelf format, with 11 ‘blades’ per shelf, network switch (or two), redundant power and battery backup. All hot swappable. 2 types of blades. Director blades provide NFS access and meta-data + volume management. Up to 3 DBs per shelf. Storage blades provide 2 disks (up two 2TB per blade) and 2xGb/s ethernet (failover). Can have 11 SBs in shelf, but must be managed by a DB somewhere else. Storage Blades Two disks (up to 2TB per blade) 2xGb/s ethernet (failover mode) 11 per shelf (must be managed by DB)

Panasas ActivStor Resilience Resilience to failure is achieved by striping the data across the SBs, with one or more SBs worth of storage allocated as spare capacity. Can configure steadily decreasing number of SBs. E.g. Start with 3 then reduce number as necessary. It is not the case that one entire SB stands idle –every SB is involved in serving the clients, therefore getting full performance capability of system. Comprehensive System monitoring, detect early signs of blade failure and initiate blade drain for pre-emptive replacement. DBs provide volume management/metadata services to rest of system, as well as NFS, and can be configured for failover. Data is striped across all Storage Blades (RAID5 or RAID1) One or more Storage Blades’ worth of space reserved for failures System monitoring for pre-emptive actions (Blade Drain) Two or more Director Blades can provide failover for each other

Panasas ActivStor Performance DirectFLOW clients Available for most major Linux distributions Directly communicates with Storage Blades RAID computations performed by client ~5000 supported (12000 option) To get best performance, need to use the DirectFLOW client, implemented as a Linux Kernel module

Panasas ActivStor Performance DirectFLOW clients Available for most major Linux distributions Directly communicates with Storage Blades RAID computations performed by client ~5000 supported (12000 option) Results show scalable claim is true RAL: 1.2 GB/s from 22 nodes (2 shelves) RoadRunner: 60GB/s (103 shelves) Summary: Feel we’ve got a good balance between SAN attached storage, where fast attached storage needed, and high-performance network-based global filespace offered by Panasas.