Storage Wide-Area Networks (SWANs) Randy H. Katz Computer Science Division Electrical Engineering and Computer Science Department University of California, Berkeley Berkeley, CA 94720-1776

Storage Networks Primary goal is to share storage among computers in complex, heterogeneous environments, with PCs, workstations, file servers, and mainframes Storage can be: Direct attached (e.g., host bus adapter/HBA) Network attached (via a file server) Channel attached (primarily fibre channel, but also IBM SSA)

Storage Networks E.g, Hitachi, EMC, IBM Storage Arrays Mainframe Fibre Channel Loops Dual Ported Disks And Controllers Remote Storage Manager DI DI DI DI Cache Workstation Cache Mainframe Crossbar Interconnect Cache Fibre Channel Or ESCON Cache HI HI HI HI Multiple Host Interconnections

Storage Networks Major development in 1990s: storage networks Native FC is a point-to-point or loop/string-oriented (“arbitrated”) method of interconnection; 1 Gbps, extend up to 10 km Emergence of FC “fabrics”: FC switches arranged hierarchically to enable connectivity between any host and any storage device (e.g., Brocade Networks dominates this product space) Standardize FC protocol stack lives on top of such fabrics: device naming, transport, CoS, etc. SCSI-3 protocol over FC fabrics Gigabit ethernet now emerging in SAN environment

Basic Attached Storage Host OS Disk Interface (DI) Allocation Table Disk, Cylinder,Track, Sector Device attached by SCSI HBA or channel interface Host manages the file-to-block mapping

Network-Attached Storage (NAS) aka Network File Service Host Network Interface (NI) Network File Server Host LAN File Name, Offset, Length OS Network Interface (NI) Host Network Interface (NI) Mapping from File to Block done in network-attached File Server, not host

Network-Attached Secure Devices (NASD) Host Gibson@CMU: Research project on device embedded protocol stack, authentication OS Network Interface (NI) Network File Server Host LAN Disk, Cylinder, Track, Sector File Name, Offset, Length OS Network Interface (NI) Host Network Interface (NI) Network-attached Secure Device (NASD)

Storage Platforms “Storage Virtualization” Main Frame LUN = Logical Unit Logical disk mapping onto underlying physical disks on logical block to physical block basis OS LUN, Offset, Length Channel Interface Main Frame Disk Storage Subsystem Work Station LUN To PHY

NAS NAS distinguished by an exported Network File System interface over a standard Local Area Network-based transport File Server Host Network Interface (NI) Host LAN File Server File Name, Offset, Length Network Interface (NI) Host Network Interface (NI) File Server

NAS + SAN SAN distinguished by a block-oriented interface; Usually implemented across a channel-oriented fabric Main Frame PHY Device, Cyl, Trk, Sector File Server Host Disk Storage Subsystem Network Interface (NI) Channel Interface CI Host File Server SAN LUN, Offset, Length LAN Tape Storage Subsystem LUN, Offset, Length File Name, Offset, Length CI Network Interface (NI) CI CI Host Network Interface (NI) Optical Disk Storage Subsystem File Server Main Frame

NAS + SAN + SWAN Now extend the NAS or the SAN over a wide-area network transport … NOTE: wide-area SAN is new idea Optical Disk Storage Subsystem SAN Main Frame Tape Channel Interface LAN Host Network Interface (NI) File Name, Offset, Length File Server CI LUN, Offset, Length Gate way WAN FS DSS Remote SAN PHY Device, Cyl, Trk, Sector

Shared Storage Reference Model Application File/Record Subsystem Host Resource Mgmt, Configuration Redundancy Mgmt, Back-up Service Subsystem High Availability, Fail-over Discovery, Monitoring Security, Billing Capacity Planning Block Aggregation Storage Domain SAN Device Block Subsystem

SAN Reference Model Block-oriented Application SAN File FS Host-based SAN-based Block Device-based DA

SAN Reference Model Application NAS Storage Host Host LAN File FS NAS Block Device

SAN Reference Model Application Heterogeneous Storage Environment Host s/w raid Host Host Host LAN File FS NAS Head NAS Host SAN SAN Block Device DA

FC-1 Encoding, Link Control Seven Layer Stack IP NFS, CIFS FTP, SNMP, TFTP, Telnet, FCP, SCSI-3 TCP, UDP LAN, MAN, WAN Phy Sw GigaE NFS, CIFS FTP, SNMP, TFTP, Telnet, FCP, SCSI-3 TCP, UDP IP Mac Client/ Control Phy FC SCSI-3 VI IP FC-4 Protocol I/F FC-3 Encrypt/ Authentication FC-2 Framing, FC, Class of Service FC-1 Encoding, Link Control FC-0 Phy 7 6 5 4 3 2 1 Application Presentation Session Transport Network Data Link Physical

Fibre Channel Protocol Stack FC-0: gigabit physical layer FC-1: data encoding and link layer control FC-2: segmentation/reassembly of data frames, flow control, class of service FC-3: common services, e.g., encryption FC-4: “upper layer protocol” upon which SCSI-3 or IP can run Contrast with Gigabit Ethernet Ethernet framing, VLAN tagging, frame prioritization (8 levels), link aggregation, 1.25 gbps IP + Gigabit Ethernet emerging for SANs

(e.g., Fibre Channel or Serial SCSI over IP) SCSI-3 Terminology Target Initiator Request Device Server LUNs Application Client SCSI Client-Server Model Response Delivery Subsystem (e.g., Fibre Channel or Serial SCSI over IP)

FC over IP (FCIP) IP Network WAN Tunnel Session Server Tape Storage Subsystem Tape Storage Subsystem Server WAN IP Network FC Switch FC Over IP FC Over IP FC Switch Server JBOD Server JBOD Tunnel Session IETF IP Storage (IPS) working group Recall FC fabrics developed in context of machine room/building-scale interconnect (e.g., no congestion control!) FC time outs in wide-area? Flow control interaction? QoS? How does bridging actually work for FC e2e management?

Internet FC Protocol (iFCP) FC_Device N_Port F_Port iFCP layer FCP Portal FC Traffic IP Network FC Device Address IP Address Mapping Control Data iFCP gateway region iFCP Frames Gateway to gateway protocol, sessions rather than tunnels TCP for congestion control, error detection, recovery Plug FC devices directly into iFCP switches Session and naming semantics

iFCP Services 24-bit N-Port Address: <Domain, Area, Port> iSNS: Internet Storage Name Service—discovery and management protocol for IP storage networks (IPNSP) Protocol specification includes address translation feature to allow remote storage devices to be assigned a local, FC fabric compliant address Local commands executed locally on the fabric Remote commands executed on top of TCP connections Error Detection/Time Outs Security

Other Protocols Metro Fibre Channel Protocol (mFCP) FCP over IP using UDP rather than TCP (link layer flow control and pacing) Internet SCSI (iSCSI) IP to the storage device Serial SCSI block data transfer over IP (SCSI Access Method Command Set—SAM) IPSec, command/data ordering, steering to application memory

iSNS Discovery Process iSNS objects Device registration WWN or iSCSI names Zoning/discovery domains iSNS objects Portals Storage Port Storage Nodes

Storage Applications Data Centers IP Storage for Remote Applications Server clustering Storage centralization, consolidation, management LAN-free back-up IP Storage for Remote Applications Remote back-up Remote mirroring Disaster recovery Content distribution

Rhapsody Networks, Inc. Storage Application Director “Alteon box for storage networks” Peek into storage packets traversing fabric (“deep frame classification”) and invoke code—for encryption/decryption, mirroring, LUN mappings, etc. Per port software processing and cut-through fabric routing Data copy engine, table lookup engine, in-transit I/O mods, data escrow/trap to software for complex errors or event processing Intelligent queue management