Investigating Serial Attached SCSI (SAS) over TCP (tSAS) UCCS Master’s Project Deepti Reddy

Outline of the Talk  Introduction to tSAS and related technology  Motivation and Challenges  tSAS Design  Performance Evaluation of a Mock Application  Lessons Learnt and Future Directions  Conclusion 211/3/2011tSAS/sreddy

SCSI vs SAS SCSI (Small Computer System Interface) The SCSI protocol is an application layer storage protocol. It's a standard for connecting peripherals to your computer via a standard hardware interface, which uses standard SCSI commands. SAS (Serial Attached SCSI) SAS is the successor of SCSI technology Initially introduced at 3Gb/s in 2004 Currently, supports 6Gb/s 12Gbps by 2012 Significantly increased the available bandwidth offered by legacy SCSI storage systems. Use of expanders increases the scalability over legacy SCSI 311/3/2011tSAS/sreddy

SCSI vs SAS SCSISAS TopologyParallel BusSerial Bus Reduces Noise (Better signal integrity) Speed3.2 Gbps3 Gbps, 6Gbps, 12Gbps Distance1 to 12 meters8 meters Number of Targets14 devices128 expanders. > 16,000 with cascaded expanders DevicesSCSI onlySAS & SATA Used widely in Data Centers today ConnectivitySingle PortDual Port Fail-over/fail-back Drive Form Factor3.5”2.5” CostLowMedium 4 11/3/2011 tSAS/sreddy

iSCSI iSCSI protocol describes a means of transporting of the SCSI packets over TCP/IP  The iSCSI specification is revised and updated by the Internet Engineering task Force (IEFT).  Work by S. Aiken, D. Grunwald, A. Pleszkun and J. Willeke shows the performance of a commercial iSCSI software implementation compared quite favorably with fibrechannel [7] 5 iSCSI protocol layering model 11/3/2011tSAS/sreddy

Typical SAS Topology - DISK DRIVES - EXPANDERS Driver ~8m 611/3/2011tSAS/sreddy

Motivation & Challenges  Overcome the distance and scalability limitations of traditional Serial Attached SCSI (SAS) by using tSAS.  Like iSCSI, tSAS takes advantage of existing internet infrastructure, internet management facilities as well as addresses distance limitations  Provide research results for future industry specification for tSAS and iSCSI. 711/3/2011tSAS/sreddy

Related Work  Michael Ko’s patent at IBM Research-Almaden on Serial Attached SCSI over Ethernet proposes a very similar solution to the tSAS solution provided in this project.  iSCSI specification (SCSI over TCP) itself is similar to a tSAS solution (SAS over TCP). The iSCSI solution can be heavily leveraged for a tSAS solution.  The Fibre Channel over TCP/IP specification also can be leveraged to design and implement a tSAS solution. 811/3/2011tSAS/sreddy

tSAS Topology 9 - DISK DRIVES - EXPANDERS SCSI Driver - TCP link - SAS link SCSI Driver 11/3/2011tSAS/sreddy

 The topology for tSAS on the previous slide (similar to Michael Ko’s proposal) can be extended such that tSAS is also used between a HBA and an expander in a SAS topology. This helps increase scalability of tSAS. 11/3/2011tSAS/sreddy10 tSAS Topology

11 - DISK DRIVES - EXPANDERS SCSI Driver - TCP link - SAS link SCSI Driver tSAS Topology 11/3/2011tSAS/sreddy

Goals of the project  Investigate how tSAS can be implemented  Design and develop a mock tSAS application −Write a client program using C/C++ that sends a set of tSAS commands to a server. −Write a server program using C/C++ that receives a set of tSAS commands, processes them and responds to the client with tSAS responses.  Evaluate the tSAS solution and compare with a mock iSCSI client/server application. 1211/3/2011tSAS/sreddy

Software and Hardware solutions for tSAS implementations  Software based tSAS solution −Cheaper than a hardware based solution −All tSAS processing is done by the processor and TCP/IP operations are also executed by the CPU −NIC is merely an interface to the network  TCP Offload Engine (TOE) solution −NICs with integrated TOEs have hardware built into the card that allows the TCP/IP processing to be done at the interface  Hardware based tSAS solution −In a hardware-based tSAS environment, the initiator and target machines contain a host bus adapter (HBA) that is responsible for both TCP/IP and tSAS processing 1311/3/2011tSAS/sreddy

 Broadcom on their h/w iSCSI solutions states the following “Software iSCSI initiators consume considerable CPU cycles when handling I/O-intensive workloads, leaving little headroom for growing user application requirements. Broadcom’s iSCSI HBA architecture minimizes the CPU overhead so valuable CPU cycles are allocated to process user applications. Broadcom iSCSI HBA only uses 1/3 of the CPU cycles required by the software initiator for a 4K I/O size read operation” 11/3/2011tSAS/sreddy14 Software and Hardware solutions for tSAS implementations

tSAS Approach  The tSAS Request is initially sent by the tSAS Initiator to the tSAS Target over TCP.  The tSAS Target strips off the TCP header and sends the SAS frames using the SAS Initiator block on the tSAS Target to the SAS expander/drive.  The SAS expander/drive sends SAS frames to the tSAS Target.  Finally, the tSAS Target embeds the SAS frames received from the expander/drive over TCP and sends it to the tSAS Initiator. 1511/3/2011tSAS/sreddy

tSAS Message Format  The above shows how a legacy SAS header and data are embedded in an Ethernet frame.  iSCSI uses the same approach where the iSCSI header and data are encapsulated in an Ethernet frame. Ethernet Header IP Header TCP Header tSAS Header tSAS Data Ethernet Trailer TCP Segment IP Datagram Ethernet Frame 16 11/3/2011tSAS/sreddy

tSAS Approach  The above diagram shows a typical tSAS (SSP Write ) Request & Response sequence diagram.  iSCSI uses a similar approach where the back-end between the iSCSI target and SCSI drives uses the legacy SCSI protocol. tSAS Request tSAS Initiator tSAS Target SAS Expander Open Address Frame Open Accept SAS/SSP Request Frame Xfer Rdy Frame. Data Frame tSAS Xfer Rdy Frame. Data Frame 17 11/3/2011 tSAS/sreddy SSP Response Frame tSAS Response Frame

 Primitives −Primitives are special 8b/10b encoded characters that are used as frame delimiters, for out of band signaling, control sequencing in SAS −Most primitives can be conveniently ignored on the tSAS side except a few such as Broadcast primitives used to initiate discovery −SAS primitive can be encapsulated in an Ethernet frame  Discovery −Discovery in tSAS will be similar to SAS and will be accomplished by sending Serial management protocol (SMP) commands over TCP to the initiators and expanders downstream to learn the topology. 18 tSAS Approach 11/3/2011tSAS/sreddy

11/3/2011tSAS/sreddy19 tSAS Approach SAS Primitives

20 SMP Request tSAS Client SMP Initiator on Expander SMP Target on Expander Open Address Frame Open Accept SMP Request SMP Response Close SMP Response tSAS Approach SMP Request & Response Sequence Diagram 11/3/2011tSAS/sreddy

Test benches used for performance calculations  tSAS and iSCSI Mock Application −The client application and server application run on two different Windows machines connected using a NetGear ProSafe Gigabit Switch  Intel Core 2 Duo CPU  2.53 GHz, 3.45 GB RAM  Intel 82567LM Gigabit Network Adapter −Client makes Read/Write requests and the server processes and responds to requests made by the Client 2111/3/2011tSAS/sreddy Windows Machine A (Running Client App) Windows Machine B (Running Server App) NetGear Switch

Test benches used for performance calculations  Legacy SAS −A 6 Gbps SAS Host Bus Adapter in a PCIe slot of a Super Micro server −A 6 Gbps SAS target (Seagate SAS drive) connected to the Host Bus Adapter −IOMeter in Windows and VDBench in RHEL used to generate Read/Write IOs and measure performance  IOMeter and VDBench are IO generator tools used to measure IO performance of a storage system −A LeCroy SAS Protocol Analyzer placed between the HBA and the SAS Target 11/3/2011tSAS/sreddy22

11/3/2011tSAS/sreddy23 - SAS DISK DRIVE - EXPANDER Driver SAS Protocol Analyzer Test benches used for performance calculations

11/3/2011 tSAS/sreddy 24 Test benches used for performance calculations  Legacy iSCSI −Two windows machines/servers were used connected using a NetGear ProSafe Gigabit Switch −On one machine the StarWind iSCSi Initiator was running −On another machine, the KernSafe iSCSi target software was used to create a iSCSI target −IOMeter was used to send Read/Write requests from the iSCSI Initiator to the iSCSI Target and measure performance −Wireshark used to capture Network Traffic iSCSI Initiator on Windows Machine A iSCSI Target on Windows Machine B NetGear Switch

tSAS vs iSCSI Mock Application at 10 Mbps 2511/3/2011tSAS/sreddy

tSAS vs iSCSI Mock Application at 100 Mbps 2611/3/2011tSAS/sreddy

tSAS vs iSCSI Mock Application at 1 Gbps 2711/3/2011tSAS/sreddy

tSAS 512K Read at 10 Mbps, 100 Mbps and 1 Gbps 2811/3/2011tSAS/sreddy

Lessons Learnt/Difficulties encountered  Using several iSCSI Target Softwares that were not licensed/purchased made it challenging since several features were either not supported or did not work on the unlicensed version.  Even using KernSafe iSCSI target, I was not able to create a iSCSI RamDisk Target or a iSCSI Hard disk target. I used a USB Flash drive as the iSCSI target −Subtracted the delay at the Flash Drive from the iSCSI IOMeter performance results −Delay of Flash Drive calculated by running IOMeter to the SCSI device on the target machine 2911/3/2011 tSAS/sreddy

 Jumbo frames  The NetGear ProSafe switch is a Gigabit switch that supports Jumbo Frames. However, I could not get jumbo frames to work since the Network Adapter on the Windows machines did not support Jumbo frames  One needs cabling and adapters as well that support Jumbo frames to use a switch that supports Jumbo frames  Understanding the SAS protocol and tools is very involved −Took substantial time to understand and learn the workings of SAS as well as the tools 11/3/2011tSAS/sreddy30 Lessons Learnt/Difficulties encountered

Future Work  More data points −The tSAS mock application can be run using a faster switch with connection rates greater than 1 Gbps to get more data points (40G/100G Ethernet)  Piggybacking −Response frames are piggybacked with the last DATA frame or a DATA frame is piggy backed with a request frame. This would increase performance  Jumbo frames −Can be used to increase the amount of DATA that is passed from the initiator and target per Ethernet packet improving the performance results.  The Storage Associations can be motivated with similar work to fuel the conception of a tSAS specification  tSAS code can be implemented in a SAS HBA and performance can be measured using this direct implementation 3111/3/2011tSAS/sreddy

Conclusion  tSAS is a viable solution  No conclusions can be made regarding latency of tSAS compared to iSCSI. However at 1 Gbps, tSAS performs faster than iSCSI on WRITEs  tSAS more reliable at smaller transfer sizes at 10 Mbps, 100 Mbps and 1 Gbps due to higher TCP retransmission rate at larger sizes  Overhead in tSAS smaller than in iSCSI  Request, Response and Transfer Ready frames in tSAS smaller than Request, Response and Ready to Transfer frames in iSCSI  tSAS will overcome the distance limitation of legacy SAS  Back-end of tSAS will be a SAS/SATA drive which are faster 32 11/3/2011tSAS/sreddy

References [1] T10/1760-D Information Technology – Serial Attached SCSI – 2 (SAS-2), T10, 18 April 2009, Available from [2] Harry Mason, Serial attached SCSI Establishes its Position in the Enterprise, LSI Corporation, available from [3] J Satran, K Meth, C. Sapuntzakis, M. Chadalapka, E. Zeidner, RFC 3720 Internet Small Computer Systems Interface (iSCSI) Specification IETF, April 2004, available from [4] Cai, Y.; Fang, L.; Ratemo, R.; Liu, J.; Gross, K.; Kozma, M.; A test case for 3Gbps serial attached SCSI (SAS) Test Conference, Proceedings. ITC IEEE International, February 2006, available from /3/2011tSAS/sreddy

References [6] Kalmath Meth, Julian Satran, Design of the iSCSI Protocol, Mass Storage Systems and Technologies, (MSST 2003). Proceedings. 20th Mass Storage Systems and Technologies, (MSST 2003). Proceedings. 20th IEEE/11th NASA Goddard Conference onIEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, April 2003, available from [7] Stephen Aiken, Dirk Grunwald, Andrew R. Pleszkun, Jesse Willeke, A Performance Analysis of the iSCSI Protocol 20 th IEEE/11 th NASA Goddard Conference on Mass Storage Systems and Technologies, 2003, available from [8] M. Rajagopal, E. Rodriguez, R. Weber, RFC 3821 Fibre Channel over TCP/IP (FCIP) standard, IETF, July 2004, available from [9] BLi Bigang, Shu Jiwu, Zheng Weimin, SCSI Target Simulator Based on FC and IP Protocols in TH-MSNS * Department of Computer Science and Technology, Tsinghua University, Beijing China, /3/2011tSAS/sreddy

References [10] S. Chaitanya, K. Butler, A. Sivasubramaniam, P. McDaniel, M. Vilayannur, Design, Implementation and Evaluation of Security in iSCSI-based Network Storage Systems, StorageSS '06 Proceedings of the second ACM workshop on Storage security and survivability, October 2006, available from [11] 1Gbps and 10Gbps Ethernet Server Scalability, NetApp, available fromhttp://partners.netapp.com/go/techontap/matl/downloads/redhat- neterion_10g.pdf [12] Michael A. Ko, LAYERING SERIAL ATTACHED SMALL COMPUTER SYSTEM INTERFACE (SAS) vOVER ETHERNET, United States Patent Application , 09/18/2008 available from [13] SCSI Primary Commands - 4 (SPC-4), Revision 31, T10, 2011/06/13, available from [14] Broadcom iSCSI HBA FAQ available from /3/2011tSAS/sreddy