1 Object-Based Network Storage Systems Shang Rong Tsai DSLab Institute of Computer and Communication Department of Electrical Engineering National Cheng-Kung University

2 Outlines Background and Introduction Network Storage/SAN/NAS Object-Based Storage Epilogue

3 The Epoch of Data Explosion Electronic data are continuously growing Data and storage managements are becoming more and more important –Continuous service and data availability –Data backup and restore – Storage and data sharing –Efficiency in data and storage management Expecting using less man power Flexibility in system configuration Efficiently expanding storage capacity

4 Network Storage Systems Storages are moving to networks for sharing and efficient management Such demands push the emerge of SAN (Storage Area Network) NAS (Network Attached Storage) has been used for longer time Object-Based Storage System –Both SAN and NAS have their own merits and drawbacks. –SAN supports data access at block level, not good for applications in data sharing –NAS imposes bottleneck on servers and has bad scalability –Object-Based approach potentially eliminates their drawbacks Network storage technologies offer a new platform for networking and storage people to play a new game.

5 Conceptual Model of SAN client Disk tape LinuxWindowsSunOS SAN to connect hosts and storages Network to connect clients and hosts Hosts Clients Storages

6 What is a SAN SAN is a high speed network (traditionally Fiber Channel) to connect storages to servers (hosts). The network is basically used as a replacement of storage bus in traditional shared bus storage systems, thus enhances the possible sharing scale and possible fail over, etc. Access to SAN storage is at block level

7 Why SANs (1/2) From the communication aspect, the SAN can bypass the possible communication bottleneck. It enables communication between –Server-to-server –Server-to-storage (typical model) –Storage-to-storage (e.g. for backup without servers’ intervention)

8 Why SANs (2/2) Improvements to application availability: Storage is accessible through multiple data paths for better reliability, availability, and serviceability. Higher application performance: Storage processing is off-loaded from servers Virtualized storage: Storages on SAN can be flexibly configured as a logical volume of any size, thus easily sharing storage space at configuration Data backup to remote sites: enabled for disaster protection Simplified centralized management: Single image of storage systems simplifies management.

9 Storage Virtualization (1/2) The key technology of SAN SNIA defines storage virtualization as: “The act of integrating one or more (back end) services or functions with additional (front end) functionality for the purpose of providing useful abstractions. Typically, virtualization hides some of the back-end complexity, or adds or integrates new functionality with existing back end services. Examples of virtualization are the aggregation of multiple instances of a service into one virtualized service, or to add security to an otherwise insecure service. Virtualization can be nested or applied to multiple layers of a system.”

10 Storage Virtualization (2/2) To be more practical, storage virtualization is the aggregation of physical storage from multiple network storage devices into a single logical storage device that is managed and used by a central host. Logical Volume Manager (LVM) is basically the concept of storage virtualization which has been used for many years. By storage virtualization we can easily and flexibly configure the size of a logical disk. For some applications or sites, the amount of storage required grows at unprecedented rates. Try to think about if a disk partition is becoming full.

11 SAN connectivity Traditionally, SAN used Fibre Channel technology to implement the storage networks Fibre Channel SANs support high bandwidth storage traffic at 200 MB/s and enhancements to 10 Gb/s in the near future. This will be mostly used for inter switch links (ISL) between switches. iSCSI SAN (SCSI over TCP/IP) is a relatively new approach for storage networks. Fibre Channel over IP (FCIP) FCIP can be used as WAN bridging transport for both FC and iSCSI based campus SANs

12 iSCSI iSCSI stands for Internet Small Computer System Interconnect iSCSI is a protocol for encapsulating SCSI commands on a TCP/IP network The iSCSI protocol enables universal access to storage devices and Storage Area Networks over TCP/IP network

13 SCSI Protocol Layers SCSI command layer –Generic commands (for all devices) –Device specific commands Transport Layer Physical Layer (Connectivity Layer)

14 SCSI application (e.g. File systems) SCSI block commandsSCSI stream commands SCSI commands for other types of devices SCSI Generic (Primary) commands SCSI commands layer SCSI transport layer Physical layer iSCSI over TCP/IP TCP IP Layer 2 (Ethernet) Parallel SCSISCSI over Fiber Channel Fiber Channel Parallel SCSI Bus SCSI Protocol Layer

15 iSCSI PDU (from SNIA)

16 Overview of iSCSI iSCSI provides initiators and targets with unique names and a discovery method. The iSCSI protocol establishes communication sessions between initiators and targets, and provides methods for them to authenticate one another. An iSCSI session may contain one or more TCP connections and provides recovery in case of connection failures. SCSI CDBs (Command Descriptor Block) are passed from the SCSI command layer to the iSCSI transport layer. The iSCSI transport layer encapsulates the SCSI CDB into an iSCSI Protocol Data Unit (PDU) and forwards it to the Transmission Control Protocol (TCP) layer. iSCSI provides the SCSI command layer with a reliable transport.

17 What is NAS Network Attached Storage (NAS) is basically a LAN attached file server that provides shared file access using a file sharing protocol such as Network File System (NFS) or Common Internet File System (CIFS) Access to NAS is at file level.

18 Why NAS NAS technology has been used for decades to share files, thus to save storage space and to keep data consistency. (in contrast to file copy, like FTP) Data sharing in NAS is at file level, which matches the semantic for applications. In contrast, sharing in SAN is at block level which is very difficult (if not impossible) for applications to R/W data sharing. (Applications recognize files, not blocks)

19 What is OSD (from Erik Riedel Seagate) The Object-based Storage Device interface standard is focused on moving chosen low- level storage, space management, and security functions into storage devices(disks, subsystems, appliances) to enable the creation of scalable, self-managed, protected and heterogeneous shared storage for storage networks.

20 What is an Object-Based Storage Storage devices that operate at object-level Traditional storage devices (Such as DAS, SAN) operate at block-level Objects are typically files which match the semantic level that applications manipulate data. In traditional systems, files are mapped to blocks before they are stored in storages.

21 Why Object-Based Storage (1/2) Drawbacks for NAS –Most of the processing on file access are on the file server => Poor scalability –Difficult to distribute users’ files to multiple servers while preserve a single global file namespace. We may delegate the management of the subset of the whole file namespace to a file server, however, the load distributed to each server may be very uneven.

22 Why Object-Based Storage (2/2) Drawbacks of SAN –SANs operate at block-level, so sharing data between applications may still need upper layer software. file read/write sharing Record/file locking Object-Based Storage can potentially eliminate both the drawbacks of NAS and SAN

23 The Value of Objects (from SNIA) Better security via capabilities –Each object can have its own security domain –All I/O is authorized by the device Easier to share data –Files and records can be stored as objects –Low-level metadata managed by device Opportunities for intelligence –Attribute-based learning for resource allocation Better caching, pre-fetching and staging of data –Self-configuring storage w/ continuous reorganization Layout objects to best serve client requests

24 Two Basic Components in File Service Directory Service –Providing the global file name space visible to users and applications –Mapping file pathname to unique file id –May need the flat file system to access directories Flat File System –Given the file id, returning the file contents –Storing file attributes –Managing file allocation on disks

25 Typical Handling in File Access File pathname lookup : open /a/b/c –Symbolic file name => file id (inode in Unix) –For each file path component, get the file id, get the file contents until getting file id of the target Get the attributes (including the location of the file on disks) of the file Ready for W/R

26 How to Restructure the file system to fit the Object-Based Storage Architecture ? We may (as Lustre did) partition the functions into –MDS (Meta Data Server) –OSD (Object-Storage Device) What is the total system architecture? What are the data and management functions delegated to MDS? What are the data and management functions delegated to OSDs? Important observation: Typically, among the file processing in systems, 90% processing is by OSDs, 10% by MDS => To balance the loads

27 Traditional structure vs. Object- Based structure Diagram taken from SNIA T10 standard document

28 Lustre A system developed at CMU, a very early working system demonstrate the concept of object-based storage systems. Modifying the Linux kernel A few number of MDSs A large number of OSDs Majorly targeting at the applications in large scale computing. –Large number of users –File service for high performance computing Good Scalability

29 OST 1 OST 2 OST 7 OST 3 OST 6 OST 5 OST 4 GigE QSW Net Lustre Clients (1,000 Lustre Lite) Up to 10,000’s MDS 1 (active) MDS 2 (failover) Lustre Object Storage Targets (OST) Linux OST Servers with disk arrays 3 rd party OST Appliances SAN Lustre Architecture (from CFS Inc.)

30 Lustre Components and Functions (1/2) Basic Components –Client Filesystems Interfacing to local file system (VFS) on clients As MDS client As OST client –MetaData Servers All the meta-data operations - creating new directories, files, symbolic links, or acquiring and updating inodes, are handled by the MDS. –Object Storage Target All the file I/O related operations are directed to the OST’s.

31 Lustre Components and Functions (2/2) The role of the client filesystem is to provide a directory tree, subdivided into filesets, which provides cluster-wide Unix file sharing semantics. The client filesystem interacts with the meta-data servers for meta-data handling, i.e. for the acquisition and updates of inodes and directory information. File I/O, including the allocation of blocks, striping, and security enforcement, is contained in the protocol between the client filesystem and the object storage targets. A third protocol exists between the OST and the MDS, largely for pre-allocation and recovery purposes.

32 SNIA T10 Specification A specification developed for Object-based Disk The SCSI command set defined to provide efficient peer-to-peer operation of input/output logical units that manage the allocation, placement, and accessing of variable-size data-storage containers, called objects.

33 Some Operations defined in T10 Format OSD -defining OSD structure on device Create Object Group -defining a set in which to create objects Create object -creating an object, returning the object ID fid Read (fid,starting byte,length) Write (fid,starting byte,length) Get attributes of an object

34 Possible Intelligent functions OSDs serve and manage data at object (most often representing files) level. This enables OSDs to smartly enforce management on a per-object basis –Automatic replication/backup –QoS –Caching/Prefetching –Optimal layout –Preprocessing/Postpressing (such as data compression/decompression, data filtering)

35 How to support the intelligent functions ? Extensible file attributes would be a good way Users can define their own file attributes. A special ‘code’ attribute can be assigned to a particular file. Setting the code attribute to a file is in effect installing the code to the OSD to execute the intelligent functions within the OSD. How to set the code attribute in practice ? –Consideration for platform (OSD) dependence –Sharing the code on OSD How to command the OSDs to execute the intelligent functions automatically

36 Epilogue Network Storage Technologies get more attention by IT industry for data sharing and effective data management for huge and dynamically growing amount of data. SAN and NAS have their roles to play, currently the major approaches for network storages. SAN suffers the difficulty in direct data sharing. NAS suffers the difficulty in system scalability Object-Based Storage emerges as a new approach to solve both problems. Object-Based Storage offers new opportunities for intelligent storage devices which potentially derive many research topics