1. Object-based Storage Long Liu 2010-10-23

2. Outline Why do we need object based storage? What is object based storage? How to take advantage of it? What's the status of object based storage? What can we do about it?

3. Why do we need object based storage? What is object based storage? How to take advantage of it? What's the status of object based storage? What can we do about it? Outline

4. Background Existing enterprise storage infrastructures are feeling the strain the volume of data generated by many network-based applications continues to escalate

5. Background Two Technologies: Files: – Flexible data sharing – Secure Blocks: – High performance – Scalable Blocks Files

6. Comparison Block Based Disk Object Based Disk Operations: Read block Write block Addressing: Block range Operations: Create object Delete object Read object Write object Addressing: [object, byte range] Objects can be regarded as the convergence of two technologies: files and blocks

7. Comparison Object Interface File System Storage component Applications Logical Block Interface File System Traditional StorageObject-based Storage Hard Drive Object-based Storage Device (OSD)

8. Comparison File System Storage Component Block I/O Manager Storage Device CPU Applications System Call Interface File System User Component File System Storage Component Block I/O Manager Storage Device CPU Applications System Call Interface File System User Component Object Interface Block Interface Two changes :  Object-based storage offloads the storage component to the storage device  The device interface changes from blocks to objects (a) Traditional model (b) Object storage model

9. Motivation Objects Improved device and data sharing – Platform-dependent metadata moved to device Improved scalability & security – Devices directly handle client requests – Object security Improved performance – Data types can be differentiated at the device Improved storage management – Self-managed, policy-driven storage – Storage devices become more autonomous

10. Outline Why do we need object based storage? What is object based storage? How to take advantage of it? What's the status of object based storage? What can we do about it?

11. Object-based Storage  Object  OSD(Object-based Storage Device)  MDS(Metadata Server)

12. Object-based Storage  Object  OSD(Object-based Storage Device)  MDS(Metadata Server)

13. Object An object is a logical unit of storage —ID (Identification) —Application data — Metadata which includes block allocation and length —Attributes that is accessible by users Objects have file-like methods — open, close, read, write

14. Object The root object -- The OSD itself User object -- Created by SCSI commands from the application or client Collection object -- A group of user objects, such as all.mp3 Partition object -- Containers that share common security and space management characteristics

15. Object P1 P2 P3 P4 User Data Attributes Object ID Metadata U1 User Objects (for user data) Collection Objects Partition Objects Root Object (one per device) OSD

16. Object-based Storage  Object  OSD(Object-based Storage Device)  MDS(Metadata Server)

17. Object Storage Devices Intelligence with its own CPU, Memory, Disk, Network Redistribution with specific strategy Objects’ metadata management

18. File System Storage Component Block I/O Manager Storage Device CPU Applications System Call Interface File System User Component File System Storage Component Block I/O Manager Storage Device CPU Applications System Call Interface File System User Component Object Interface Interface Two changes :  Object-based storage offloads the storage component to the storage device  The device interface changes from blocks to objects (a) Traditional model (b) Object storage model Object Storage Devices

19. Expect wide variety of Object Storage Devices Disk array subsystem Stores up to 5 TBs per shelf 2 SATA disks – 240/500 GBHighly integrated, single disk 4 Gbps per shelf to cluster Orchestrates system activity Balances objects across OSDs

20. Object-based Storage  Object  OSD(Object-based Storage Device)  MDS(Metadata Server)

21. Metadata Server Control the interaction between clients and OSDs

22. Client Metadata Servers (MDS) File System Applications Storage component Object-based Storage Device (OSD) Metadata Manager Object Storage Devices (OSDs) Client Dataflow of Metadata

23. Outline Why do we need object based storage? What is object based storage? How to take advantage of it? What's the status of object based storage? What can we do about it?

24. T10 OSD Commands(face) CREATE/REMOVE READ/WRITE GET/SET ATTR Object 10001110101 10000001110 11001110111 10001111000..… User Data Opaque attributes (stored only) Shared attributes (stored & processed) Attribute pages OSD Target Interface OSD Client OSD Interface

25. OSD Commands(Interface) Basic Protocol – READ – WRITE – CREATE – REMOVE – GET ATTR – SET ATTR Specialized – APPEND – CREATE & WRITE – FLUSH – LIST Security – Authorization – Integrity – SET KEY – SET MASTER KEY Groups – CREATE COLLECTION – REMOVE COLLECTION – LIST COLLECTION Management – CREATE PARTITION – REMOVE PARTITION – FLUSH PARTITION – PERFORM SCSI COMMAND – PERFORM TASK MGMT very basic space mgmt attributes shared secrets opaque internal shared

26. Storage Technology Today Direct attached storage (DAS) Fabric Attached Storage (FAS) -Network Attached Storage (NAS) -Storage Area Networks (SAN) 1234

27. Direct Attached Storage BACKUP SERVER TAPE RAID Windows UNIX LAN Windows A traditional Direct Attached Storage model

28. Fabric Attached Storage CLIENTS SERVER DATA C/S Fabric Attached Storage

29. Network Attached Storage Clients File I/O IP network File server Block I/O Storage area network Block storage This figure illustrates NAS being used to share files among a number of clients. The files themselves may be stored on a fast SAN

30. Storage Area Networks Clients Metadata Data Block-based storage devices Storage area network Management Servers This figure illustrates a SAN file system The files themselves are stored on a fast storage Area to which the clients are also attached.

31. Security NETWORK Management METADATA SERVERS Attribute Capability CLIENTS Data Capability Attribute OBJECT-BASED STORAGE DEVICE

32. Outline Why do we need object based storage? What is object based storage? How to take advantage of it? What's the status of object based storage? What can we do about it?

33. Status Industrial – Lustre – Panasas Academic

34. Status Industrial – Lustre – Panasas Academic

35. Lustre First open sourced system with object storage High-performance parallel file system Consist of clients, MDS and OST(Object Storage Targets)

36. Lustre NETWORK Metadata Data Management Metadata METADATA SERVER CLUSTER CLIENTS OBJECT STORAGE TARGETS(OST)

37. Panasas High-performance file system Consist of OSD, Panasas File System, MDS

38. Panasas KeyObjectStorageFeatures Key Object Storage Features Intelligent space management in storage layer ß Media geometry aware placement ß Data aware prefetching, caching & recovery Encapsulation of data and attributes ß Native object interface, good programming model ß Storage interpreted attributes for per file properties KeyObjectStorageAdvantages Key Object Storage Advantages ß Robust, shared access by many clients ß Scalable performance via an offloaded data path ß Strong fine-grained end-to-end security

39. Panasas Clients are from Energy, Government, Finance, Manufacturing and Higher Education

40. Status Industrial – Lustre – Panasas Academic

41. A Design of Metadata Server Cluster In Large Distributed Object-based Storage Motivation: Metadata server cluster maybe the bottleneck Frequent metadata access and movement Terrible load balance management

42. VoD Server Web Server Database Server E-mail Server File Server Storage Network (Fibre Channel) Data MDS Cluster Security Metadata Application Server Cluster Object-based Storage Device Cluster Object-based Storage System Architecture A Design of Metadata Server Cluster In Large Distributed Object-based Storage

43. Application Servers Application File Hashing Manager Mapping Manager Metadata Server Cluster Hashing Partition Logical Partition Manager Metadata Server Backend Common Storage Space Figure 3. Hashing Partition A Design of Metadata Server Cluster In Large Distributed Object-based Storage Hashing Partition: A total solution for –File hashing –Metadata partitioning –Metadata storage

44. A Design of Metadata Server Cluster In Large Distributed Object-based Storage Pathname: /Dir1/Dir2/filename Mapping Manager Pathname Hashing Result (i) Pathname Metadata & etc Pathname Hashing Result (i+1) Pathname Metadata & etc Metadata Server Cluster Logical Partitions 14 2 3 Figure 4. Metadata Access Pattern ①.Filename hashing ②.Selecting MDS through Mapping Manager ③.Accessing metadata by pathname hashing result ④.Returning metadata to application server

45. Hashing Partition Mapping Manager 2 1 3 Metadata Server Cluster Logical Partitions Common Storage Space 4 Figure 5. MDS cluster failover procedure A Design of Metadata Server Cluster In Large Distributed Object-based Storage

46. Outline Why do we need object based storage? What is object based storage? How to take advantage of it? What's the status of object based storage? What can we do about it?

47. From Our Perspective

48. The End Thank you