1. PARALLEL DATA LABORATORY Carnegie Mellon University An Architecture for Self-  Storage Systems Andrew Klosterman, John Strunk Greg Ganger

2. Klosterman, Strunk, Ganger June 4, 2016 http://www.pdl.cmu.edu/2 Self-  Overview Object store Provide clients with object-based interface Aggregate object-based workers Support snapshot, clone, and versioning Easy to manage Goal-based & complaint-based tuning Problem diagnosis via history Automatic integration of new resources No “on-call” administrator

3. Klosterman, Strunk, Ganger June 4, 2016 http://www.pdl.cmu.edu/3 Maintenance & Fault-Tolerance Repairs necessary, but accident-prone Tolerate mistakes during repair Support simulated failures: fire-drills Keep maintenance procedures short Reduce number of “destructive” activities Time pressure causes mistakes No repairs required in less than 1 week!

4. Klosterman, Strunk, Ganger June 4, 2016 http://www.pdl.cmu.edu/4 System Deployment Single datacenter environment High bandwidth, tightly coupled How big can it be? Integrated is easier to manage Vertical: disk  file system Horizontal: large device vs. small ones Capacity: 16 PB with current technology Objects: 128 billion (128 KB object size)

5. Klosterman, Strunk, Ganger June 4, 2016 http://www.pdl.cmu.edu/5 System Architecture I/O request routing Administrator Clients Management hierarchy Workers

6. Klosterman, Strunk, Ganger June 4, 2016 http://www.pdl.cmu.edu/6 Admin Console Supervisor Worker Discovery, Group Membership and Directory Service Event Notification Service Security Service Metadata Service Head-end Interface Messaging LayerRouterRead / Write ProtocolEncode / Decode Administrator Management hierarchy Workers I/O request routing Clients Router

7. Klosterman, Strunk, Ganger June 4, 2016 http://www.pdl.cmu.edu/7 Management Hierarchy Admin Console System-wide monitoring Goal determination Goal distribution Complaint-based tuning Supervisor Monitoring Sub-systems Goals Performance tuning Fault detection Fault recovery

8. Klosterman, Strunk, Ganger June 4, 2016 http://www.pdl.cmu.edu/8 Worker Object-based storage device Objects: read / write Attributes: read / write Comprehensive versioning Fast-copy  clone() A copy-on-write object “Intelligent bricks” 1U, P4, 2 GB RAM, 2 Gb NICs 4 SATA 250 GB or 4 SCSI 73 GB

9. Klosterman, Strunk, Ganger June 4, 2016 http://www.pdl.cmu.edu/9 Request Routing #1 Discovery Detect new components Assign system ID Group Membership Aggregate components Service groups Directory Lookup service  “DNS” Query for contact info Event Notification post() subscribe() Situations trigger event posting Receipt of a subscribed message triggers reaction

10. Klosterman, Strunk, Ganger June 4, 2016 http://www.pdl.cmu.edu/10 Request Routing #2 Security Service Authenticity Token based Checked at Workers against ACLs Confidentiality: PASIS Encode / Decode Integrity: Messaging Layer Key management

11. Klosterman, Strunk, Ganger June 4, 2016 http://www.pdl.cmu.edu/11 Request Routing #2 Security Service

12. Klosterman, Strunk, Ganger June 4, 2016 http://www.pdl.cmu.edu/12 Request Routing #3 Metadata Service Gigantic B-tree of object metadata Goals Encoding Share locations Can be rebuilt from data on workers Supports enumeration for fsck

13. Klosterman, Strunk, Ganger June 4, 2016 http://www.pdl.cmu.edu/13 Head-end Interface Object-storage interface Supports additional calls Goal assignment Side-band performance tuning Two types of head-ends Translation: exports NFS, AFS, CIFS, etc. Direct: raw access to self-  objects Clients

14. Klosterman, Strunk, Ganger June 4, 2016 http://www.pdl.cmu.edu/14 Communication Infrastructure #1 Encode/Decode Encoding chosen to meet Goals Breaks objects into shares on write() Reconstructs objects from shares on read() Read / Write Protocol Atomic changes to ~64 kB chunks of objects

15. Klosterman, Strunk, Ganger June 4, 2016 http://www.pdl.cmu.edu/15 Communication Infrastructure #2 Router Decision maker Picks destination Distributed services Shares on read() Messaging Layer Forwards messages Picks network Interacts with Directory service

16. Klosterman, Strunk, Ganger June 4, 2016 http://www.pdl.cmu.edu/16 Admin Console Supervisor Worker Discovery, Group Membership and Directory Service Event Notification Service Security Service Metadata Service Head-end Interface Messaging LayerRouterRead / Write ProtocolEncode / Decode Administrator Management hierarchy Workers I/O request routing Clients