1. Flexibility, Manageability and Performance in a Grid Storage Appliance John Bent, Venkateshwaran Venkataramani, Nick Leroy, Alain Roy, Joseph Stanley, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, and Miron Livny University of Wisconsin

2. Two trends D a t a s e t s P e r f o rm a n c e Storage appliances address both trends.

3. Storage Appliances: + and - Storage Appliances: Great for basic file service Easy to manage: Plug in and it works Good performance: Specialized just for I/O Reliable and available too Storage Appliances for the Grid: Mismatch? Inflexible: Few, specific protocols (e.g., NFS) Costly: 10x the cost of PC + a few disks Difficult to integrate: Just one piece of the puzzle

4. A Solution: NeST NeST: A Storage Appliance for the Grid Flexible: Multiple simultaneous protocols Virtual protocol layer Low-cost: Use commodity machines Dynamic adaptation Grid-aware: Integrate w/ higher-level systems Design specifically for the Grid

5. Outline Introduction General architecture Design goals Flexibility Low-cost Grid-aware features NeST in the Grid example Conclusions

6. Physical storage layer ChirpHTTPGrid FTPNFS Common protocol layer Dispatcher Transfer Mgr Concurrency Models Storage Mgr Control flow Datal flow NeST: Protocol layer Physical network layer Virtualizes different protocols Mediates access to network

7. Physical storage layer ChirpHTTPGrid FTPNFS Common protocol layer Dispatcher Transfer Mgr Concurrency Models Storage Mgr Control flow Datal flow NeST: Dispatcher Physical network layer Mediates interaction between other components Gathers information, advertises

8. Physical storage layer ChirpHTTPGrid FTPNFS Common protocol layer Dispatcher Transfer Mgr Concurrency Models Storage Mgr Control flow Datal flow NeST: Storage manager Physical network layer Space management Access control Virtualizes physical storage

9. Physical storage layer ChirpHTTPGrid FTPNFS Common protocol layer Dispatcher Transfer Mgr Concurrency Models Storage Mgr Control flow Datal flow NeST: Transfer manager Physical network layer Implementss cheduling policies Chooses concurrency model

10. Outline Introduction General architecture Design goals Flexibility Low-cost Grid-aware features NeST in the Grid example Conclusions

11. Flexibility: Multiple protocols Problem: How to support multiple protocols? One approach: Just a Bunch of Servers (JBOS) Problems with JBOS Lack of control (scheduling) Painful administration No shared code Larger memory footprint nfsd wu-ftpd httpd JBOS Server

12. NeST: Flexibility By Design NeST: Integrate protocols and gain advantage Implementation like VFS Integration introduces new challenges Different protocols allow different auth models More expensive to add a new protocol Less fault isolation

13. NeST vs JBOS Chirp GridFTP HTTP NFS 0 5 10 15 20 25 30 35 Server bandwidth (MB/s) 0 5 10 15 20 25 30 35 Total For each protocol, NeST is comparable to JBOS server. wu-ftpd Apache linux nfsd Linux cluster - Dual PIII - 1 GB Ram - linux 2.2.19 Each protocol - 4 clients - 10 MB files

14. Exerting scheduling control Different scheduling policies FCFS Cache-aware [USENIX ‘02] Proportional share Proportional share scheduling Allows administrators to set protocol proportions e.g. favor NFS Very difficult in JBOS

15. Proportional share 0 5 10 15 20 25 30 35 Server bandwidth (MB/s) FCFS1:1:1:11:2:1:11:1:1:4 Scheduling configuration In most cases, achieves Jain’s metric of fairness > 0.98 (1 is “fair”). Linux cluster - Dual PIII - 1 GB Ram - linux 2.2.19 Each protocol - 4 clients - 10 MB files

16. Outline Introduction General architecture Design goals Flexibility Low-cost Grid-aware features NeST in the Grid example Conclusions

17. Low-Cost: New challenges Desire: Run on arbitrary OS on arbitrary PC Software-only, user-level storage appliance Currently on Linux (release 0.9) and Solaris (beta) Problem: Portable performance Performance under load is platform / workload dependent Threads or processes on some systems, events on others May also be workload dependent (e.g. whether in cache) NeST approach: Dynamic adaptivity Simultaneously support multiple concurrency models Monitor performance using each model Bias towards better model over time

18. Adaptive Concurrency 0 1 2 3 Solaris: 1K files Ave time per request (sec) Events Threads Linux: 10 MB files Events Threads 0 1 0.5 1.5 Ave time per request (ms) Adaptive Dynamic adaptation approaches “ideal” without static information.

19. Outline Introduction General architecture Design goals Flexibility Low-cost Grid-aware features NeST in the Grid example Conclusions

20. Grid-Aware Mechanisms Basic functionality Users and groups: Dynamic creation/deletion does not need administrative intervention Access control: Generic AFS-style ACLs Advanced functionality QoS: Preferential scheduling Advertises into global scheduling systems Flexible protocol and authentication mechanisms Self-cleaning storage guarantees: Lots

21. Storage guarantees: Lots Characteristics of Lots: Capacity: Total amount of data lot can store Duration: Time for which data is guaranteed to exist Set of files: Multiple files may co-exist within lot Self-cleaning Expired lots become “best-effort” lots Lot management Either default set created by administrator, OR use resource management protocol to create before usage Implementation: File system quotas Advantage: Integrates cleanly with local access methods Disadvantage: Performance hit for large writes

22. Outline Introduction General architecture Design goals Flexibility Low-cost Grid-aware features NeST in the Grid example Conclusions

23. NeST in the Grid Linux NeST Solaris NeST HomeRemote Global Execution Manager compute node compute node compute node compute node Condor Tribune Advertisement

24. NeST in the Grid N N Home Remote Global Execution Mgr GridFTP 3) 3) Global coordinates xfer 1) 1) Home submits jobs 6) 6) Global terminates space 5) 5) Global coordinates xfer NFS 4) 4) Global starts jobs 2) 2) Global reserves space

25. Conclusions NeST: A storage appliance for the Grid Gain manageability Without sacrificing performance Design goals: Flexibility: Virtual protocol architecture Low-cost: Adaptation mechanisms Grid-aware: Space management Current status: release 0.9 available Future work Hot deployable NeSTs, lot management extensions

26. Questions? http://www.cs.wisc.edu/condor/nest

27. NeST: Architecture Common Protocol Layer GridFTPNFSHTTPChirp Dispatcher Storage Manager FTP The Network Physical Storage Transfer Manager

28. Example client interaction Common Protocol Layer GFTPNFSHTTPChirp Dispatcher Storage Manager FTP The Network Physical Storage Transfer Manager Client sends mkdir Dispatcher asks protocol handler to parse asks storage mgr to mkdir checks permissions sends ack through handler Client puts file in new dir Dispatcher asks protocol handler to parse asks storage mgr for permission hands off to transfer mgr Transfer manager monitors transfer sends final ack