1. Mondo Bringing Distributed File Systems to the People Xavid Pretzer Bringing Distributed File Systems to the People Xavid Pretzer

2. Location-Independent Files What do we want to be able to do? Existing Strategies Distributed File Systems Peer-to-Peer File Sharing Mondo: a different approach What do we want to be able to do? Existing Strategies Distributed File Systems Peer-to-Peer File Sharing Mondo: a different approach

3. What’s the problem? Sharing files between home and work Use existing computers to help distribute data Distributing information through often-partitioned networks Sharing files between home and work Use existing computers to help distribute data Distributing information through often-partitioned networks

4. Distributed File Systems NFS emulates local access All peers trusted AFS uses trusted servers, untrusted clients Also introduced a global namespace Coda: disconnected operation NFS emulates local access All peers trusted AFS uses trusted servers, untrusted clients Also introduced a global namespace Coda: disconnected operation NFS AFS Coda

5. Why isn’t this enough? Need dedicated, trusted servers Servers can bottleneck file access Need more servers for more clients Extra client space not used Localized failures disrupt system Need dedicated, trusted servers Servers can bottleneck file access Need more servers for more clients Extra client space not used Localized failures disrupt system

6. Peer-to-peer File Sharing Composed of untrusted peers Popular files easier to obtain Gnutella: share files without centralized server Bittorrent: cooperate to share large files with low bandwidth Composed of untrusted peers Popular files easier to obtain Gnutella: share files without centralized server Bittorrent: cooperate to share large files with low bandwidth Gnutella Bittorrent Tracker Seeder

7. Limitations of File Sharing Limited access control and authentication No dynamic files No useful directory structure Difficult to ensure availability Limited access control and authentication No dynamic files No useful directory structure Difficult to ensure availability Seeder F F’ ? ?

8. Combining P2P with DFS Farsite: serverless Uses encryption, Byzantine protocols Trusts self-reporting Files delocalized Ivy: log-structured Logs changes to distributed hash table Must agree on which logs to trust Can recover from broken trust Farsite: serverless Uses encryption, Byzantine protocols Trusts self-reporting Files delocalized Ivy: log-structured Logs changes to distributed hash table Must agree on which logs to trust Can recover from broken trust Ivy Farsite /farsite /farsite/user 01 25 42 6a 8c b9 e1

9. Mondo: a different approach File data served both by primary hosts and caching clients File location info stored in a Distributed Hash Table Encryption and cryptographic signing used for file permissions No central authority needed Expandable to large, heterogeneous groups File data served both by primary hosts and caching clients File location info stored in a Distributed Hash Table Encryption and cryptographic signing used for file permissions No central authority needed Expandable to large, heterogeneous groups

10. Data Storage Primary hosts always keep designated files Clients cache used files and share with other peers Mutual exchanges for file replication File blocks requested in parallel from multiple peers Primary hosts always keep designated files Clients cache used files and share with other peers Mutual exchanges for file replication File blocks requested in parallel from multiple peers P H  1  2  3  4  H  1 2  3 4  H  1  2  3 4 3 3 1 4

11. Permissions without Trust File permissions signed by owner File data, version, block checksums signed by writer Read-restricted files encrypted with unique key Read key encrypted in header with readers’ public keys File permissions signed by owner File data, version, block checksums signed by writer Read-restricted files encrypted with unique key Read key encrypted in header with readers’ public keys Signed, Alice File 7ce5ab92 Owner: Alice Writers: Alice, Bob Alice’s Read Key: 523ea220 Bob’s Read Key: 9a45bc31 Charlie’s Read Key: efed3238 … Signed, Bob Version 2 File length: 3214 bytes Block size: 1024 bytes Block 1 checksum: a4b23ac4 Block 2 checksum: 8bed0123 Block 3 checksum: 76f3dc13 Encrypted File Data

12. Directories Directories are files and use permissions Directories store file ids for contents Also store public keys for file owners as a “web of trust” Directories can be multiply linked Directories are files and use permissions Directories store file ids for contents Also store public keys for file owners as a “web of trust” Directories can be multiply linked info.txt: 8cd349a3 Owned by Alice Signed, Alice Directory 5ab9217e … Signed, Bob photo.jpeg: 29468ecd Owned by Charles junk: ea2bc891 Owned by Joe Version 7 … Contents:

13. Locating Files Peers serve as a Distributed Hash Table Each peer tells DHT what files it stores Primary host labels signed by file owner Changes propagated to all primary hosts; clients update copies lazily Peers serve as a Distributed Hash Table Each peer tells DHT what files it stores Primary host labels signed by file owner Changes propagated to all primary hosts; clients update copies lazily Cached by 18.3.11.92 5ab9217e Signed, Alice Primary host 18.244.3.61 5ab9217e 01 25 42 6a 8c b9 e1

14. Handling Malicious Peers Refuse to store improperly signed files DHT pairs stored redundantly Block checksums verify file data Tit-for-tat data sharing Periodically verify mutual replication Refuse to store improperly signed files DHT pairs stored redundantly Block checksums verify file data Tit-for-tat data sharing Periodically verify mutual replication Signed, Alice File 7ce5ab92 Owner: Alice Writers: Alice, Bob … Signed, Joe … H  1  2  3 4  H  1’ 2’ 3’ 4’ 1

15. Example: Reading a file Start with a root directory id and its owner’s public key Look up that id in the DHT to find peers with that directory Retrieve it and verify with the key Read contents and repeat recursively Start with a root directory id and its owner’s public key Look up that id in the DHT to find peers with that directory Retrieve it and verify with the key Read contents and repeat recursively /mondo: 5ab9217e 18.244.3.61 /mondo/info.txt: 8cd349a3 DHT Contents: info.txt photo.jpeg

16. Potential Difficulties Syncing changes efficiently Variable TTL? Availability of unpopular files Bogus DHT announcements Changes in file ownership Merging parallel changes Syncing changes efficiently Variable TTL? Availability of unpopular files Bogus DHT announcements Changes in file ownership Merging parallel changes P Δ ? Signed, Alice File 7ce5ab92 Owner: Alice … Signed, ??? File 7ce5ab92 Owner: Bob … chown ???

17. Applications Location-independent files without dedicated servers Making better use of existing resources Sharing mutable files with a large audience Distributing files on failure-prone networks Location-independent files without dedicated servers Making better use of existing resources Sharing mutable files with a large audience Distributing files on failure-prone networks

18. What Mondo Provides Scalable and secure distribution of mutable files among untrusted computers Advantages of DFS without dedicated servers Web of trust in directory hierarchy Effective distribution over normally-partitioned networks Scalable and secure distribution of mutable files among untrusted computers Advantages of DFS without dedicated servers Web of trust in directory hierarchy Effective distribution over normally-partitioned networks

19. References Adya, Atul, William J. Bolosky, Miguel Castro, Gerald Cermak, Ronnie Chaiken, John R. Douceur, Jon Howell, Jacob R. Lorch, Marvin Theimer, and Roger P. Wattenhofer. “FARSITE: Federated, Available, and Reliable Storage for an Incompletely Trusted Environment”. 2002. Bolosky, William J., John R. Douceur, David Ely, and Marvin Theimer. “Feasibility of a Serverless Distributed File System Deployed on an Existing Set of Desktop PCs”. 2000. Cohen, Bram. “Incentives Build Robustness in BitTorrent”. 2003. Howard, John H., Michael L. Kazar, Sherri G. Menees, David A. Nichols, M. Satyanarayanan, Robert N. Sidebotham, and Michael J. West. “Scale and Performance in a Distributed File System”. 1988. Kon, Fabio. "Distributed File Systems Past, Present, and Future: A Distributed File System for 2006". 1996. Maymounkov, Petar and David Mazières. “Kademlia: A Peer-to-peer Information System Based on the XOR Metric”. 2002. Muthitacharoen, Athicha, Robert Morris, Thomer M. Gil, and Benjie Chen. “Ivy: A Read/Write Peer-to-Peer File System”. 2002. Stoica, Ion, Robert Morris, David Karger, M. Frans Kaashoek, and Hari Balakrishnan. “Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications”. 2001 Adya, Atul, William J. Bolosky, Miguel Castro, Gerald Cermak, Ronnie Chaiken, John R. Douceur, Jon Howell, Jacob R. Lorch, Marvin Theimer, and Roger P. Wattenhofer. “FARSITE: Federated, Available, and Reliable Storage for an Incompletely Trusted Environment”. 2002. Bolosky, William J., John R. Douceur, David Ely, and Marvin Theimer. “Feasibility of a Serverless Distributed File System Deployed on an Existing Set of Desktop PCs”. 2000. Cohen, Bram. “Incentives Build Robustness in BitTorrent”. 2003. Howard, John H., Michael L. Kazar, Sherri G. Menees, David A. Nichols, M. Satyanarayanan, Robert N. Sidebotham, and Michael J. West. “Scale and Performance in a Distributed File System”. 1988. Kon, Fabio. "Distributed File Systems Past, Present, and Future: A Distributed File System for 2006". 1996. Maymounkov, Petar and David Mazières. “Kademlia: A Peer-to-peer Information System Based on the XOR Metric”. 2002. Muthitacharoen, Athicha, Robert Morris, Thomer M. Gil, and Benjie Chen. “Ivy: A Read/Write Peer-to-Peer File System”. 2002. Stoica, Ion, Robert Morris, David Karger, M. Frans Kaashoek, and Hari Balakrishnan. “Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications”. 2001