Small-World File-Sharing Communities Adriana Iamnitchi, Matei Ripeanu and Ian Foster,

Selecting tradeoffs Understand users’ requirements … AND users’ behavior –Patterns exist and can be exploited: Zipf request popularity distribution (web caching) Breslau, Cao, Fan, Phillips, and Shenker. Web caching and Zipf-like distributions: Evidence and implications. Infocom, 1999 Power-law node degree distribution in graphs (search) Adamic, Lukose, Puniyani, Huberman. Search in Power-law Networks. In Physical Review E, , 2001

“No 24 in B minor, BWV 869” “Les Bonbons” “ Yellow Submarine” “Les Bonbons” “Yellow Submarine” “Wood Is a Pleasant Thing to Think About” “Wood Is a Pleasant Thing to Think About” System view: The Data-Sharing Graph G m T (V, E):  V is set of users active during interval T  An edge in E connects users that asked for at least m common files within T

Three Data-Sharing Graphs The DØ Experiment –High-energy physics data Grid –70+ institutions, 18 countries The Web: Boeing traces –Web requests from Boeing sites to the Web The KaZaA music-sharing network –Peer-to-peer system UsersRequestsDuration traces D0 3172,757, days Web 60,82616,527,19410 hours Kazaa 14,404976,1845 days

Small average path length Large clustering coefficient The DØ Collaboration Small World! CCoef = # Existing Edges # Possible Edges 6 months of traces (January – June 2002) 300+ users, 2 million requests for 200K files

Small-World Graphs Small path length, large clustering coefficient –Typically compared against random graphs Think of: –“It’s a small world!” –“Six degrees of separation” Milgram’s experiments in the 60s

Other Small Worlds Word co-occurrences Film actors LANL coauthors Internet Web Food web Power grid D. J. Watts and S. H. Strogatz, Collective dynamics of small-world networks. Nature, 393: , 1998 R. Albert and A.-L. Barabási, Statistical mechanics of complex networks, R. Modern Physics 74, 47 (2002).

Web Data-Sharing Graphs 7200s, 50files 3600s, 50files 1800s, 100files 1800s, 10file 300s, 1file

DØ Data-Sharing Graphs 7days, 1file 28 days, 1 file

KaZaA Data-Sharing Graphs 7day, 1file 28 days 1 file 2 hours 1 file 1 day 2 files 4h 2 files 12h 4 files

Motivation : Understand patterns (user behavior) to choose the right tradeoffs Understand what?! –The data-sharing graph Data-sharing communities –The D0 Experiment (physicists in a data grid) –Web –Kazaa It’s a small world! So what? –News or consequence of known patterns? –Relevance for mechanism design Outline

Exploiting Patterns: Intuition  Exploit the data-sharing graph (as overlay) Exploit its small-world properties –Large clustering coefficient: optimize existing (e.g., centralized) mechanisms –… and small average path length: design a new file location mechanism D0 Web

Information Dissemination: Local Answers D0 Web Kazaa Hit rate due to previous information dissemination within clusters (experience sharing): up to 70% of requests will find answers locally

The Message Be imaginative when looking for patterns There are small-world patterns in file-sharing communities …that can be exploited for mechanism design –Location of files –Data placement