Lada Adamic, HP Labs, Palo Alto, CA

Talk outline Information flow through blogs Information flow through Search through networks Search within the enterprise Search in an online community

Implicit Structure and Dynamics of BlogSpace Eytan Adar, Li Zhang, Lada Adamic, & Rajan Lukose Blog use: –Record real-world and virtual experiences –Note and discuss things “seen” on the net Blog structure: blog-to-blog linking Use + Structure –Great to track “memes” (catchy ideas)

Approaches and uses of blog analysis Patterns of information flow –How does the popularity of a topic evolve over time? –Who is getting information from whom? Ranking algorithms that take advantage of transmission patterns

Popularity Time Slashdot Effect BoingBoing Effect Tracking popularity over time Blogdex, BlogPulse, etc. track the most popular links/phrases of the day

Different kinds of information have different popularity profiles Products, etc. Major-news site (editorial content) – back of the paper % of hits received on each day since first appearance Slashdot postings Front-page news

Micro example: Giant Microbes

Microscale Dynamics What do we need track specific info ‘epidemics’? –Timings –Underlying network b1b1 b1b1 Time of infection t0t0 t1t1 b2b2 b2b2 b3b3 b3b3

Microscale Dynamics Challenges –Root may be unknown –Multiple possible paths –Uncrawled space, alternate media ( , voice) –No links b1b1 b1b1 Time of infection t0t0 t1t1 b2b2 b2b2 b3b3 b3b3 ? ? bnbn bnbn

Microscale Dynamics who is getting info from whom Explicit blog to blog links (easy) –Via links are even better Implicit/Inferred transfer (harder) –Use ML algorithm for link inference problem Support Vector Machine (SVM) Logistic Regression –What we can use Full text Blogs in common Links in common History of infection

Visualization Zoomgraph tool –Using GraphViz (by AT&T) layouts Simple algorithm –If single, explicit link exists, draw it –Otherwise use ML algorithm Pick the most likely explicit link Pick the most likely possible link Tool lets you zoom around space, control threshold, link types, etc.

Giant Microbes epidemic visualization via link explicit link inferred linkblog

iRank Find early sources of good information using inferred information paths or timing b1b1 b1b1 b2b2 b2b2 b3b3 b3b3 b4b4 b4b4 b5b5 b5b5 bnbn bnbn … True source Popular site

iRank Algorithm Draw a weighted edge for all pairs of blogs that cite the same URL higher weight for mentions closer together run PageRank control for ‘spam’ Time of infection t0t0 t1t1

Do Bloggers Kill Kittens? 02:00 AM Friday Mar. 05, 2004 PST Wired publishes:Wired "Warning: Blogs Can Be Infectious.” 7:25 AM Friday Mar. 05, 2004 PST Slashdot posts:Slashdot "Bloggers' Plagiarism Scientifically Proven" 9:55 AM Friday Mar. 05, 2004 PST Metafilter announcesMetafilter "A good amount of bloggers are outright thieves."

Information flow in social groups Fang Wu, Bernardo Huberman, Lada Adamic, Joshua Tyler

co-worker mike mom college friend Spread of disease is affected by the underlying network

co-worker mike mom college friend Spread of computer viruses is affected by the underlying network

Viruses (computer and otherwise) are shared indiscriminately (involuntarily) Information is passed selectively from one host to another based on knowledge of the recipient’s interests Difference between information flow and disease/virus spread

co-worker mike mom college friend Spread of information is affected by its content, potential recipients, and network topology

homophily: individuals with like interests associate with one another personal homepages at Stanford distance between personal homepages

The Model: Decay in transmission probability as a function of the distance m between potential target and originating node T (m) = (m+1) -  Tm=0 m=1 m=2 power-law implies slowest decay

Degree distribution of all senders of passing through the HP server outdegree k Virus, information transmission on a scale free network P(k)

 critical threshold  = ,  =0  =100,  =0  =100,  = nodes, epidemic if 1% (10 4 ) infected Pastor-Satorras & Vespignani (2001) epidemics on scale free graphs Newman (2002) Wu et al. (2004)

40 participants (30 within HPL, 10 elsewhere in HP & other orgs) 6370 URLs and 3401 attachments crypotgraphically hashed Question: How many recipients in our sample did each item reach? caveats: messages are deleted (still, the median number of messages > 2000) non-uniform sample Study of the spread of URLs and attachments

forwarded URLs forwarded message Only forwarded messages are counted

short term expense control ads at the bottom of hotmail & yahoo messages average = 1.1 for attachments, and 1.2 for URLs Results

02/19/200315:45:33I-1I-2 02/19/200315:45:33I-1I-3 02/19/200315:45:40E-1I-4 02/19/200315:45:52I-5E-2 02/19/200315:45:55E-3I-6 02/19/200315:45:58I-7I-8 02/19/200315:46:00E-4I-9 02/19/200315:46:05I-10I-11 02/19/200315:46:10I-12I-13 02/19/200315:46:10I-12I-14 02/19/200315:46:10I-12I-15 02/19/200315:46:14I-16E Simulate transmission on log each message has a probability p of transmitting information from an infected individual to the recipient internal node external node

Simulation of information transmission on the actual HP Labs graph an individual is infected if they receive a particular piece of information individuals remain infected for 24 hours start by infecting one individual at random every time an infected individual sends an they have a probability p of infecting the recipient track epidemic over the course of a week, most run their course in 1-2 days

Introduce a decay in the transmission probability based on the hierarchical distance distance 1 distance 2 distance 1 A B h AB = 5

7119 potential recipients p0p0

Conclusions on info flow in social groups Information spread typically does not reach epidemic proportions Information is passed on to individuals with matching properties The likelihood that properties match decreases with distance from the source Model gives a finite threshold Results are consistent with observed URL & attachment frequencies in a sample Simulations following real patterns also consistent

NE MA Milgram’s experiment: Given a target individual and a particular property, pass the message to a person you correspond with who is “closest” to the target. How to search in a small world

Small world experiment at Columbia Dodds, Muhamad, Watts, Science 301, (2003) experiement conducted in targets in 13 different countries 24,163 message chains 384 reached their targets average path length 4.0

Why study small world phenomena? Curiosity: Why is the world small? How are people able to route messages? Social Networking as a Business: Friendster, Orkut, MySpace LinkedIn, Spoke, VisiblePath

Six degrees of separation - to be expected Pool and Kochen (1978) - average person has acquaintances Ignoring clustering, other redundancy … ~ 10 3 first neighbors, 10 6 second neighbors, 10 9 third neighbors But networks are clustered: my friends’ friends tend to be my friends Watts & Strogatz (1998) - a few random links in an otherwise clustered graph give an average shortest path close to that of a random graph

How to choose among hundreds of acquaintances? Strategy: Simple greedy algorithm - each participant chooses correspondent who is closest to target with respect to the given property Models geography Kleinberg (2000) hierarchical groups Watts, Dodds, Newman (2001), Kleinberg(2001) high degree nodes Adamic, Puniyani, Lukose, Huberman (2001), Newman(2003) But how are people are able to find short paths?

Kleinberg (2000) nodes are placed on a lattice and connect to nearest neighbors additional links placed with f(d)~ d(u,v) -r if r = 2, can search in polylog (< (logN) 2 ) time Spatial search “The geographic movement of the [message] from Nebraska to Massachusetts is striking. There is a progressive closing in on the target area as each new person is added to the chain” S.Milgram ‘The small world problem’, Psychology Today 1,61,1967

Kleinberg: searching hierarchical structures ‘Small-World Phenomena and the Dynamics of Information’, NIPS 14, 2001 Hierarchical network models: h is the distance between two individuals in hierarchy with branching b f(h) ~ b -  h If  = 1, can search in O(log n) steps Group structure models: q = size of smallest group that two individuals belong to f(q) ~ q -  If  = 1, can achieve in O(log n) steps

Identity and search in social networks Watts, Dodds, Newman (2001) individuals belong to hierarchically nested groups multiple independent hierarchies coexist p ij ~ exp(-  x)

Identity and search in social networks Watts, Dodds, Newman (2001) There is an attrition rate r Network is ‘searchable’ if a fraction q of messages reach the target N= N= N=204800

Mary Bob Jane Who could introduce me to Richard Gere? High degree search Adamic et al. Phys. Rev. E, (2001)Phys. Rev. E, (2001)

number of nodes found power-law graph

93 number of nodes found Poisson graph

size of graph covertime for half the nodes random walk  = 0.37 fit degree sequence  = 0.24 fit Scaling of search time with size of graph Sharp cutoff at k~N 1/   2 nd degree neighbors

Use a well defined network: HP Labs correspondence over 3.5 months Edges are between individuals who sent at least 6 messages each way Node properties specified: degree geographical location position in organizational hierarchy Can greedy strategies work? Testing the models on social networks ( w/ Eytan Adar)

Degree distribution of all senders of passing through the HP server Strategy 1: High degree search outdegree

Filtered network (6 messages sent each way) 450 users median degree = 10 mean degree = 13 average shortest path = 3 High degree search performance (poor): median # steps = 16 mean = 40 Degree distribution no longer power-law, but Poisson

Strategy 2: Geography

1U 2L3L 3U 2U 4U 1L 87 % of the 4000 links are between individuals on the same floor Communication across corporate geography

Cubicle distance vs. probability of being linked optimum for search

Finding someone in a sea of cubicles median = 7 mean = 12

Strategy 3: Organizational hierarchy

correspondence scrambled

Actual correspondence

Example of search path distance 1 distance 2 hierarchical distance = 5 search path distance = 4 distance 1

Probability of linking vs. distance in hierarchy in the ‘searchable’ regime: 0 <  < 2 (Watts 2001)

Results distancesearchgeodesicorgrandom median43628 mean5.7 (4.7)

Group size vs. probability of linking

optimum for search (Kleinberg 2001) Group size and probability of linking group size g

Search Conclusions Individuals associate on different levels into groups. Group structure facilitates decentralized search using social ties. HP Labs as a social network is searchable but not quite optimal. searching using the organizational hierarchy is faster than using physical location A fraction of ‘important’ individuals are easily findable Humans may be much more resourceful in executing search tasks: making use of weak ties using more sophisticated strategies

PeopleFinder 2 – a search engine for HP people Live Demo If live demo fails: Current PeopleFinder functionality PeopleFinder 2 info on a person Extracted topics for a person Social network Social network visualization Search for individuals by topic Visualize knowledge network Find social network paths to experts Extract & disambiguate names from publicly available documents Enrich information available about individuals Search for them by topic Identify knowledge communities from co-occurrence of names

To find out more: (papers, slides, other research in the group) Information dynamics group (IDL) at HP Labs: List of publications