1. 1 How Could We All Get Along on the Web 2.0? The Power of Structured Data on the Web Sihem Amer Yahia Yahoo! Research

2. 2 Outline Web search and web 2.0 search Why should we all get along? How could we all get along? Related work Conclusion

3. Yahoo! Research 3 Web search Access to “heterogeneous”, distributed information –Heterogeneous in creation –Heterogeneous in motives Keyword search very effective in connecting people to information Search web pages

4. Yahoo! Research 4 Web search vs web 2.0 search? Content creators Content aggregators Feeds Content consumers Anonymous Subscribers Web 2.0 search Web search

5. Yahoo! Research 5 Web 2.0 a generation of internet-based services that – let people form online communities – in order to collaborate – and share information in previously unavailable ways

6. Yahoo! Research 6 Online communities Subscribers join communities where they –exchange content: emails, comments, tags –rate content from other subscribers –exhibit common behavior About 500M unique Y! visitors per month, about 200M subscribers (login visitors) to more than 130 Y! services

7. Yahoo! Research 7 Web 2.0 search Web 2.0 Connecting people to people Flickr Y!Answers YouTube Y!Groups

8. Yahoo! Research 8 Web 2.0 search examples Mary is a professional photographer and is looking for aerial photos of the Hoggar desert She is also an amateur Jazz dancer and wants to ask about dance schools w/flexible schedules in SF She is also looking for the latest video on bird migration in Central Park, NY She has heart problems but loves biking and is interested in finding about email discussions on biking trails in northern California

9. Yahoo! Research 9 Outline Web search and web 2.0 search Why should we all get along? How could we all get along? Related work Conclusion

10. Yahoo! Research 10 Improving users’ experience Keyword search should be maintained: simple and intuitive Keyword queries usually short –only express a small fraction of the user's true intent Users's interactions within community-based systems can be used to infer a lot more about intent and return better answers

11. Yahoo! Research 11 Why should we all get along? Contributed content is structured –This is what DB community knows how to do best Relevance to query keywords is key –This is what IR community knows how to do best

12. Yahoo! Research 12 Searching online communities idauthordate 001s21/1/06 002s41/8/06 003s43/9/06 sub trust s1s3c13 s1s4c14 s2s3c23 s4s6c46 … data table community relationship table idsu b annotation 001s21/1/06 002s41/8/06 003s43/9/06 Tags, ratings, Reviews table

13. Yahoo! Research 13 Searching online communities Search for most relevant data on some topic –Querying data: selection over data table –Querying annotations: selection over annotation table + join w/data table –Personalizing answers: join w/subscribers table Relevance: use data relevance + annotation table

14. Yahoo! Research 14 Why should we all get along? Query interpretation depends on subscriber’s interest at the time of querying Data annotations are dynamic –Precompute all (sub,sub,trust) for each topic? Need for dynamic query generation

15. Yahoo! Research 15 DB and IR Shared interactions help focus search –User-input, community-input, extraction –Personalizing answers with community information Ranking as a combination of –Relevance –Relationship strengths between people in the same community

16. Yahoo! Research 16 Outline Web search and web 2.0 search Why should we all get along? How could we all get along? – Applications – Technical challenges Related work Conclusion

17. Yahoo! Research 17 Applications Flickr enables sharing and tagging photos Y! Answers enables asking and answering questions in natural language YouTube enables sharing videos, rating videos, commenting on videos and subscribing to new videos from favorite users Y! Groups enables creating groups, joining existing groups, posting in a group

18. Yahoo! Research 18 Flickr Acquired by Y! in 2005 Tag search Photos grouped into categories. Set privacy levels on each photo

19. Yahoo! Research 19

20. Yahoo! Research 20

21. Yahoo! Research 21 The new inputs to Flickr search Users tag and rate photos Users tagging same photos with similar tags form a community of interest Combine tag-based search with community knowledge Combine photo rating with relationship strength Search SubscriberQueryCommunities

22. Yahoo! Research 22 Y! Answers Launched in second half of 2005 Incentive system based on points and voting for best answers Questions grouped by category Some statistics: –over 60 million users –over 120 million answers, available in 18 countries and in 6 languages

23. Yahoo! Research 23

24. Yahoo! Research 24 Y! Answers

25. Yahoo! Research 25 Y! Answers

26. Yahoo! Research 26 The new inputs to Y!Answers search Users provide Questions/Answers Voting information reflects communities of interest Combine community information with answer rating Search SubscriberQueryCommunities

27. Yahoo! Research 27 YouTube Founded in February 2005 Tag search Videos grouped by category Some statistics: –100 million views/day –65,000 new videos/day

28. Yahoo! Research 28

29. Yahoo! Research 29

30. Yahoo! Research 30 The new inputs to YouTube search Users provide videos, tags, ratings, comments Similar tags on same videos imply communities of interest Combine community information with video rating Search SubscriberQueryCommunities

31. Yahoo! Research 31 Yahoo! Groups Yahoo! acquired eGroups in 2000 Group moderators Groups belong to categories Public and private groups Some statistics: –over 7M groups –over 190M subscribers –over 100K new subscribers/day –over 12M emails/day

32. Yahoo! Research 32

33. Yahoo! Research 33

34. Yahoo! Research 34

35. Yahoo! Research 35 Alternative query interpretations Return all group postings relevant to a query. Return only posting by subscribers sharing the same interests: women with heart disease interested in steep slopes

36. Yahoo! Research 36 The new inputs to Group search Users participate in many groups Group membership and postings imply communities of interest Combine community information with postings relevance Search SubscriberQueryCommunities

37. Yahoo! Research 37 Outline Web search and web 2.0 search Why should we all get along? How could we all get along? – Applications – Technical challenges Related work Conclusion

38. Yahoo! Research 38 So, how can we all get along? Augment keyword query with conditions on structure to focus and personalize search (DB) – Flickr: tags – Answers: points – YouTube: reviews and ratings – Groups: emails Combine it with relevance (IR)

39. Yahoo! Research 39 Search architecture Subscriber Query evaluation search terms Query tightening Find relevant community of interest Ranking content relevance + relationship structured query

40. Yahoo! Research 40 Example “biking trails northern california” Query tightening message contains “…” and from = “s1” or “s2” From: To: Date: Subject: Content: message structure S1 S2 S3 S4 S5 S6 S7 ( si, sj, cij ) Many such relationships depending on subscriber’s interests

41. Yahoo! Research 41 Can we really all get along? IR may think that user weights are enough to target communities of interest and personalize queries DB thinks expressiveness of query languages cannot all be captured by ranking functions

42. Yahoo! Research 42 Query rewriting Content-Only Content in context Loose interpretation of context

43. Yahoo! Research 43 Query relaxation Primitive operations for dropping query predicates Answers to relaxed query contain answers to exact one Scores relaxed answer no higher than score of exact one

44. Yahoo! Research 44 Query tightening Primitive operations for adding query predicates Tighter answers are found but looser answers should be maintained Scores tighter answers no lower than scores of other answers

45. Yahoo! Research 45 More technical challenges Query tightening primitives to focus search Subscriber has a different profile/community of interest Topk processing needs to enforce user profiles

46. Yahoo! Research 46 Outline Web search and web 2.0 search Why should we all get along? How could we all get along? – Applications – Technical challenges Related work Conclusion

47. Yahoo! Research 47 Related Work Language models: Ask Bruce Croft Web search personalization – Search behavior – HARD track at TREC Building relationship graphs: – Collaborative filtering –Clustering – Unsupervised learning

48. Yahoo! Research 48 Tempting conclusion Little information could be gathered on users to greatly improve new-generation search IR and DB views both needed

49. Yahoo! Research 49 More technical challenges Subscriber belongs to different communities of interest Should subscriber turn off personalization? How is efficiency affected? (revisiting topk processing) Back from community search to web search?

50. Yahoo! Research 50 Beyond search in online communities Are online communities a way to build more accurate user profiles or more? –display relevant groups when user is asking a question on Y! Answers: mashups?

51. Yahoo! Research 51 Danger of online communities Are we discouraging diversity?

52. 52 Thank you. Questions? sihem@yahoo-inc.com http://research.yahoo.com/~sihem