Twarql Tapping Into the Wisdom of the Crowd Pablo N. Mendes, Pavan Kapanipathi, Alexandre Passant I-SEMANTICS Graz, Austria September 2 nd, 2010

Outline Introduction – Motivation – Contributions Use Cases – IPad Scenario – Location, Sentiment, Recommendations, Competitors System – Demo – Architecture – Activity Flow – Annotation Pipeline Conclusion

Tap into the Wisdom of the Crowd? “taking into account the collective opinion of a group of individuals rather than a single expert to answer a question” (Wikipedia) Has been done successfully – box-office revenue prediction for movies (CoRR’10) – earthquake detection (WWW’10) Can be useful in many scenarios

Social Media Information Overload!

Twitter 140 characters Users can “follow” updates of other users Hashtags – Category markers Short URLs

Twarql Contributions Expressive description of an information need – Beyond keywords (uses SPARQL) Flexibility on the point of view – Ability to “slice and dice” data in several dimensions: thematic, spatial, temporal, sentiment, etc. Streaming data + background knowledge – Enables automatic evolution and serendipity Scalable real time delivery – Using sparqlPuSH (SFSW’10)

Use Cases (IPad Scenario) 1.Location – Retrieve stream of locations where my product is being mentioned right now. 2.Consumer sentiment – Retrieve all people that have said negative things about my product. 3.Content suggestion – Retrieve all URLs that people recommend with relation to my product. 4.Related entities – What competitors are being mentioned with my product?

Use Case 1: Location (query) Retrieve a stream of locations where my product is being mentioned right now. SELECT ? location WHERE { ?tweet moat:taggedWith dbpedia:IPad. ?presence opo:currentLocation ?location. ?presence opo:customMessage ?tweet. } SELECT ? location WHERE { ?tweet moat:taggedWith dbpedia:IPad. ?presence opo:currentLocation ?location. ?presence opo:customMessage ?tweet. }

Use Case 1: Location SELECT ? location WHERE { ? tweet moat : taggedWith dbpedia : IPad. ? presence opo: currentLocation ? location. ? presence opo: customMessage ? tweet. } ?presence ?location ?tweet dbpedia: IPad moat:taggedWith opo:customMessage Lorem ipsum bla bla this is an example Lorem ipsum bla bla this is an example Lorem ipsum bla bla this is an example tweet

Use Case 2: Consumer Sentiment Retrieve all people that have said negative things about my product. SELECT ? user WHERE { ? tweet sioc:has_creator ? user. ? tweet moat:taggedWith dbpedia:IPad. ? tweet twarql:sentiment twarql:Negative. } SELECT ? user WHERE { ? tweet sioc:has_creator ? user. ? tweet moat:taggedWith dbpedia:IPad. ? tweet twarql:sentiment twarql:Negative. }

Use Case 2: Consumer sentiment ?user :Negative ?tweet dbpedia: IPad moat:taggedWith sioc:has_creator Lorem ipsum bla bla this is an example tweet Invite users for testing our

Use Case 3: Content suggestion Retrieve all URLs that people recommend with relation to my product SELECT ?url WHERE { ? tweet moat:taggedWith dbpedia:IPad. ? tweet sioc:links_to ?url. } SELECT ?url WHERE { ? tweet moat:taggedWith dbpedia:IPad. ? tweet sioc:links_to ?url. }

Use Case 3: Content Suggestion SELECT ? user WHERE { ? tweet sioc : has_creator ? user. ? tweet moat : taggedWith dbpedia : IPad. ? tweet twarql : sentiment twarql : Negative. } ?url ?tweet dbpedia: IPad moat:taggedWith Lorem ipsum bla bla this is an example tweet

Use Case 4: Competitors ?competitor ?category ?tweet dbpedia: IPad moat:taggedWith skos:subject category:Wi-Fi category:Touchscreen skos:subject Background Knowledge (e.g. Lorem ipsum bla bla this is an example tweet HPTabletPC IPhone

Use Case 4: Competitors (contd.) Highlights – When a new competitor “appears” in the KB, no change is needed in the query => Automatic Evolution – We found interesting products that we didn’t initially consider as competitors of IPad (e.g. IPhone) => Serendipity

Use Case 4: Competitors (query) What competitors of my product are being mentioned? SELECT ? competitor WHERE { dbpedia:IPad skos:subject ?category. ?competitor skos:subject ?category. ?tweet moat:taggedWith ?competitor. } SELECT ? competitor WHERE { dbpedia:IPad skos:subject ?category. ?competitor skos:subject ?category. ?tweet moat:taggedWith ?competitor. } ?tweet moat:taggedWith dbpedia:Ipad. - …are being mentioned with my product?

Demonstration Cuebee – query formulation Twarql – information extraction – stream querying sparqlPuSH – real time delivery Demo link:

Architecture Mendes, Passant, Kapanipathi, Sheth. Linked Open Social Signals, Web Intelligence 2010

Twarql Streaming Activity Diagram Web Client Web Client APP SERVER APP SERVER DIST. HUB (SEMANTIC) PUBLISHER (SEMANTIC) PUBLISHER SOCIAL SENSOR SOCIAL SENSOR Twitter API /register query, #id REGISTER (query, new hubURL()) LISTEN(tweet) ANNOTATE(tweet) STREAM(tweet) keywords FILTER(tweet, for all query) STORE(tweet) PUBLISH(tweet) SETUP REQUEST(#id)PULL(hubURL, req) hubURL UPDATE INTERFACE PUSH(tweet, subscriber) UPDATE(hubURL, rssTweet) POLL QUERY RELAY QUERY(#id, query) /feed feed update STREAM(query, #id) FORMULATE QUERY cache UPDATE(tweet) CACHE(tweet) /publish RDF store /subscribe /sparql topic idHub URL #id1http://hub1 #id2http://hub2 /sparql #id

Annotation URL extraction – Regex based, short URL resolution via http redirects Hashtag extraction – Regex based, “resolution” via TagDef and Tagal.us Entity mention extraction – “Spotting” via string matching (prefix tree) based on a dictionary (Dbpedia) – Disambiguation on the way! (est. October) Conversion to RDF triples – using SIOC, FOAF, MOAT, etc.

Conclusion Flexibility and expressiveness in managing real time streams of information! Triples generated for the IPad scenario – From June 3 rd to June 8 th – 511,147 tweets – 4,479,631 triples … and counting! You can generate triples too: 53,237 positive; 6,739 negative; 451,171 neutral

Thank you Connect @pavankaps Collaborate: – –