Ubiquitous Human Computation KAIST KSE Uichin Lee May 11, 2011

Outline Review recent papers: – Crowd-Sourced Sensing and Collaboration Using Twitter, WoWMOM 2010 – Earthquake Shakes Twitter User: Analyzing Tweets for Real-Time Event Detection, WWW 2010 – Location-based Crowdsourcing: Extending Crowdsourcing to the Real World, NordiCHI 2010 – Social Sensors and Pervasive Services: Approaches and Perspectives, PerCol 2011 Understand the potential of ubiquitous human computation (+social networking)

Crowd-Sourced Sensing and Collaboration Using Twitter Murat Demirbas, Murat Ali Bayir, Cuneyt Gurcan Akcora, Yavuz Selim Yilmaz SUNY Buffalo WoWMOM 2010 Slides are based on

Cellphones! 3-4B cellphone users worldwide 1.13 billion phones sold in 2009 (36 per sec) vs 0.3 billion PCs 174M were smartphones – 15% (up from 12.8% in 2008) – Expected to exceed # feature phones

Status quo in cellphones Each device connects to the Internet – to download/upload data and – to accomplish a task that does not require collaboration and coordination

What is missing? An infrastructure to assist mobile users to perform collaboration and coordination ubiquitously Any user should be able to search & aggregate the data published by other users in a region

Our goal To provide a crowdsourced sensing and collaboration service using Twitter To enable aggregation and sharing of data; dynamically assign sensing tasks to other cellphone users

Why Twitter? Open publish-subscribe system: 105 million users, over 30 million users in US, 55 million tweets 600 million search queries everyday Each tweet has 140 char limit Twitter provides an open source search API and a REST API (that enables developers to access tweets, timelines, and user data) Different actors may integrate published data differently and can offer new services in unanticipated ways

Crowdsourcing architecture

Sensweet Employs the smartphone’s ability to work in the background without distracting a mobile user – Sense the surrounding environment and send the resulting data to Twitter To search and process sensor values on Twitter, we need to agree on a standard for publishing these sensor readings – Bio-code: Uses Twitter bio sections & allows users to search for the sensors they are looking for on-the-fly – TweetML: Uses pre-defined hashtags to improve searchability

Askweet Accepts a question from Twitter – tries to answer the question using the data on Twitter, potentially data published by Sensweets – if that is not possible, Askweet finds experts on Twitter and forwards the question to these experts (not clear how this was done in the paper) Parallelizable, easy to “cloudify” for scalable service provisioning

Applications Crowdsourced weather Noise map application Location-based queries (with Foursquare)

1. Crowdsourced weather Current weather, everybody on Twitter can be an expert Question to Askweet: “?Weather Loc:Buffalo,NY” Forwarded question:“How is the weather there now? reply 0 for sunny, 1 for cloudy, 2 for rainy, and 3 for snowy

Experimental results for NYC in different time slices

2. Noise map application Implemented a Sensweet client for the Nokia N97 Smartphone series Sensweet client detects a noise level of the surrounding environment and forwards this data to Twitter in the TweetML format Sound sample is classified into: Low, Medium, High state – Each level is modeled using normal distribution – Input signal is compared with 3 distributions (Low, Medium, and High)

Noise map application

Noise levels for a user

3. Location based queries Factual vs. non-factual queries – Factual: “hotels in Miami” – Non-factual: “Anyone knows any cheap, good hotel, price ranges between 100 to 200 dollars in Miami?” Traditional search engine performs poorly! Significant fraction of location-based queries (in Twitter) is non-factual – e.g., 63% of the queries were non-factual, while only 37% of them were factual (manual classification of 269 queries) Crowdsourcing Location-based Queries, Bulut et al., Pervasive Collaboration and Social Networking,

Location based queries Aardvark uses a social network of the asker to find suitable answerers for the query and forwards this query to the answerers, and returns any answer back to the asker. How about Twitter + Foursquare? – Use Foursquare to determine users that frequent the queried locale and that have interests on the queried category (e.g., food, nightlife) – Find a right set of people to ask!

[Questions to be asked] [Users] [Valid questions][Valid answers] [Questions detected][Answer detected] [Answer to be forwarded] Moderator Asker tweet starting with ? keyword checking (anyone, suggestion, where) label the category and quality of questions forwards validated questions to appropriate people (using Twitter bio or Foursquare info) Constantly polling Twitter account to check answers

Experiment Setup Question dataset consists of 269 questions that the system collected over Twitter and validated as acceptable by the moderators. Manually categorize questions as factual and nonfactual: 63% - non-factual; 37% factual Some examples of questions for each type.

Foursquare Reply Rate vs. Random User Reply Rate Foursquare

Response Time 13 minutes median response time which is comparable with Aardvark 50% of the answers were received within the first 20 minutes.

Earthquake Shakes Twitter User: Analyzing Tweets for Real-Time Event Detection Takehi Sakaki Makoto Okazaki @ymatsuo Tokyo University WWW 2010 Conference

What’s happening? Twitter – is one of the most popular microblogging services – has received much attention recently Microblogging – is a form of blogging that allows users to send brief text updates – is a form of micromedia that allows users to send photographs or audio clips In this research, we focus on an important characteristic real-time nature

Real-time Nature of Microblogging – Twitter users write tweets several times in a single day. – There is a large number of tweets, which results in many reports related to events – We can know how other users are doing in real-time – We can know what happens around other users in real- time. social events parties baseball games presidential campaign disastrous events storms fires traffic jams riots heavy rain-falls earthquakes

Our Goals propose an algorithm to detect a target event – do semantic analysis on Tweet to obtain tweets on the target event precisely – regard Twitter user as a sensor to detect the target event to estimate location of the target produce a probabilistic spatio-temporal model for – event detection – location estimation propose Earthquake Reporting System using Japanese tweets

Twitter and Earthquakes in Japan a map of earthquake occurrences world wide a map of Twitter user world wide The intersection is regions with many earthquakes and large twitter users.

Twitter and Earthquakes in Japan Other regions: Indonesia, Turkey, Iran, Italy, and Pacific coastal US cities

Event detection algorithms do semantic analysis on Tweet – to obtain tweets on the target event precisely regard Twitter user as a sensor – to detect the target event – to estimate location of the target

Semantic Analysis on Tweet Search tweets including keywords related to a target event – Example: In the case of earthquakes “shaking”, “earthquake” Classify tweets into a positive class or a negative class – Example: “Earthquake right now!!” --- positive “Someone is shaking hands with my boss” --- negative – Create a classifier

Semantic Analysis on Tweet Create classifier for tweets – use Support Vector Machine(SVM) Features (Example: I am in Japan, earthquake right now!) – A: Statistical features (7 words, the 5 th word) the number of words in a tweet message and the position of the query within a tweet – B: Keyword features ( I, am, in, Japan, earthquake, right, now) the words in a tweet – C: Word context features (Japan, right) the words before and after the query word

Tweet as Sensor Data ・・・ tweets ・・・ Probabilistic model Classifier observation by sensors observation by twitter users target event target object Probabilistic model values Event detection from twitter Object detection in ubiquitous environment the correspondence between tweets processing and sensor data processing for event detection

Tweet as Sensor Data some users posts “earthquake right now!!” some earthquake sensors responses positive value We can apply methods for sensory data detection to tweets processing ・・・ tweets Probabilistic model Classifier observation by sensors observation by twitter users target event target object Probabilistic model values Event detection from twitter Object detection in ubiquitous environment ・・・ search and classify them into positive class detect an earthquake earthquake occurrence

Tweet as Sensor Data We make two assumptions to apply methods for observation by sensors Assumption 1: Each Twitter user is regarded as a sensor – a tweet → a sensor reading – a sensor detects a target event and makes a report probabilistically – Example: make a tweet about an earthquake occurrence “earthquake sensor” return a positive value Assumption 2: Each tweet is associated with time and location info – time : posting timestamp – location : GPS data or location information in user’s profile By processing time and location information, we can detect target events and find events’ locations

Probabilistic Model Why we need probabilistic models? – Sensor readings are noisy and sometimes sensors work incorrectly – We cannot judge whether a target event occurred or not from a single tweet – We have to calculate the probability of an event occurrence from a series of data We propose probabilistic models for – event detection from time-series data – location estimation from a series of spatial information

Temporal Model We must calculate the probability of an event occurrence from a set of sensor readings We examine the actual time-series data to create a temporal model

Temporal Model with Exponential Dist. Example: Earthquake and Typhoon

Spatial Model We must calculate the probability distribution of location of a target We apply Bayes filters to this problem which are often used in location estimation by sensors – Kalman Filters – Particle Filters

Bayesian Filters for Location Estimation Kalman Filters – are the most widely used variant of Bayes filters – approximate the probability distribution which is virtually identical to a uni-modal Gaussian representation – advantages: computational efficiency – disadvantages: limited to accurate sensors or sensors with high update rates

Bayesian Filters for Location Estimation Particle Filters – represent the probability distribution by sets of samples, or particles – advantages: able to represent arbitrary probability densities particle filters can converge to the true posterior even in non-Gaussian, nonlinear dynamic systems. – disadvantages: difficult to apply to high-dimensional estimation problems

Information Diffusion Related to Real-time Events Proposed spatiotemporal models need to meet one condition that – sensors are assumed to be independent We check if information diffusions about target events happen because – if an information diffusion happened among users, Twitter user sensors are not independent, they affect each other (correlation!)

Information Diffusion Related to Real-time Events Nintendo DS Game an earthquakea typhoon Information Flow Networks on Twitter In the case of an earthquake and a typhoon, very little information diffusion takes place on Twitter, compared to Nintendo DS Game → We assume that Twitter user sensors are independent about earthquakes and typhoons

Experiments and Evaluation We demonstrate performances of – tweet classification – event detection from time-series data → show this result in “application” – location estimation from a series of spatial information

Evaluation of Semantic Analysis Queries – Earthquake query: “shaking” and “earthquake” – Typhoon query:”typhoon” Examples to create classifier – 597 positive examples

Evaluation of Semantic Analysi We obtain highest F-value when we use Statistical features and all features. Keyword features and Word Context features don’t contribute much to the classification performance A user becomes surprised and might produce a very short tweet It’s apparent that the precision is not so high as the recall FeaturesRecallPrecisionF-Value Statistical87.50%63.64%73.69% Keywords87.50%38.89%53.85% Context50.00%66.67%57.14% All87.50%63.64%73.69%

Evaluation of Spatial Estimation Target events – earthquakes 25 earthquakes from August.2009 to October 2009 – typhoons name: Melor Baseline methods – weighed average simply takes the average of latitudes and longitudes – median simply takes the median of latitudes and longitudes Metric: distance from an epicenter – The smaller the better!

Evaluation of Spatial Estimation Tokyo Osaka actual earthquake center Kyoto estimation by median estimation by particle filter balloon: each tweets color : post time

Evaluation of Spatial Estimation Typhoon

Discussions of Experiments Particle filter performs better than other methods If the center of a target event is in an oceanic area, it’s more difficult to locate it precisely from tweets It becomes more difficult to make good estimation in less populated areas

Results of Earthquake Detection JMA intensity scale2 or more3 or more4 or more Num of earthquakes78253 Detected70(89.7%)24(96.0%)3(100.0%) Promptly detected*53(67.9%)20(80.0%)3(100.0%) Promptly detected: detected in a minutes JMA intensity scale: the original scale of earthquakes by Japan Meteorology Agency Period: Aug.2009 – Sep Tweets analyzed : 49,314 tweets Positive tweets : 6291 tweets by 4218 users We detected 96% of earthquakes that were stronger than scale 3 or more during the period.

Conclusions We investigated the real-time nature of Twitter for event detection Semantic analyses were applied to tweets classification We consider each Twitter user as a sensor and set a problem to detect an event based on sensory observations Location estimation methods such as Kaman filters and particle filters are used to estimate locations of events We developed an earthquake reporting system, which is a novel approach to notify people promptly of an earthquake event We plan to expand our system to detect events of various kinds such as rainbows, traffic jam etc.

Location-based Crowdsourcing: Extending Crowdsourcing to the Real World Alt et al. NordiCHI 2010

Motivation Crowdsourcing beyond the digital? – Seeker and solvers – Important aspects: right time and location for matchmaking. Several scenarios: – Recommendations on demand (e.g., buying something?) – Recording on demand (e.g., missing lectures?) – Remotely looking around? (e.g., apartment?) – Real-time weather information – Translations on demand

System Architecture

The mobile client screenshots: (a) Main menu where users can search tasks. (b) A sample task retrieved from the database.

Lessens learned Users prefer address-based task selection (GPS is too hard to parse) Picture tasks are most popular (easy to handle) Tasks were mainly solved at or close to home Tasks are solved after work Response times vary

Lessens learned Informative tasks are as popular as picture tasks Time-critical tasks are out of interest Solution should be achievable in 10 minutes Tasks are still solved after work Mid-day breaks are good times to search for task Solving a task can take up to one day Home and surrounding areas are the most favorite places for solving tasks Voluntary tasks have lower chance (monetary rewards: 77%) Users search for tasks in their current location

Social Sensors and Pervasive Services: Approaches and Perspectives Rosi et al., PerCol 2011

Social Sensors? Device intelligence with various on-board sensors such as GPS Human intelligence with “social sensors” – Twitter posts, Facebook status updates, pictures posted on Flickr – Personal information: shopping patterns, place visit patterns, etc. (with some potential social interactions)

Approaches to integrate social sensing and pervasive services

A: Extracting data from social networks – Detecting crowded sites (Fujisaka et al., 2010) – Mining landmarks from blogs (Ji et al., 2009) – Event detection using Flickr (Zhao et al., 2006) B: Exploiting social networks as a socio- pervasive middleware – Twitter with sensors (Demirbas et al., 2010) – S-Sensors with micro-blogging (Baqer et al., 2009) – Status update feeds to social networks (CenceMe)

Approaches to integrate social sensing and pervasive services C: Pervasive overlays on social networks – Interconnecting and sharing data sensed from personal devices with the rest of the world – SenseFace: Capture and process (local), and disseminate data (social nets) Dynamically mash-up sensor data and social networks D: App-specific socio-pervasive networks – Fusing mobile, sensor, and social data to fully enable context-aware computing

Some Issues Key issues – Rich data, yet comes at the cost of understanding the data – Sheer size (raw facts and data produced by sensors) Un-structured, noisy data – Unified data representation and interpretation – Overcoming uncertainty of data No guarantee on the delivery of specific info about facts and at specific times by social sensors – Systems require “critical mass”; heterogeneous popularity based on location (e.g., rural area vs. urban area) Completely out-of-loop of system managers and app developers