FaitCrowd: Fine Grained Truth Discovery for Crowdsourced Data Aggregation   Fenglong Ma1, Yaliang Li1, Qi Li1, Minghui Qiu2, Jing Gao1, Shi Zhi3, Lu Su1, Bo Zhao4, Heng Ji5, Jiawei Han3 Presenter: Jing Gao 1SUNY Buffalo; 2Singapore Management University; 3University of Illinois Urbana-Champaign; 4LinkedIn; 5Rensselaer Polytechnic Institute

Which of these square numbers also happens to be the sum of two smaller numbers? 16 25 36 49 https://www.youtube.com/watch?v=BbX44YSsQ2I

A Straightforward Aggregation Method Voting/Averaging Take the value that is claimed by majority of the sources (users) Or compute the mean of all the claims

Which of these square numbers also happens to be the sum of two smaller numbers? 16 25 36 49 https://www.youtube.com/watch?v=BbX44YSsQ2I

A Straightforward Aggregation Method Voting/Averaging Take the value that is claimed by majority of the sources (users) Or compute the mean of all the claims Limitation Ignore source reliability (user expertise) Source reliability Is crucial for finding the true fact but unknown

Object Aggregation Source 1 Source 2 Source 3 Source 4 Source 5

Truth Discovery Principle To learn users’ reliability degree and discover trustworthy information (i.e., the truths) from conflicting data provided by various users on the same object. A user is reliable if it provides many pieces of true information A piece of information is likely to be true if it is provided by many reliable users

Existing Work on Truth Discovery Existing methods Assign single expertise (reliability degree) to each user (source). Expertise Barack Obama Albert Einstein Michael Jackson

Example--Existing Truth Discovery Methods Input Question Set User Set Answer Set Output Users’ Expertise Truths Question User u1 u2 u3 q1 1 2 q2 q3 q4 q5 q6 User u1 u2 u3 Expertise 5.00E-11 0.961 3.989 Question q1 q2 q3 q4 q5 q6 Truth 1 2 Question q1 q2 q3 q4 q5 q6 Ground Truth 1 2

Overview of Our Work Goal To learn fine-grained (topical-level) user expertise and the truths from conflicting crowd-contributed answers. Politics Physics Music

Example--Our Model Input Output Question Set User Set Answer Set Word u1 u2 u3 q1 1 2 a b q2 c q3 q4 d e q5 f q6 Input Question Set User Set Answer Set Question Content Output Questions’ Topic Topical-Level Users’ Expertise Truths Topic Question K1 q1 q2 q3 K2 q4 q5 q6 User u1 u2 u3 Expertise K1 2.34 2.70E-4 1.00 K2 1.30E-4 2.35 Question q1 q2 q3 q4 q5 q6 Truth 1 2 Question q1 q2 q3 q4 q5 q6 Ground Truth 1 2

FaitCrowd Model Overview Jointly modeling question content and users’ answers by introducing latent topics. Modeling question content can help estimate reasonable user reliability, and in turn, modeling answers leads to the discovery of meaningful topics. Learning topic-level user expertise, truths and topics simultaneously.

Modeling Question Content Word Generation Assume that each question is about a single topic (the length of each question is short). Draw a topic indicator

Modeling Question Content Word Generation Assume that each question is about a single topic (the length of each question is short). Draw a topic indicator Assume that a word can be drawn from topical word distribution or background word distribution. Draw a word category

Modeling Question Content Word Generation Assume that each question is about a single topic (the length of each question is short). Draw a topic indicator Assume that a word can be drawn from topical word distribution or background word distribution. Draw a word category Draw a word

Modeling Answers Answer Generation The correctness of a user’s answer may be affected by the question’s topic, user’s expertise on the topic and the question’s bias. Draw user’s expertise

Modeling Answers Answer Generation The correctness of a user’s answer may be affected by the question’s topic, user’s expertise on the topic and the question’s bias. Draw user’s expertise Draw the truth

Modeling Answers Answer Generation The correctness of a user’s answer may be affected by the question’s topic, user’s expertise on the topic and the question’s bias. Draw user’s expertise Draw the truth Draw the bias

Modeling Answers Answer Generation The correctness of a user’s answer may be affected by the question’s topic, user’s expertise on the topic and the question’s bias. Draw user’s expertise Draw the truth Draw the bias Draw a user’s answer

Inference Method Gibbs-EM Gibbs sampling to learn the hidden variables and . Gradient descent to learn hidden factors and .

Datasets & Measure Datasets Measure The Game Dataset The SFV Dataset Collected from a crowdsourcing platform via an Android App based on a TV game show “Who Wants to Be a Millionaire”. 2,103 questions, 37,029 sources, 214,849 answers and 12,995 words The SFV Dataset Extracted from Slot Filling Validation (SFV) task of the NITS Text Analysis Conference Knowledge Base Population (TAC-KBP) track. 328 questions, 18 sources, 2,538 answers and 5,587 words Measure Error Rate The lower the better

Baseline Methods Basic Method Truth Discovery Crowdsourcing MV Truth Discovery Truth Finder AccuPr Investment 3-Estimates CRH CATD Crowdsourcing D&S ZenCrowd Variations of FaitCrowd FaitCrowd-b FaitCrowd-b-g

Performance Validation Table 1: Performance on the Game Dataset. Analysis For easy questions (from Level 1 to Level 7), all the methods can estimate most answers correctly. For difficult questions (from Level 8 to Level 10) , the performance of FaitCrowd is much better than that of the baseline methods. FaitCrowd performs well on both Game and SFV datasets. Table 2: Performance on the SFV Dataset.

Model Validation Goal Explanation Illustrate the importance of joint modeling question content and answers by comparing with the method that conducts topic modeling and true answer inference separately. Explanation Dividing the whole dataset into sub-topical datasets will reduce the number of responses per topic, which leads to insufficient data for baseline approaches. Table 3: Results of Model Validation.

Topical Expertise Validation Goal Validate the correctness of topical expertise learned by FaitCrowd. Ideally, the expertise estimated by the proposed method is consistent with the ground truth accuracy. Figure 1: Topic 2 on the Game Dataset. Figure 2: Topic 4 on the SFV Dataset.

Expertise Diversity Analysis Goal Demonstrate that the topical expertise for each source varies on different topics. Ideally, the topical expertise should correspond to the ground truth accuracy, i.e., the higher expertise, the higher the ground truth accuracy. Figure 3: Source 7 on the Game Dataset. Figure 4: Source 16 on the SFV Dataset.

Problem Solution Results Summary Recognize the difference in source reliability among topics on the truth discovery task and propose to incorporate the estimation of fine grained reliability into truth discovery. Solution Propose a probabilistic model that simultaneously learns the topic-specific expertise for each source, aggregates true answers, and assigns topic labels to questions. Results Empirically show that the proposed model outperforms existing methods in multi-source aggregation with two real world datasets.

Thank you! Questions?