Reputation Systems For Open Collaboration, CACM 2010 Bo Adler, Luca de Alfaro et al. Nishith Agarwal

What are Reputation Systems? Wikipedia Definition: A reputation system computes reputation scores for a set of objects within a community or domain, based on a collection of opinions other users hold about the objects. Why do we need them? Reputation Systems can help stem abuse of content, and can offer indications of content quality. In many ways, reputation systems are the on-line equivalent of the body of laws that regulates the real-world interaction of people. Who uses reputation systems? 2

Types of Reputation Systems? 3 WikiTrust and Crowdsensus are both content-driven systems

WikiTrust A content-driven reputation system for Wiki authors and content on Wikipedia Goals: Incentivize users to give lasting contributions Help users and editors increase the quality of content Offer content consumers a guide to content quality Components: User reputation system: Users gain reputation when they add content that is preserved by subsequent users. Content reputation system: Content gains reputation when it is revised by highly reputed authors. Firefox Extension: Lets users view content reputation by changing text background color. 4

WikiTrust – User Reputation System Contribution quality: Relies on edit distance between revisions. -1 if changes made by b are completely reverted +1 if changes made by b are completely preserved 5 Contributions are considered good quality if the change is preserved in subsequent revisions User reputation is computed according to quality and quantity of contributions they make User Reputation: Is proportional to the edit distance and contribution quality of b. r(B) ≈ d(a,b) + q(b | a,c) + r(C) r(B) being the reputation of author B of revision b r(C) being the reputation of author C of revision c

WikiTrust – Content Reputation System (TextTrust) Based on extent to which content was revised, and reputation of users who revised it. High content reputation requires consensus from reputed authors. Basic Algorithm: Content that is edited is assigned a small fraction of the revision user’s reputation. Unedited content gains more reputation. Some Tweaks: Ensures that re-arranging or deleting text leaves a low reputation mark Content reputation cannot exceed the revision user’s reputation Users cannot raise arbitrary reputation by multiple edits 6

Crowdsensus A content-driven reputation system built to analyze user edits to Google Maps  Goals: To measure accuracy of a user who contributes information To display accurate details for a business (title, address, phone etc.) Differences from WikiTrust: There exists a “ground truth” User reputation is not visible. Hence, no need to keep algorithm simple. User identity is stronger 7

Crowdsensus - Algorithm Structured as fixed point graph algorithm Vertices are users u, and business attributes a Edges are attribute values v Each user has truthfulness value q u Algorithm Details: User vertices send (q u, v u ) pairs to value vertices An attribute inference algorithm is used to derive probability distribution over values (v 1, v 2..v n ) Crowdsensus sends back to user u, the estimate probability that v u is correct Different attribute inference algorithms tailored to every attribute type Comparison to Bayesian Inference Model: For 1000 attributes, 100 users, 10 attribute values: –CrowdSensus error rate: 2.8% –Bayesian error rate: 7.9% For 1000 attributes, 100 users, 5 attribute values –CrowdSensus error rate: 12.6% –Bayesian error rate: 22% 8 a q1,v 1 q2,v 2 qn,v n

Design Considerations User vs. Content Driven Visibile to User? Weak vs. Strong Identity Existence of Ground Truth Chronological vs. Global Updates WikipediaContentYesWeakNoChronological Google Maps ContentNoStrongYesBoth YelpUserYesStrongNoChronological 9

Conclusion Reputation Systems are becoming increasing important to regulate real-world interactions of people Reputation Systems can be very different depending on the content Wikipedia’s WikiTrust relies on edit distance between revisions Google’s Crowdsensus measure’s the accuracy of contributing users using attribute inference algorithms 10 Research Directions: How can reputation systems lead to happy, active, healthy communities? How can we build reputation systems that meet multiple goals?

Pros Highlights the importance of reputation systems for online collaboration of content. Compares how two different content-driven reputation systems fulfill very different practical needs. Gives insight on how Google approaches reputation systems, where they might not explicitly share the algorithm. E.g. PageRank 11

Cons Mentions that CrowdSensus yields better results than Bayesian approach but does not elaborate on it. Provides little detail on how the CrowdSensus algorithm works for map edits. Does not provide detials about how attribute inference models work. 12