The Scientific Community Game as A Crowdsourcing Platform to Distinguish Good from Bad Presentation to Clients by Software Development Organization 4/24/20111

Crowdsourcing Platform Crowdsourcing – is the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call. – enlists a crowd of humans to help solve a problem defined by the system owners. A crowdsourcing platform is a generic tool that makes it easy to develop a crowdsourcing system. 4/24/20112

Crowdsourcing Platform The job, target problem is – to solve instances of a problem and make claims about the solution process. – to build knowledge base of claims and techniques to defend the claims 4/24/20113

Requirements for Crowdsourcing Platform Find a good way to combine user contributions to solve the target problem. Find a good way to evaluate users and their contributions. Find a good way to recruit and retain users. 4/24/20114

Combine user contributions We found an excellent way. Users build on each others work: strengthening. Users check each others claims for correct judgment. – Claims are defended and refuted. Users trade reputation and information. 4/24/20115

Voting with Justification I vote – for this claim (agree) because I can defend it and refute its negation. – against this claim because I can oppose it (refute or strengthen). 4/24/20116

Evaluate users and their contributions Calculate reputation – confidence by the proposer that a claim is good (gc) – confidence by the opposer (refute or strengthen) that the claim is bad (bc) The scholars are encouraged to set their confidences truthfully. Otherwise they don't gain enough reputation or they lose too much reputation. 4/24/20117

Reputation Update Claimgoodbad proposeupdown opposedownup up: if you are good, there is a chance that you win down: if the other is good, there is a chance that you lose up: reputation goes up, but has to provide knowledge that might reveal secret technique. down: reputation goes down, but might gain knowledge that reveals secret technique. 4/24/20118

Reputation Update Claimgoodbad proposeupdown opposedownup up: if you are good, there is a chance that you win down: if the other is good, there is a chance that you lose confidence: proposer: claim is good: gc opposer: claim is bad: bc r = result of reputation protocol. Reputation update: r*gc*bc (various refinements are possible) 4/24/20119

Perfect Being perfect means to make perfect decisions. up: if you are perfect, you will not lose. down: if the other is perfect, you will not win. Claimgoodbad proposeupdown opposedownup up: if you are good, there is a chance that you win down: if the other is good, there is a chance that you lose 4/24/201110

Result: We have successfully developed the SCG Crowdsourcing platform: SCG Court Requirements Document User’s Guide Continuous Testing Approach Well documented software with Design Document for Maintainers Acceptance Test – CSP tournament – HSR tournament – NetworkFlow tournament 4/24/201111

Software Development Process Customized Spiral Process using continuous testing – Software Management Approach – Motivating software developers with interesting project that provides numerous learning opportunities. – Overcoming initial resistance by persistence and significant involvement of teaching staff. Adding Ahmed. 4/24/201112

Software Management Approach Hire good software developers with significant initiative and involvement in the project. Flexible project organization: very open to numerous requirements changes. Trusting software developers with significant responsibilities. Greek Gods: found subproject that was optimal for their time table. Dan Marcucci: Let’s use DemeterF the right way. 4/24/201113

Software Management Approach I amplified the good ideas that I spotted and did not force them on you! I continued to refine the game with Ahmed and Yue and we carefully justified all requirements changes. Reto: continuous testing approach. Setting up svn on assembla. 4/24/201114

Domain of Requests and Responses Three kinds of requests – Instance Requests Response: Instance Boolean valid(Instance) – Solution Requests Response: Solution Boolean valid(Instance,Solution) – Claim Analysis Requests Response: good/bad decision determine which claims are – good: agree – bad: refute or strengthen Make prediction about how protocol will run – Claims are about the relationship between instances and their solutions 4/24/201115

What is good/bad? A claim is – good if both scholars agree if it is predominantly defended – bad if scholars don’t agree if it is predominantly refuted Refutation is the complement of defense and is based on the requests and responses exchanged. 4/24/201116

Good/ Bad Meaning for Mathematical Claims Good – true claim: if there is no strengthening relation between claims (strengthen(c1,c2) is false if c1!=c2.) – optimal claim: if there is a strengthening relation between claims. Bad – false claim – non-optimal claim 4/24/201117