Virtual Scientific-Community-Based Foundations for Popperian e-Science Karl Lieberherr Ahmed Abdelmeged Northeastern University, CCIS, PRL, Boston 12/3/2018

inspired by ScienceWISE Ontology Mathematics CS Mathematical Logic Programming Game Theory MetaGaming ExtensiveForm Socio-Technical System The Global Brain Dialog Games IF Logic 12/3/2018

A claim is … information about one’s performance when interacting with another clever being in a specific domain. information about the performance of one’s program. 4/24/2011 Crowdsourcing

Outline Introduction Theoretical Background Methods of Exploration Theory Methods of Exploration Methods Results Results Conclusions and Future Work Conclusion 12/3/2018

Introduction Theory Methods Results Conclusion Introduction SCG = Scientific Community Game = Specker Challenge Game Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, tunedit.org, Renaissance, … Make SCG a part of cyber-infrastructure (e-science) to support teaching and innovation in constructive domains. SCG usage for teaching Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis Avatar competitions are not for teaching (but for competitive innovation) Theoretical Properties of SCG 12/3/2018

Popper One of the philosophers of science who has had a big impact. Introduction Theory Methods Results Conclusion Popper One of the philosophers of science who has had a big impact. Popper’s solution: Falsification: A claim is falsifiable if you can imagine an observation that would cause you to reject the claim. That a claim is "falsifiable" does not mean it is false; rather, that if it is false, then some observation or experiment will produce a reproducible result that is in conflict with it. 12/3/2018

Introduction Theory Methods Results Conclusion What SCG helps with Build and maintain knowledge bases (sets of claims believed to be true). How to identify experts? How to decide if an answer is worthwhile? Use scholars to choose the winners How to organize egoistic scholars to produce social welfare: knowledge base and know-how how to defend it. The scholars try to reverse engineer the solutions of winning scholars. 12/3/2018

Abstraction from 4 Examples From a CS journal paper Insilico experiment From kaggle.com: Facebook competition From a calculus problem 12/3/2018

Example 1: From an Abstract of a 2005 Journal Paper An instance of a constraint satisfaction problem (CSP) is variable k-consistent if any subinstance with at most k variables has a solution. For a fixed constraint language L, r(k,L) is the largest ratio such that any variable k-consistent instance has a solution that satisfies at least a fraction of r(k,L) of the constraints. 12/3/2018

Example 1 From a 2005 TCS paper: Locally Consistent Constraint Satisfaction Problems by Manuel Bodirsky and Daniel Kral. Example L = CNF k = 1 What is r(1,CNF)? Claims: r(1,CNF) = 0.6, r(1,CNF) = 0.7 12/3/2018

Example 1: Making a game to determine r(1,CNF) Observation: claims are falsifiable playing a two person game. 12/3/2018

Example 2: Claim involving Insilico Experiment Claim InsilicoExperimental(X,Y,q,r) I claim, given raw materials x in X, I can produce product y in Y of quality q and using resources at most r. 4/24/2011 Crowdsourcing

Example 2: Making a game to determine InsilicoExperimental(X,Y,q,r) Observation: claims are falsifiable playing a two person game. 12/3/2018

Example 3: Data mining Facebook competition from Kaggle.com: Introduction Theory Methods Results Conclusion Example 3: Data mining Facebook competition from Kaggle.com: Given a social network graph x with deleted edges and the original social network graph gs (secret, from a family X of social networks) guess the complete social network graph y quality(x, gs, y) = mean average precision (adapted from IR) I claim I can achieve a mean average precision of q for social graphs in family X: DM1(X,q) for a specific reduced social graph: DM2(x,q) 12/3/2018

Example 3: Making a game to determine the optimal claims Observation: claims DM1(X,q) are falsifiable playing a two person game. Claim DM2(x,q) is falsifiable when the secret is revealed. 12/3/2018

Example 4: Specker Claims: Introduction Theory Methods Results Conclusion Example 4: Specker Claims: Specker(set X, set Y(X), function f(X,Y)->[0,1], constant c): ForAll x in X Exists y in Y(X): f(x,y)≥c Example 1 X = Conjunctive Normal Forms with various restrictions Y(X) = Assignments to CNFs f(x,y) = fraction of satisfied clauses in x under y c in [0,1], e.g., c= 0.61 Example 2 (a reduction of example 1) X = [0,1] Y(X) = [0,1] f(x,y)=x*y+(1-x)(1-y^2)) c in [0,1], e.g., c=0.61 12/3/2018

Example 4: Specker Observation: claims Specker(X,Y,f,c) are falsifiable playing a two person game. 12/3/2018

What is the abstraction? Sets of claims Claims are falsifiable … 12/3/2018

Each playground defines: domain claims language specific protocol Playgrounds Each playground defines: domain claims language specific protocol data exchanged configuration data RP1 PG1 claims C11 C12 C13 … SC1 SC2 D1 RP2 PG2 claims C21 C22 C23 … SC3 SC4 SC5 SC1 D2 SCG defines: refutation protocol interface generic rules for all playgrounds 12/3/2018

Example 1: Making a game to determine r(1,CNF) Observation: claims are falsifiable playing a two person game. defendable = !refutable propose r(1,CNF) = 0.7 refutable propose r(1,CNF) = 0.6 can be strengthened to r(1,CNF) = 0.61 which is defendable (refutation attempts will be unsuccessful) propose r(1,CNF) = (sqrt(5)-1)/2 ~ 0.618 … optimum: defendable and cannot be strengthened 12/3/2018

Who are the scholars? Scientists Students in a class room High school University Members of the Gig Economy Between 1995 and 2005, the number of self-employed independent workers grew by 27 percent. Potential employees (Facebook on kaggle.com) Anyone with web access; Intelligent crowd. 12/3/2018

Kaggle.com Competitions 2012 Facebook recruiting competitions Task: Data scientist Reward: Job Teams: 197 Heritage Health Prize Task: Hospital admissions Reward: $ 3 million Teams: 1118 Chess ratings – Elo versus the Rest of the World Task: Predict outcome of chess games Reward: $ 617 Teams: 257 12/3/2018

Kaggle.com Competitions 2012 Eye Movements Verification and Identification Task: Identify people Reward: Kudos Teams: 51 EMC Data Science Global Hackathon Task: Air Quality Prediction Reward $ 7030 Teams: 114 12/3/2018

What Scholars think about! Introduction Theory Methods Results Conclusion What Scholars think about! If I propose claim C, what is the probability that C is successfully refuted C is successfully strengthened If I try to refute claim C, what is the probability that I will fail. If I try to strengthen claim C, what is the probability that I will fail? Scholars are free to invent; game rules don’t limit creativity! 12/3/2018

Degree of automation with SCG(X) Introduction Theory Methods Results Conclusion Degree of automation with SCG(X) avatar Bob scholar Alice degree of automation used by scholar 1 no automation human plays some automation human plays full automation avatar plays human plays: communicate structured messages through email (SMTP) agent plays: HTTP more applications: test constructive knowledge transfer to reliable, efficient software 12/3/2018

Organizational Problem Solved happy = can be creative, can thrive, have opportunity to learn, not ignored Organizational Problem Solved How to design a happy scientific community that encourages its members to really contribute. Control of scientific community tunable SCG rules Specific domain, claim definition to narrow scope. 12/3/2018

Playground defines what is wanted, e.g., an algorithm S in a particular domain (inputs/outputs) evaluation, e.g., how S is evaluated (quality) claims, e.g., what kind of claims can be made about S (expression with quantifiers) A playground defines WHAT is desired and the scholars/avatars define the HOW.

Theory Extensive Form Representation of Game Properties Introduction Theory Methods Results Conclusion Theory Extensive Form Representation of Game Properties Community Property: All faulty actions can be exposed. SCG Equilibrium Convergence to optimum claim 12/3/2018

Extensive-form representation Introduction Theory Methods Results Conclusion Extensive-form representation the players of a game: 1 and 2 for every player every opportunity they have to move what each player can do at each of their moves what each player knows for every move the payoffs received by every player for every possible combination of moves 12/3/2018

Large Action Spaces Thick arrows mean: select from a usually large number of choices 1 2 12/3/2018

Refutation Protocol Collects data given to predicate p. Alternates. Introduction Theory Methods Results Conclusion Refutation Protocol Collects data given to predicate p. Alternates. refute(C,proposer,other) other tries to make p false while proposer tries to make p true. p false means successful refutation. p true means successful defense. p(C, …)?(1,-1):(-1,1) claim payoff for proposer if p true (defense) payoff for other if p true (defense) payoff for proposer if p false (refutation) payoff for other if p false (refutation) 12/3/2018

SCG Core p(C, …)?(1,-1):(-1,1) p(C, …)?(0,0):(1,-1) Introduction Theory Methods Results Conclusion 1 scholar 2 scholar 1 SCG Core refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) propose claim C from Claims agree attempt C 2 refute attempt C strengthen attempt C’ => C refute(C,1,2) p(C, …)?(1,-1):(-1,1) refute(C’,2,1) refute(C,2,1) p(C, …)?(1,-1):(-1,1) p(C’, …)?(1,-1):(-1,1) p(C, …)?(0,0):(1,-1) p(C’, …)?(-1,1):(1,-1) 12/3/2018

Game Rules for Playground Introduction Theory Methods Results Conclusion Game Rules for Playground All objects exchanged during protocol must be legal and valid. Each move must be within time-limit. Scholar who first violates a playground rule, loses. 12/3/2018

not just true/false claims, but optimum/non-optimum claims: Introduction Theory Methods Results Conclusion Logic with Soundness claims sentences good bad not just true/false claims, but optimum/non-optimum claims: good: true/optimum bad: false/non-optimum 4/24/2011 Crowdsourcing

Scientific Community Game Logic with Community Principle Introduction Theory Methods Results Conclusion Scientific Community Game Logic with Community Principle claims sentences good bad disagreed by two scholars agreed by two scholars there exists a two-party certificate to expose misclassification 4/24/2011 Crowdsourcing

Comparison Logic and SCG Scientific Community Game sentences true false proof for being true proof system, checkable guaranteed defense proof for being false guaranteed refutation Universal sentences sentences = claims good bad evidence for goodness defense, checkable uncertainty of defense evidence for badness refutation, checkable uncertainty of refutation Personified sentences 4/24/2011 Crowdsourcing

Introduction Theory Methods Results Conclusion Community Property For every faulty decision action there exists an exposing reaction that blames the bad decision. Reasons: We want the system to be egalitarian. It is important that clever crowd members can shine and expose others who don’t promote the social welfare of the community. Faulty decisions must be exposable. It may take effort. 12/3/2018

Begin delete 12/3/2018

Community Property Alternative formulation If all decisions by Alice are not faulty, there is no chance of Alice losing against Bob. if Alice is perfect, there is no chance of losing. If there exists a faulty decision by Alice, there is a chance of Alice losing against Bob. egalitarian game 12/3/2018

Summary: faulty decisions propose(Alice,C),C=false propose(Alice,C),C=not optimum, C=true refute(Alice,Bob,C),C=true strengthen(Alice,Bob,c,cs),c=optimum strengthen(Alice,Bob,c,cs),c=false agree(Alice,Bob,c),c=false agree(Alice,Bob,c),c=not optimum, c=true 12/3/2018

SCG Equilibrium Points (reputations) of scholars are stable. Introduction Theory Methods Results Conclusion SCG Equilibrium Points (reputations) of scholars are stable. The science does not progress; bugs are not fixed, no new ideas are introduced. Extreme, desirable situation: All scholars are perfect: they propose optimal claims that can neither be strengthened nor refuted. 12/3/2018

Claims: convergence to optimum 1 over strengthening correct valuation false claims (refutable) quality strengthening true claims (defendable) 12/3/2018 42

Introduction Theory Methods Results Conclusion Convergence Given a family of claims totally ordered with respect to strengthening and containing an optimum claim. If every faulty action is exposed, convergence to the optimum claim is guaranteed by using a search strategy such as binary search. 12/3/2018

end delete 12/3/2018

Methods of Exploration Introduction Theory Methods Results Conclusion Methods of Exploration Developed Platform SCG Court = Generator of teaching/innovation playgrounds http://sourceforge.net/p/generic-scg/code-0/11 0/tree/GenericSCG/ Developed numerous playgrounds for avatars. Developed Algorithms Course using Piazza based on SCG Court experience role of scholar played by humans piazza.com: encourages students to answer each other’s questions. 12/3/2018

Avatar Interface AvatarI Introduction Theory Methods Results Conclusion Avatar Interface AvatarI public List<Claim> propose(List<Claim> forbiddenClaims); public List<OpposeAction> oppose(List<Claim> claimsToBeOpposed); public InstanceI provide(Claim claimToBeProvided); public SolutionI solve(SolveRequest solveRequest); from http://sourceforge.net/p/generic-scg/code-0/110/tree/GenericSCG/src/scg/scg.beh 12/3/2018

Instance Interface (Domain) Introduction Theory Methods Results Conclusion Instance Interface (Domain) InstanceI boolean valid(SolutionI solution, Config config); double quality(SolutionI solution); 12/3/2018

InstanceSet Interface (Domain) Option<String> belongsTo(InstanceI instance); Option<String> valid(Config config); }} 12/3/2018

Protocol Interface ProtocolI double getResult(Claim claim, SolutionI[] solutions, InstanceI[] instances); ProtocolSpec getProtocolSpec(); boolean strengthenP(Claim oldClaim, Claim strengthenedClaim); 12/3/2018

Claim Class, for all playgrounds public Claim(InstanceSetI instanceSet, ProtocolI protocol, double quality, double confidence) 12/3/2018

Protocol Library ExistsForAll.java ForAllExists.java Renaissance.java Introduction Theory Methods Results Conclusion Protocol Library ExistsForAll.java ForAllExists.java Renaissance.java AsGoodAsYou.java Survivor.java 12/3/2018

Begin delete 12/3/2018

Claim Kinds in SCG Court Introduction Theory Methods Results Conclusion Claim Kinds in SCG Court Claim C(instance, q) Claim C(InstanceSet, q) Claim MaxResource(Algorithm,InstanceSet,n,ResExp) Claim MinResource(Algorithm,InstanceSet,n,ResExp) Claim IAmTheBest(), AtLeastAsGoodAsYou() Claims from predicate logic and some of its generalizations (IF Logic) 12/3/2018

Reverse Engineering Playground Design Issues input, output, algorithm unknown for some problems input-output pairs help to invent the algorithm input unknown, output unknown, algorithm e.g., output for worst-case running-time, find input which is worst-case 12/3/2018

Introduction Theory Methods Results Conclusion Playground Design Make sure that the scholars don’t have to reveal their secrets. If they do. Remember who was first? Provenance of winning idea. 12/3/2018

Second Exploration with piazza.com forum with threads (one per claim and protocol execution) 12/3/2018

SCG with piazza.com use piazza.com for posting Community policing Introduction Theory Methods Results Conclusion SCG with piazza.com use piazza.com for posting playground definition by playground designer for scholars claims executions of refutation protocols Community policing Used JSON to exchange claims, instances and solutions. Used links when they did not fit in piazza window. 12/3/2018

Piazza: Gale-Shapley Gale-Shapley (GS) algorithm is fixed. Alice provides secret instance of size n, solves it with GS (counts number of iterations) and publishes it: Ac(n). Bob provides secret instance of size n, solves it with GS and publishes it Bc(n). Alice wins if Ac(n) > Bc(n). 12/3/2018

Introduction Theory Methods Results Conclusion JSON example We define the JSON notation for defining a preference p as follows: {"n":3, "manPref" : [[2,1,0],[1,0,2],[0,1,2]], "womanPref : [[2,1,0],[1,3,2],[3,1,2]] } Easy to parse in a variety of programming languages and also readable by humans. 12/3/2018

end delete 12/3/2018

Piazza Experience Gale-Shapley We propose that, for all integers n > 0, the maximum iterations the Gale-Shapely algorithm with n men and n women can produce is n(n-1)+1. Note: Thus far, the inputs used for all other claims arrives at only (n(n+1))/2. 12/3/2018

Piazza Experience Leaf Covering: Improved running time from quadratic to constant time. 12/3/2018

Introduction Theory Methods Results Conclusion Results SCG = Scientific Community Game = Specker Challenge Game Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, Operations Research Competitions, tunedit.org, http://eterna.cmu.edu/ … SCG usage for teaching using forum Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis Avatar competitions are not for teaching (but good for competitive innovation) Theoretical Properties of SCG 12/3/2018

Competition tuning: minimum For each scholar count claims that were successfully opposed (refuted or strengthened) encourages strong claims gather information from competitors for free count claims that were not successfully agreed Good for teaching students want minimum competition good students want to build social capital and help weaker students 12/3/2018

High competition p(C, …)?(1,-1):(-1,1) p(C, …)?(0,0):(1,-1) Introduction Theory Methods Results Conclusion 1 scholar 2 scholar 1 High competition refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) propose claim C from Claims agree attempt C 2 refute attempt C strengthen attempt C’ => C refute(C,1,2) p(C, …)?(1,-1):(-1,1) refute(C’,2,1) refute(C,2,1) p(C, …)?(0,0):(1,-1) p(C’, …)?(-1,1):(1,-1) 12/3/2018

Low competition p(C, …)? (0,0) :(0,1) p(C, …)?(0,0): (1,0) Introduction Theory Methods Results Conclusion 1 scholar 2 scholar 1 Low competition refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) s: successful u: unsuccessful propose claim C from Claims agree attempt C 2 refute attempt C strengthen attempt C’ => C refute(C,1,2) p(C, …)? (0,0) :(0,1) refute(C’,2,1) refute(C,2,1) p(C, …)?(0,0): (1,0) p(C’, …)?(0,1): (0,0) 12/3/2018

end delete 12/3/2018

Piazza Results Lower competition knob for teaching. Introduction Theory Methods Results Conclusion Piazza Results Lower competition knob for teaching. For optimization claims got significant scientific discourse. Playgrounds cannot have too many scholars, otherwise they are overwhelmed. about 5 is a good size use hierarchical playgrounds: winning teams compete again 12/3/2018

Introduction Theory Methods Results Conclusion Piazza Results Do not give hints at solutions. This significantly decreased the amount of discourse taking place. 12/3/2018

Conclusions and Future Work Introduction Theory Methods Results Conclusion Conclusions and Future Work We propose a systematic gamification of teaching STEM domains: Design an SCG playground where the winning students demonstrate superior domain knowledge. STEM = Science, Technology, Engineering, and Mathematics 12/3/2018

Gamification of Software Development for Computational Problems Introduction Theory Methods Results Conclusion Gamification of Software Development for Computational Problems Want reliable software to solve a computational problem? Design an SCG playground where the winning team will create the software you want. playground design = requirements Programming the Global Brain socio-technical system (playground) will produce solution to requirements. 4/24/2011 Crowdsourcing

Introduction Theory Methods Results Conclusion Conclusions Flexible use of SCG using a forum environment with threads and replies using optimization playgrounds is productive: teams took turns leapfrogging each other reached state-of-the-art and even improved it SCG has desirable theoretical properties. faulty decision –> exposing reaction equilibria convergence to optimum claim 12/3/2018

Introduction Theory Methods Results Conclusion Future Work Make SCG part of cyber-infrastructure (e-science) both for avatars and human scholars. Polish SCG Court The administrator software needs to be very reliable (to avoid cheating by avatars). Playground development and testing needs tool support. Further develop SCG with forum software Playground design defines requirements for know-how. Hierarchical playgrounds: partitioning into balanced groups. Restart playground after publishing all current ideas in playground (if optimum is not yet reached). 12/3/2018

Links / Questions SCG Home Piazza page for Algorithms Algorithms Home http://www.ccs.neu.edu/home/lieber/evergreen/specker/scg-home.html Piazza page for Algorithms http://piazza.com/class#winter2012/cs4800/0 Algorithms Home http://www.ccs.neu.edu/home/lieber/courses/algorithms/cs4800/sp12/course-description.html Algorithms Feedback http://www.ccs.neu.edu/home/lieber/courses/algorithms/cs4800/sp12/feedback/ SCG Court Source http://sourceforge.net/p/generic-scg/code-0/110/tree/GenericSCG/ 12/3/2018

The End More Questions? 12/3/2018

Extra slides 12/3/2018

Essence of Game Rules without Payoff blamed decisions: propose(1,C) refute(1,2,C) strengthen(1,2,C,C’) agree(1,2,c) Essence of Game Rules without Payoff scholars: 1, 2 LifeOfClaim(C) = propose(1,C) followed by (oppose(1,2,C)|agree(1,2,C)). oppose(1,2,C) = (refute(1,2,C)|strengthen(1,2,C,C’)), where stronger(C,C’). strengthen(1,2,C,C’) = !refute(2,1,C’). agree(1,2,C) = !refute(2,1,C) 12/3/2018

p(C, …)?(1,-1):(-1,1) p(C, …)?(0,0):(1,-1) p(C’, …)?(-1,1):(1,-1) Introduction Theory Methods Results Conclusion 1 scholar 2 scholar 1 refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) s: successful u: unsuccessful propose claim C from Claims agree attempt C 2 refute attempt C strengthen attempt C’ => C refute(C,1,2) p(C, …)?(1,-1):(-1,1) refute(C’,2,1) u:1 2 s:1 2 refute(C,2,1) p(C, …)?(1,-1):(-1,1) p(C’, …)?(1,-1):(-1,1) p(C, …)?(0,0):(1,-1) p(C’, …)?(-1,1):(1,-1) s:1 2 u:1 2 s:1 2 u:1 2 12/3/2018

Low competition p(C, …)? (0,0) :(0,1) p(C, …)?(0,0): (1,0) Introduction Theory Methods Results Conclusion 1 scholar 2 scholar 1 Low competition refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) s: successful u: unsuccessful propose claim C from Claims agree attempt C 2 refute attempt C strengthen attempt C’ => C refute(C,1,2) p(C, …)? (0,0) :(0,1) refute(C’,2,1) u:1 2 s:1 2 refute(C,2,1) p(C, …)?(0,0): (1,0) p(C’, …)?(0,1): (0,0) s:1 2 u:1 2 s:1 2 u:1 2 12/3/2018

Conclusions for Teaching Introduction Theory Methods Results Conclusion Conclusions for Teaching Transition refute: (1,-1):(-1,1) -> (0,0) :(0,1) strengthen: (-1,1):(1,-1) -> (0,1): (0,0) agree: (0,0):(1,-1) -> (0,0): (1,0) creates better playgrounds for learning by lowering competition and increasing teaching between scholars. 12/3/2018

Claims Protocol. Defines scientific discourse. Scholars make a prediction about their performance in protocol. Predicate that decides whether refutation is successful. Refutation protocol collects data for predicate. As a starter: Think of a claim as a mathematical statement: EA or AE. all planar graphs have a 4 coloring. 12/3/2018

More examples of Protocols Introduction Theory Methods Results Conclusion More examples of Protocols Let f(x,y)=x*y+(1-x)(1-y^2)). Alice claims Math(0.61): Bob constructs an x in [0,1] and Alice constructs a y in [0,1], and Alice guarantees that f(x,y)> 0.61. True claim but can be strengthened to 0.618. Alice claims Solar(RawMaterials,m,0.61). Bob constructs raw materials r in RawMaterials and Alice constructs a solar cell s in Solution from r using money m and so that efficiency(s)> 0.61. 12/3/2018

Questions received In learning game, give credit to all contributors, not just final one (DARPA 10 ball challenge) Predicate logic -> SCG: make explicit Playground design: involve competitors 12/3/2018

Questions Credit first time the best claim is made linear order by time linear order by strength (quality) 12/3/2018

New insight Need to know very little about refutation protocol. collect data, what is available when is not important evaluate predicate with collected data 12/3/2018

What is a loose collaboration? Introduction Theory Methods Results Conclusion What is a loose collaboration? Scholars can work independently on an aspect of the same problem. Problem = decide which claims in playground to oppose or agree with. How is know-how combined? Using a protocol. Alice claimed that for the input that Alice provides, Bob cannot find an output of quality q. But Bob finds such an output. Alice corrects. Bug reports that need to be addressed and corrections. Playground = Instantiation of Platform 12/3/2018

Example: Independent Set Alice = proposer, Bob = other. Protocol / claim: AtLeastAsGood. Alice claims to be at least as good as Bob at IS. Bob provides undirected graph G. Bob computes independent set sB for G (secret). Alice computes independent set sA for G. Alice wins, if size(sA) >= size(sB) (= p(sA,sB)). 12/3/2018

Introduction Theory Methods Results Conclusion Specker Claims: Specker(set X, set Y(X), function f(X,Y)->[0,1], constant c): ForAll x in X Exists y in Y(X): f(x,y)≥c Example 1 X = Conjunctive Normal Forms with various restrictions Y(X) = Assignments to CNFs f(x,y) = fraction of satisfied clauses in x under y c in [0,1], e.g., c= 0.61 Example 2 (a reduction of example 1) X = [0,1] Y(X) = [0,1] f(x,y)=x*y+(1-x)(1-y^2)) c in [0,1], e.g., c=0.61 12/3/2018

Kaggle.com Facebook competition: Introduction Theory Methods Results Conclusion Kaggle.com Facebook competition: X = Social Network Graph with deleted edges, Original Social Network Graph (secret) Y(X) = estimated complete Social Network Graph quality(x,y) = mean average precision adapted from IR 12/3/2018

Simpler talk Introduction: parameterized models of scientific communities Theory 12/3/2018

Abstraction from 4 Examples From a CS journal paper Insilico experiment From kaggle.com: Facebook competition From a calculus problem 12/3/2018

Example 1: From an Abstract of a 2005 Journal Paper An instance of a constraint satisfaction problem (CSP) is variable k-consistent if any subinstance with at most k variables has a solution. For a fixed constraint language L, r(k,L) is the largest ratio such that any variable k-consistent instance has a solution that satisfies at least a fraction of r(k,L) of the constraints. 12/3/2018

Example 1 From a 2005 TCS paper: Locally Consistent Constraint Satisfaction Problems by Manuel Bodirsky and Daniel Kral. Example L = CNF k = 1 What is r(1,CNF)? Claims: r(1,CNF) = 0.6, r(1,CNF) = 0.7 12/3/2018

Example 1: Making a game to determine r(1,CNF) Observation: claims are falsifiable playing a two person game. 12/3/2018

Example 2: Claim involving Insilico Experiment Claim InsilicoExperimental(X,Y,q,r) I claim, given raw materials x in X, I can produce product y in Y of quality q and using resources at most r. 4/24/2011 Crowdsourcing

Example 2: Making a game to determine InsilicoExperimental(X,Y,q,r) Observation: claims are falsifiable playing a two person game. 12/3/2018

Example 3: Data mining Facebook competition from Kaggle.com: Introduction Theory Methods Results Conclusion Example 3: Data mining Facebook competition from Kaggle.com: Given a social network graph x with deleted edges and the original social network graph gs (secret, from a family X of social networks) guess the complete social network graph y quality(x, gs, y) = mean average precision (adapted from IR) I claim I can achieve a mean average precision of q for social graphs in family X: DM1(X,q) for a specific reduced social graph: DM2(x,q) 12/3/2018

Example 3: Making a game to determine the optimal claims Observation: claims DM1(X,q) are falsifiable playing a two person game. Claim DM2(x,q) is falsifiable when the secret is revealed. 12/3/2018

Example 4: Specker Claims: Introduction Theory Methods Results Conclusion Example 4: Specker Claims: Specker(set X, set Y(X), function f(X,Y)->[0,1], constant c): ForAll x in X Exists y in Y(X): f(x,y)≥c Example 1 X = Conjunctive Normal Forms with various restrictions Y(X) = Assignments to CNFs f(x,y) = fraction of satisfied clauses in x under y c in [0,1], e.g., c= 0.61 Example 2 (a reduction of example 1) X = [0,1] Y(X) = [0,1] f(x,y)=x*y+(1-x)(1-y^2)) c in [0,1], e.g., c=0.61 12/3/2018

Example 4: Specker Observation: claims Specker(X,Y,f,c) are falsifiable playing a two person game. 12/3/2018

What is the abstraction? Sets of claims Claims are falsifiable … 12/3/2018

Example 1: Making a game to determine r(1,CNF) Observation: claims are falsifiable playing a two person game. defendable = !refutable propose r(1,CNF) = 0.7 refutable propose r(1,CNF) = 0.6 can be strengthened to r(1,CNF) = 0.61 which is defendable (refutation attempts will be unsuccessful) propose r(1,CNF) = (sqrt(5)-1)/2 ~ 0.618 … optimum: defendable and cannot be strengthened 12/3/2018

What we get Engaged software developers Clear Feedback Sense of Progress What we get Authenticity Engaged software developers let them produce software that models an organism that fends for itself in a real virtual world while producing the software we want. Have fun. Focus them. let them propose claims about the software they produce. Reward them when they defend their claims successfully or oppose the claims of others successfully. Possibility of Success 4/24/2011 Crowdsourcing

Reinterpret Refutation Refutation leads to successful strengthening or successful agreement. 12/3/2018