A Popperian Platform for Programming and Teaching the Global Brain Karl Lieberherr Ahmed Abdelmeged Northeastern University, CCIS, PRL, Boston 10/9/20151

A claim is … information about one’s performance when interacting with another clever being in a specific domain. information about the performance of one’s program. 4/24/20112Crowdsourcing

Outline Theory Methods Results Conclusion Introduction Theoretical Background Methods of Exploration Results Conclusions and Future Work 10/9/20153

Introduction Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, tunedit.org, Renaissance, … Make SCG a part of cyber-infrastructure (e-science) to support teaching and innovation in constructive domains. SCG usage for teaching – Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis – Avatar competitions are not for teaching (but for competitive innovation) Theoretical Properties of SCG 10/9/20154 Introduction Theory Methods Results Conclusion SCG = Scientific Community Game = Specker Challenge Game

Specker Claims: – Specker(set X, set Y(X), function f(X,Y)->[0,1], constant c): ForAll x in X Exists y in Y(X): f(x,y)≥c Example 1 – X = Conjunctive Normal Forms with various restrictions – Y(X) = Assignments to CNFs – f(x,y) = fraction of satisfied clauses in x under y – c in [0,1], e.g., c= 0.61 Example 2 – X = [0,1] – Y(X) = [0,1] – f(x,y)=x*y+(1-x)(1-y^2)) – c in [0,1], e.g., c= /9/20155 Introduction Theory Methods Results Conclusion

Popper One of the philosophers of science who has had the biggest impact. Popper’s solution: Falsificationism: A claim is falsifiable if you can imagine an observation that would cause you to reject the theory. That a claim is "falsifiable" does not mean it is false; rather, that if it is false, then some observation or experiment will produce a reproducible result that is in conflict with it. 10/9/20156 Introduction Theory Methods Results Conclusion

What SCG helps with How to identify experts? How to decide if an answer is worthwhile? –Use scholars to choose the winners How to organize egoistic scholars to produce social welfare: knowledge base and know-how how to defend it. –The scholars try to reverse engineer the solutions of winning scholars. 10/9/20157 Introduction Theory Methods Results Conclusion

Claim involving Experiment Claim ExperimentalTechnique(X,Y,q,r) I claim, given raw materials x in X, I can produce product y in Y of quality q and using resources at most r. 8Crowdsourcing4/24/2011

Who are the scholars? Scientists Students in a class room – High school – University Members of the Gig Economy – Between 1995 and 2005, the number of self- employed independent workers grew by 27 percent. Potential employees (Facebook on kaggle.com) Anyone with web access; Intelligent crowd. 10/9/20159

Kaggle.com Competitions 2012 Facebook recruiting competitions – Task: Data scientist – Reward: Job – Teams: 197 Heritage Health Prize – Task: Hospital admissions – Reward: $ 3 million – Teams: 1118 Chess ratings – Elo versus the Rest of the World – Task: Predict outcome of chess games – Reward: $ 617 – Teams: /9/201510

Kaggle.com Competitions 2012 Eye Movements Verification and Identification – Task: Identify people – Reward: Kudos – Teams: 51 EMC Data Science Global Hackathon – Task: Air Quality Prediction – Reward $ 7030 – Teams: /9/201511

What Scholars think about! If I propose claim C, what is the probability that – C is successfully refuted – C is successfully strengthened If I try to refute claim C, what is the probability that I will fail. If I try to strengthen claim C, what is the probability that I will fail? Scholars are free to invent; game rules don’t limit creativity! 1210/9/2015 Introduction Theory Methods Results Conclusion

Degree of automation with SCG(X) 13 no automation human plays full automation avatar plays degree of automation used by scholar some automation human plays 0 1 more applications: test constructive knowledge transfer to reliable, efficient software avatar Bob scholar Alice 10/9/2015 Introduction Theory Methods Results Conclusion

Organizational Problem Solved How to design a happy scientific community that encourages its members to really contribute. Control of scientific community – tunable SCG rules – Specific domain, claim definition to narrow scope. 10/9/ happy = can contribute, not ignored, can be creative, can thrive, have opportunity to learn

What is a loose collaboration? Scholars can work independently on an aspect of the same problem. Problem = decide which claims in playground to oppose or agree with. How is know-how combined? Using a protocol. – Alice claimed that for the input that Alice provides, Bob cannot find an output of quality q. But Bob finds such an output. Alice corrects. – Bug reports that need to be addressed and corrections. 10/9/ Playground = Instantiation of Platform Introduction Theory Methods Results Conclusion

Theory Extensive Form Representation of Game Community Property: All faulty actions can be exposed. SCG Equilibrium Convergence to optimum claim 10/9/ Introduction Theory Methods Results Conclusion

Extensive-form representation 1.the players of a game: 1 and 2 2.for every player every opportunity they have to move 3.what each player can do at each of their moves 4.what each player knows for every move 5.the payoffs received by every player for every possible combination of moves 10/9/ Introduction Theory Methods Results Conclusion

Large Action Spaces Thick arrows mean: select from a usually large number of choices /9/201518

Refutation Protocol Collects data given to predicate p. Alternates. refute(C,proposer,other) p(C, …)?(1,-1):(-1,1) claimpayoff for proposer if p true (defense) payoff for other if p true (defense) payoff for other if p false (refutation) payoff for proposer if p false (refutation) other tries to make p false while proposer tries to make p true. p false means successful refutation. p true means successful defense. 10/9/ Introduction Theory Methods Results Conclusion

1 propose claim C from Claims 2 refute(C,1,2) p(C, …)?(1,-1):(-1,1) 1 scholar 2 scholar strengthen attempt C’ => C refute(C’,2,1) agree attempt C refute(C,2,1) p(C’, …)?(1,-1):(-1,1) p(C, …)?(1,-1):(-1,1) 10/9/2015 refute attempt C refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) Introduction Theory Methods Results Conclusion p(C’, …)?(-1,1):(1,-1) p(C, …)?(0,0):(1,-1) 20

Reinterpret Refutation Refutation leads to successful strengthening or successful agreement. 10/9/201521

Game Rules for Playground All objects exchanged during protocol must be legal and valid. Each move must be within time-limit. Scholar who first violates a playground rule, loses. 10/9/ Introduction Theory Methods Results Conclusion

Example: Independent Set Alice = proposer, Bob = other. Protocol / claim: AtLeastAsGood. Alice claims to be at least as good as Bob at IS. – Bob provides undirected graph G. – Bob computes independent set sB for G (secret). – Alice computes independent set sA for G. – Alice wins, if size(sA) >= size(sB) (= p(sA,sB)). 10/9/201523

4/24/2011Crowdsourcing24 goodbad Logic with Soundness claims sentences not just true/false claims, but optimum/non-optimum claims: good: true/optimum bad: false/non-optimum Introduction Theory Methods Results Conclusion

bad 4/24/2011Crowdsourcing25 good Scientific Community Game Logic with Community Principle agreed by two scholars disagreed by two scholars there exists two-party certificate to expose misclassification claims sentences Introduction Theory Methods Results Conclusion

Comparison Logic and SCG Logic sentences – true – false proof for being true – proof system, checkable – guaranteed defense proof for being false – proof system, checkable – guaranteed refutation Universal sentences Scientific Community Game sentences = claims – good – bad evidence for goodness – defense, checkable – uncertainty of defense evidence for badness – refutation, checkable – uncertainty of refutation Personified sentences 4/24/2011Crowdsourcing26

Community Property For every faulty decision action there exists an exposing reaction that blames the bad decision. – Reasons: We want the system to be egalitarian. – It is important that clever crowd members can shine and expose others who don’t promote the social welfare of the community. Faulty decisions must be exposable. It may take effort. 10/9/ Introduction Theory Methods Results Conclusion

Community Property Alternative formulation If all decisions by Alice are not faulty, there is no chance of Alice losing against Bob. – if Alice is perfect, there is no chance of losing. If there exists a faulty decision by Alice, there is a chance of Alice losing against Bob. – egalitarian game 10/9/201528

Summary: faulty decisions 1.propose(Alice,C),C=false 2.propose(Alice,C),C=not optimum, C=true 3.refute(Alice,Bob,C),C=true 4.strengthen(Alice,Bob,c,cs),c=optimum 5.strengthen(Alice,Bob,c,cs),c=false 6.agree(Alice,Bob,c),c=false 7.agree(Alice,Bob,c),c=not optimum, c=true 10/9/201529

SCG Equilibrium Points (reputations) of scholars are stable. The science does not progress; bugs are not fixed, no new ideas are introduced. Extreme, desirable situation: All scholars are perfect: they propose optimal claims that can neither be strengthened nor refuted. 3010/9/2015 Introduction Theory Methods Results Conclusion

Claims: convergence to optimum quality strengthening correct valuation over strengthening true claims (defendable) false claims (refutable) 10/9/2015

Convergence Given a family of claims totally ordered with respect to strengthening and containing an optimum claim. If every faulty action is exposed, convergence to the optimum claim is guaranteed by using a search strategy such as binary search. 10/9/ Introduction Theory Methods Results Conclusion

Methods of Exploration Developed Platform SCG Court = Generator of teaching/innovation playgrounds – 0/tree/GenericSCG/ – Developed numerous playgrounds for avatars. Developed Algorithms Course using Piazza based on SCG Court experience – role of scholar played by humans – piazza.com: encourages students to answer each other’s questions. 10/9/ Introduction Theory Methods Results Conclusion

Avatar Interface AvatarI – public List propose(List forbiddenClaims); – public List oppose(List claimsToBeOpposed); – public InstanceI provide(Claim claimToBeProvided); – public SolutionI solve(SolveRequest solveRequest); from 0/110/tree/GenericSCG/src/scg/scg.beh Introduction Theory Methods Results Conclusion 10/9/201534

Instance Interface InstanceI – boolean valid(SolutionI solution, Config config); – double quality(SolutionI solution); Introduction Theory Methods Results Conclusion 10/9/201535

InstanceSet Interface InstanceSetI – Option belongsTo(InstanceI instance); – Option valid(Config config); }} 10/9/201536

Protocol Interface ProtocolI – double getResult(Claim claim, SolutionI[] solutions, InstanceI[] instances); – ProtocolSpec getProtocolSpec(); – boolean strengthenP(Claim oldClaim, Claim strengthenedClaim); 10/9/201537

Claim Class, for all playgrounds Claim – public Claim(InstanceSetI instanceSet, ProtocolI protocol, double quality, double confidence) 10/9/201538

Protocol Library ExistsForAll.java ForAllExists.java Renaissance.java AsGoodAsYou.java Survivor.java 10/9/ Introduction Theory Methods Results Conclusion

Claim Kinds in SCG Court Claim C(instance, q) Claim C(InstanceSet, q) Claim MaxResource(Algorithm,InstanceSet,n,ResExp) Claim MinResource(Algorithm,InstanceSet,n,ResExp) Claim IAmTheBest(), AtLeastAsGoodAsYou() Claims from predicate logic and some of its generalizations (IF Logic) 10/9/ Introduction Theory Methods Results Conclusion

Piazza: Gale-Shapley Gale-Shapley (GS) algorithm is fixed. Alice provides secret instance of size n, solves it with GS (counts number of iterations) and publishes it: Ac(n). Bob provides secret instance of size n, solves it with GS and publishes it Bc(n). Alice wins if Ac(n) > Bc(n). 10/9/201541

Reverse Engineering Playground Design Issues input, output, algorithm unknown – for some problems input-output pairs help to invent the algorithm input unknown, output, algorithm – e.g., output for worst-case running-time, find input which is worst-case – for some problems finding 10/9/201542

Playground Design Make sure that the scholars don’t have to reveal their secrets. If they do. Remember who was first? Provenance of winning idea. 10/9/ Introduction Theory Methods Results Conclusion

SCG with piazza.com use piazza.com for posting – playground definition by playground designer – for scholars claims executions of refutation protocols Community policing Used JSON to exchange claims, instances and solutions. Used links when they did not fit in piazza window. 10/9/ Introduction Theory Methods Results Conclusion

JSON example We define the JSON notation for defining a preference p as follows: – {"n":3, – "manPref" : [[2,1,0],[1,0,2],[0,1,2]], – "womanPref : [[2,1,0],[1,3,2],[3,1,2]] } Easy to parse in a variety of programming languages and also readable by humans. 10/9/ Introduction Theory Methods Results Conclusion

Piazza Experience Gale-Shapley We propose that, for all integers n > 0, the maximum iterations the Gale-Shapely algorithm with n men and n women can produce is n(n-1)+1. Note: Thus far, the algorithm used for all other claims arrives at only (n(n+1))/2. 10/9/201546

Piazza Experience Leaf Covering: Improved running time from quadratic to constant time. 10/9/201547

Results Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, Operations Research Competitions, tunedit.org, … SCG usage for teaching using forum – Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis Avatar competitions are not for teaching (but good for competitive innovation) Theoretical Properties of SCG 10/9/ SCG = Scientific Community Game = Specker Challenge Game Introduction Theory Methods Results Conclusion

Competition tuning: minimum For each scholar – count claims that were successfully opposed (refuted or strengthened) encourages strong claims gather information from competitors for free – count claims that were not successfully agreed Good for teaching – students want minimum competition – good students want to build social capital and help weaker students 10/9/201549

1 propose claim C from Claims 2 refute(C,1,2) p(C, …)?(1,-1):(-1,1) 1 scholar 2 scholar strengthen attempt C’ => C refute(C’,2,1) agree attempt C refute(C,2,1) 10/9/2015 refute attempt C refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) p(C’, …)?(-1,1):(1,-1) p(C, …)?(0,0):(1,-1) High competition Introduction Theory Methods Results Conclusion 50

1 propose claim C from Claims 2 refute(C,1,2) p(C, …)? (0,0) :(0,1) 1 scholar 2 scholar strengthen attempt C’ => C refute(C’,2,1) agree attempt C refute(C,2,1) 10/9/2015 refute attempt C refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) s: successful u: unsuccessful p(C’, …)?(0,1): (0,0) p(C, …)?(0,0): (1,0) Low competition Introduction Theory Methods Results Conclusion 51

Piazza Results Lower competition knob for teaching. For optimization claims got significant scientific discourse. Playgrounds cannot have too many scholars, otherwise they are overwhelmed. – about 5 is a good size – use hierarchical playgrounds: winning teams compete again 10/9/ Introduction Theory Methods Results Conclusion

Piazza Results Do not give hints at solutions. This significantly decreased the amount of discourse taking place. 10/9/ Introduction Theory Methods Results Conclusion

Conclusions and Future Work We propose a systematic gamification of teaching STEM domains: – Design an SCG playground where the winning students demonstrate superior domain knowledge. 10/9/ STEM = Science, Technology, Engineering, and Mathematics

Gamification of Software Development for Computational Problems Want reliable software to solve a computational problem? Design an SCG playground where the winning team will create the software you want. playground design = requirements – Programming the Global Brain socio-technical system (playground) will produce solution to requirements. Crowdsourcing554/24/2011 Introduction Theory Methods Results Conclusion

What we get Engaged software developers – let them produce software that models an organism that fends for itself in a real virtual world while producing the software we want. Have fun. Focus them. – let them propose claims about the software they produce. Reward them when they defend their claims successfully or oppose the claims of others successfully. Crowdsourcing56 Clear FeedbackSense of Progress Possibility of Success Authenticity 4/24/2011

Conclusions Flexible use of SCG using a forum environment with threads and replies using optimization playgrounds is productive: – teams took turns leapfrogging each other – reached state-of-the-art and even improved it SCG has desirable theoretical properties. – faulty decision –> exposing reaction – equilibria – convergence to optimum claim 10/9/ Introduction Theory Methods Results Conclusion

Future Work Make SCG part of cyber-infrastructure (e-science) both for avatars and human scholars. Polish SCG Court – The administrator software needs to be very reliable (to avoid cheating by avatars). – Playground development and testing needs tool support. Further develop SCG with forum software – Playground design defines requirements for know-how. – Hierarchical playgrounds: partitioning into balanced groups. – Restart playground after publishing all current ideas in playground (if optimum is not yet reached). 10/9/ Introduction Theory Methods Results Conclusion

The End More Questions? 10/9/201559

Links SCG Home – home.html home.html Piazza page for Algorithms – Algorithms Home – 2/course-description.html 2/course-description.html Algorithms Feedback – 2/feedback/ 2/feedback/ SCG Court Source – /9/201560

Extra slides 10/9/201561

Essence of Game Rules without Payoff scholars: 1, 2 LifeOfClaim(C) = propose(1,C) followed by (oppose(1,2,C)|agree(1,2,C)). oppose(1,2,C) = (refute(1,2,C)|strengthen(1,2,C,C’)), where stronger(C,C’). strengthen(1,2,C,C’) = !refute(2,1,C’). agree(1,2,C) = !refute(2,1,C) 10/9/ blamed decisions: propose(1,C) refute(1,2,C) strengthen(1,2,C,C’) agree(1,2,c)

1 propose claim C from Claims 2 refute(C,1,2) p(C, …)?(1,-1):(-1,1) 1 scholar 2 scholar strengthen attempt C’ => C refute(C’,2,1) agree attempt C refute(C,2,1) p(C’, …)?(1,-1):(-1,1) p(C, …)?(1,-1):(-1,1) 10/9/2015 refute attempt C refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) s: successful u: unsuccessful Introduction Theory Methods Results Conclusion p(C’, …)?(-1,1):(1,-1) u:1 2s:1 2 u:1 2 p(C, …)?(0,0):(1,-1) s:1 2u:1 2 63

1 propose claim C from Claims 2 refute(C,1,2) p(C, …)? (0,0) :(0,1) 1 scholar 2 scholar strengthen attempt C’ => C refute(C’,2,1) agree attempt C refute(C,2,1) 10/9/2015 refute attempt C refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) s: successful u: unsuccessful p(C’, …)?(0,1): (0,0) u:1 2s:1 2 u:1 2 p(C, …)?(0,0): (1,0) s:1 2u:1 2 Low competition Introduction Theory Methods Results Conclusion 64

Conclusions for Teaching Transition – refute: (1,-1):(-1,1) -> (0,0) :(0,1) – strengthen: (-1,1):(1,-1) -> (0,1): (0,0) – agree: (0,0):(1,-1) -> (0,0): (1,0) creates better playgrounds for learning by lowering competition and increasing teaching between scholars. 10/9/ Introduction Theory Methods Results Conclusion

Claims Protocol. Defines scientific discourse. Scholars make a prediction about their performance in protocol. Predicate that decides whether refutation is successful. Refutation protocol collects data for predicate. As a starter: Think of a claim as a mathematical statement: EA or AE. – all planar graphs have a 4 coloring. 10/9/201566

More examples of Protocols Let f(x,y)=x*y+(1-x)(1-y^2)). Alice claims Math(0.61): Bob constructs an x in [0,1] and Alice construct a y in [0,1], and Alice guarantees that f(x,y)> True claim but can be strengthened to Alice claims Solar(RawMaterials,m,0.61). Bob constructs raw materials r in RawMaterials and Alice constructs a solar cell s in Solution from r using money m and so that efficiency(s)> /9/ Introduction Theory Methods Results Conclusion