A Popperian Platform for Programming and Teaching the Global Brain Karl Lieberherr Ahmed Abdelmeged Northeastern University, CCIS, PRL, Boston 6/26/20161

Outline Introduction Theory & hypotheses Methods Results & analysis Conclusion Theory Methods Results Conclusion Introduction Theoretical background Methods for playground design Results Conclusions

Results Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, … SCG usage for teaching – Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis – Avatar competitions are not for teaching (but for competitive innovation) Theoretical Properties of SCG 6/26/20163 Introduction Theory Methods Results Conclusion SCG = Scientific Community Game = Specker Challenge Game

What SCG helps with How to identify experts? How to decide if an answer is worthwhile? –Use scholars to choose the winners How to organize egoistic scholars to produce social welfare: knowledge base and know-how how to defend it. –The scholars try to reverse engineer the solutions of winning scholars. 6/26/20164 Introduction Theory Methods Results Conclusion

Claims Protocol. Defines scientific discourse. Scholars make a prediction about their performance in protocol. Predicate that decides whether refutation is successful. Refutation protocol collects data for predicate. As a starter: Think of a claim as a mathematical statement: EA or AE. – all planar graphs have a 4 coloring. 4/24/20115Crowdsourcing

Who are the scholars? Students in a class room – High school – University Members of the Gig Economy – Between 1995 and 2005, the number of self- employed independent workers grew by 27 percent. Potential employees Anyone with web access; Intelligent crowd. 4/24/20116Crowdsourcing

What Scholars think about! If I propose claim C, what is the probability that – C is successfully refuted – C is successfully strengthened If I try to refute claim C, what is the probability that I will fail. If I try to strengthen claim C, what is the probability that I will fail? 7Crowdsourcing4/24/2011 Introduction Theory Methods Results Conclusion

Degree of automation with SCG(X) Crowdsourcing8 no automation human plays full automation avatar plays degree of automation used by scholar some automation human plays 0 1 more applications: test constructive knowledge transfer to reliable, efficient software avatar Bob scholar Alice 4/24/2011 Introduction Theory Methods Results Conclusion

Organizational Problem Solved How to design a happy scientific community that encourages its members to really contribute. Control of scientific community – tunable SCG rules – Specific domain, claim definition to narrow scope. 4/24/2011Crowdsourcing9 happy = no scholar is ignored.

What is a loose collaboration? Scholars can work independently on an aspect of the same problem. Problem = decide which claims in playground to oppose or agree with. How is know-how combined? Using a protocol. – Alice claimed that for the input that Alice provides, Bob cannot find an output of quality q. But Bob finds such an output. Alice corrects. – Bug reports that need to be addressed and corrections. 4/24/201110Crowdsourcing Playground = Instantiation of Platform Introduction Theory Methods Results Conclusion

Theory Extensive Form Representation of Game Community Property: All faulty actions can be exposed.

Extensive-form representation 1.the players of a game: 1 and 2 2.for every player every opportunity they have to move 3.what each player can do at each of their moves 4.what each player knows for every move 5.the payoffs received by every player for every possible combination of moves 6/26/ Introduction Theory Methods Results Conclusion

1 propose claim C from Claims 2 refute(C,1,2) p(C, …)?(1,-1):(-1,1) 1 scholar 2 scholar strengthen attempt C’ => C refute(C’,2,1) agree attempt C refute(C,2,1) p(C’, …)?(1,-1):(-1,1) p(C, …)?(1,-1):(-1,1) 6/26/2016 refute attempt C refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) s: successful u: unsuccessful Introduction Theory Methods Results Conclusion p(C’, …)?(-1,1):(1,-1) u:1 2s:1 2 u:1 2 p(C, …)?(0,0):(1,-1) s:1 2u:1 2

Refutation Protocol Collects data given to predicate p. Alternates. refute(C,proposer,other) p(C, …)?(1,-1):(-1,1) claimpayoff for proposer if p true (defense) payoff for other if p true (defense) payoff for other if p false (refutation) payoff for proposer if p false (refutation) other tries to make p false while proposer tries to make p true. p false means successful refutation. p true means successful defense.

Reinterpret Refutation Refutation leads to successful strengthening or successful agreement.

Essence of Game Rules without Payoff actors: 1, 2 LifeOfClaim(C) = propose(1,C) followed by (oppose(1,2,C)|agree(1,2,C)). oppose(1,2,C) = (refute(1,2,C)|strengthen(1,2,C,C’)), where stronger(C,C’). strengthen(1,2,C,C’) = !refute(2,1,C’). agree(1,2,C) = !refute(2,1,C) 4/24/ blamed decisions: propose(1,C) refute(1,2,C) strengthen(1,2,C,C’) agree(1,2,c) Crowdsourcing

Winning/Losing Scholar who first violates a game rule, loses. If none violate a game rule: the claim predicate c.p(1,2, …) decides. 4/24/201117Crowdsourcing

Game Rules for Playground All objects exchanged during protocol must be legal and valid. Each move must be within time-limit. 4/24/201118Crowdsourcing

Example: Independent Set Alice = proposer, Bob = other. Protocol / claim: AtLeastAsGood. Alice claims to be at least as good as Bob at IS. – Bob provides undirected graph G. – Bob computes independent set sB for G (secret). – Alice computes independent set sA for G. – Alice wins, if size(sA) >= size(sB) (= p(sA,sB)). 4/24/201119Crowdsourcing

More examples of Protocols Let f(x,y)=x*y+(1-x)(1-y^2)). Alice claims Math(0.61): Bob constructs an x in [0,1] and Alice construct a y in [0,1], and Alice guarantees that f(x,y)> True claim but can be strengthened to Alice claims Solar(RawMaterials,m,0.61). Bob constructs raw materials r in RawMaterials and Alice constructs a solar cell s in Solution from r using money m and so that efficiency(s)> /26/ Introduction Theory Methods Results Conclusion

Community Property For every faulty decision action there exists an exposing reaction that blames the bad decision. – Reasons: We want the system to be egalitarian. – It is important that clever crowd members can shine and expose others who don’t promote the social welfare of the community. Faulty decisions must be exposable. It may take effort. 4/24/201121Crowdsourcing Introduction Theory Methods Results Conclusion

Community Property Alternative formulation If all decisions by Alice are not faulty, there is no chance of Alice losing against Bob. – if Alice is perfect, there is no chance of losing. If there exists a faulty decision by Alice, there is a chance of Alice losing against Bob. – egalitarian game 4/24/201122Crowdsourcing

Summary: faulty decisions 1.propose(Alice,C),C=false 2.propose(Alice,C),C=not optimum, C=true 3.refute(Alice,Bob,C),C=true 4.strengthen(Alice,Bob,c,cs),c=optimum 5.strengthen(Alice,Bob,c,cs),c=false 6.agree(Alice,Bob,c),c=false 7.agree(Alice,Bob,c),c=not optimum, c=true 4/24/201123Crowdsourcing

SCG Equilibrium Reputations of scholars are stable. The science does not progress; bugs are not fixed, no new ideas are introduced. Extreme, desirable situation: All scholars are perfect: they propose optimal claims that can neither be strengthened nor refuted. Crowdsourcing244/24/2011 Introduction Theory Methods Results Conclusion

Claims: convergence to optimum Crowdsourcing quality strengthening correct valuation over strengthening true claims (defendable) false claims (refutable) 4/24/2011

Convergence if every faulty action is exposed, convergence is guaranteed. 4/24/201126Crowdsourcing Introduction Theory Methods Results Conclusion

Methods Developed Platform SCG Court = Generator of teaching/innovation playgrounds – 0/tree/GenericSCG/ Developed Algorithms Course using Piazza based on platform experience 4/24/201127Crowdsourcing Introduction Theory Methods Results Conclusion

Avatar Interface AvatarI – public List propose(List forbiddenClaims); – public List oppose(List claimsToBeOpposed); – public InstanceI provide(Claim claimToBeProvided); – public SolutionI solve(SolveRequest solveRequest);

Instance Interface InstanceI – boolean valid(SolutionI solution, Config config); – double quality(SolutionI solution);

InstanceSet Interface InstanceSetI – Option belongsTo(InstanceI instance); – Option valid(Config config); }}

Protocol Interface ProtocolI – double getResult(Claim claim, SolutionI[] solutions, InstanceI[] instances); – ProtocolSpec getProtocolSpec(); – boolean strengthenP(Claim oldClaim, Claim strengthenedClaim);

Claim Class, for all playgrounds Claim – public Claim(InstanceSetI instanceSet, ProtocolI protocol, double quality, double confidence)

Protocol Library AsGoodAsYou.java ExistsForAll.java ForAllExists.java Renaissance.java Survivor.java

Piazza

1 propose claim C from Claims 2 refute(C,1,2) p(C, …)?(1,-1):(-1,1) 1 scholar 2 scholar strengthen attempt C’ => C refute(C’,2,1) agree attempt C refute(C,2,1) 6/26/2016 refute attempt C refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) s: successful u: unsuccessful Introduction Theory Methods Results Conclusion p(C’, …)?(-1,1):(1,-1) u:1 2s:1 2 u:1 2 p(C, …)?(0,0):(1,-1) s:1 2u:1 2 High competition

1 propose claim C from Claims 2 refute(C,1,2) p(C, …)? (0,0) :(0,1) 1 scholar 2 scholar strengthen attempt C’ => C refute(C’,2,1) agree attempt C refute(C,2,1) 6/26/2016 refute attempt C refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) s: successful u: unsuccessful Introduction Theory Methods Results Conclusion p(C’, …)?(0,1): (0,0) u:1 2s:1 2 u:1 2 p(C, …)?(0,0): (1,0) s:1 2u:1 2 Low competition

Competition Knob: minimum For each scholar – count claims that were successfully opposed (refuted or strengthened) encourages strong claims gather information from competitors for free – count claims that were not successfully agreed Good for teaching – students want minimum competition – good students want to build social capital and help weaker students 6/26/201637

Piazza Results Lower competition knob for teaching. For optimization claims got significant scientific discourse. Playgrounds cannot have too many scholars, otherwise they are overwhelmed. – about 5 is a good size – use hierarchical playgrounds: winning teams compete again

Piazza Results Do not give hints at solutions. This significantly decreased the amount of discourse taking place.

Conclusions Transition – refute: (1,-1):(-1,1) -> (0,0) :(0,1) – strengthen: (-1,1):(1,-1) -> (0,1): (0,0) – agree: (0,0):(1,-1) -> (0,0): (1,0) creates a better playground for learning by lowering competition and increasing teaching between scholars.

Conclusions Flexible use of SCG using a forum environment with threads and replies using optimization optimization playgrounds is productive: – teams took turns leapfrogging each other

Introduction Theory Methods Results Conclusion Transformation: performance debate Diverse governance modes Liberalization Underperformance - legitimacy Mixed results of privatization Challenges for state Public Private Public-private Input Output