Download presentation
Presentation is loading. Please wait.
Published byKevin Flynn Modified over 9 years ago
1
Virtual Scientific Communities for Innovation Karl Lieberherr Northeastern University College of Computer and Information Science Boston, MA joint work with Ahmed Abdelmeged and Bryan Chadwick Karl Lieberherr Northeastern University College of Computer and Information Science Boston, MA joint work with Ahmed Abdelmeged and Bryan Chadwick Supported by Novartis and GMO
2
Larger Context “Open innovation” describes a new paradigm for the management of industrial innovation (Henry Chesbrough) “Wikinomics reveals the next historic step— the art and science of mass collaboration where companies open up to the world.” How to give your problem solving software an ego !? 9/19/2015Innovation2
3
Applications Develop algorithms/software for new optimization problem domain X – Scientific Community Game Software Development: Describe a problem domain X so that SCG(X) provides the best algorithms and their implementations for problems in X. (best within the participating scientific community) Teach computer science, mathematics 9/19/2015Innovation3 SCG = Scientific Community Game = Specker Challenge Game
4
Opening the development approach Problem to be solved: Develop the best practical algorithms for solving optimization problems in domain X. Issue: There are probably hundreds of papers on the topic with isolated implementations. What are the best practical algorithms? Our solution: Use the virtual scientific agent community game SCG(X) with a suitably designed hypotheses language to compare the algorithms. The winning agent has the best practical algorithms. 9/19/20154Innovation
5
Plan Why is it relevant, useful? – Larger context: Open Innovation, Wikinomics – Applications: Netflix in the small, teaching What is it? What is new? – Map problem domain to “second life”, find best solution there and map it back to real life. What do we improve: benefits of SCG How to use SCG Disadvantages Experience with current implementation Related work Detailed example Conclusions 9/19/2015Innovation5
6
What is it: Like Reviewing Process SCG hypothesis valuation [0,1] of hypothesis competition support upon opposition as strong as possible (otherwise someone will strengthen it) Agent loses reputation when someone strengthens or successfully opposes Conference conference paper/review valuation (A,B,C,D) of paper/review committee meeting support upon opposition as strong as possible (otherwise someone will strengthen it) Scholar loses reputation when someone strengthens or successfully opposes 9/19/2015Innovation6
7
Experiments, Related Work SCG Goal: find correct valuation of hypothesis. Has someone else proposed this hypothesis? What was the experience? Experiment with hypothesis before you offer or oppose it. Conference Goal: find correct valuation of paper/review. Has someone else published a similar paper? What was the experience? “Experiment” with paper before you submit or oppose it. Is there software to download and experiment? 9/19/2015Innovation7
8
Like Reviews SCG Competition has goal to find correct valuation of agents Interactive protocol between agents involving problems and their solution. The scholars fend for themselves! Conference Program committee meeting has goal to find correct valuation of submitted papers Interactive protocol between program committee members. 9/19/2015Innovation8
9
Introduction (1) Problem domains to be addressed: – Optimal control of a complex system (mixed continuous and discrete variables) – Optimal assembly of a system from components hardware software – Maximum constraint satisfaction problem (MAX-CSP) – Transporting goods minimizing energy consumption – Schedule tasks minimizing cost 9/19/2015Innovation9
10
Introduction (2) Scientific Community Game(X) [SCG(X)] – Goal: Foster innovation and reliable software for solving optimization problems in some domain X A virtual scientific community consists of virtual scholars that propose and oppose hypotheses maximizing their reputations 9/19/2015Innovation10
11
Agents propose and oppose 9/19/2015Innovation11 HA1 HA2 HA3 HA4 egoistic Alice egoistic Bob reputation 1000 reputation 10 HB1 HB2 opposes (1) provides problem (2) solves problem not as well as she expected based on HA2 (3) WINS! LOSES proposed hypotheses transfer 200 social welfare Life of a scholar: (propose+ oppose+ provide* solve*)*
12
Hypothesis Niche N – subset of problems Confidence [0,1] Valuation [0,1] 9/19/2015Innovation12 confidence 0 1 valuation (how well problems in N can be solved)
13
Hypothesis 9/19/2015Innovation13 0 1 valuation strengthening correct valuation over strengthening
14
Hypothesis hypothesis by Alice: for all problems F in niche N there exists a solution J: p(F,J) Bob opposes: F’ to Alice, Alice cannot find J’:p(F’,J’) therefore she loses reputation. 9/19/2015Innovation14
15
Full Round Robin Tournaments or Swiss-Style Agents to play the SCG(X). Repeat a few times with feedback used to update agents. Within the group of participating agent, the winning agent has the – best solver for X-problems – best supported knowledge about X 9/19/201515Innovation
16
What is the purpose of SCG? The purpose of playing an SCG(X) competition is to assess the "skills" of the agents in: – "approximating" optimization problems in domain X, – "figuring-out" the wall-clock-time-bounded approximability of niches in domain X, – "figuring-out" hardest problems in a specific niche, and – "being-aware" of the niches in which their own solution algorithm works best. This multi-faceted evaluation makes SCG(X) more superior to competitions based on benchmarks that only test the player's skills in approximating optimization problems. During SCG, players cross-test each others' skills. 9/19/2015Innovation16
17
How to use SCG Company A provides a problem domain description X and submits it to the SCG server. The game SCG(X) runs on the web (with human algorithm/software developers involved) and company A receives good, tested software and knowledge about problem domain X 9/19/2015Innovation17
18
Plan Why is it relevant, useful? – Larger context: Open Innovation, Wikinomics – Applications: Netflix in the small, teaching What is it? What is new? – Map problem domain to “second life”, find best solution there and map it back to real life. What do we improve: benefits of SCG How to use SCG Disadvantages Experience with current implementation Related work Detailed example Conclusions 9/19/2015Innovation18
19
Benefit: Improving Benchmark-Driven Algorithm Development A number of autonomous teams Each team develops an agent that embodies their own heuristics Agents participate in a competition (various benchmarks are used) Teams develop their agents for the next competition Examples: SAT, CSP, SMT, ASP (Answer Set Programming) competitions, etc. 9/19/201519Innovation
20
From Benchmark-Driven to SCG-Driven Algorithm Development Hard to measure and detect what is fraud. Instead: Design a system that needs a much weaker “gentleman’s agreement” or none at all The Static Benchmark Problem is ONE problem that SCG solves. Dynamic Benchmarks Others: crowd sourcing management, new software development process that engages software developers and that fosters ease of evolution (e.g., good separation of concerns, …) 9/19/2015Innovation20
21
Problems with Static Benchmarks http://www.cs.kuleuven.be/~dtai/events/ASP- competition/index.shtml Policy against special purpose solutions The purpose of the competition is to be as informative as possible about strengths and weaknesses of … Submission of special purpose programs for solving certain benchmark problems falsifies the information that we get from the rankings and goes against the spirit of the competition. … the use of special purpose programs for certain benchmarks can rightfully be considered as scientific fraud. We appeal to participants … 9/19/2015Innovation21
22
SCG-Driven Algorithm Development Differences to Benchmark-Driven – You don’t rank chess players by giving them a benchmark; you let them play – We turn the algorithms into egoistic virtual scientists that fend for themselves – social welfare: constructive knowledge based on good algorithms 9/19/2015Innovation22
23
What is SCG(X)? Teams Design Problem Solver Develop Software Deliver Agent Agent AliceAgent Bob Administrator SCG police I am the best No!! Let’s play constructively 9/19/201523Innovation Team Alice Team Bob
24
What is SCG(X) 9/19/2015Innovation24 no automation human plays full automation agent plays degree of automation used by scholar our focus some automation human plays 0 1 more applications: test constructive knowledge transfer to reliable, efficient software agent robot Bob Alice
25
competitive / collaborative 9/19/2015Innovation25 Agent Alice: claims hypothesis H Agent Bob: challenges H, discounts: provides evidence for !H loses reputation rwins knowledge k wins reputation rmakes public knowledge k
26
Scholars and Agents: Same rules Are encouraged to 1.offer results that are not easily improved. 2.offer results that they can successfully support. 3.strengthen results, if possible. 4.publish results of an experimental nature with an appropriate confidence level. 5.stay active and publish new results or oppose current results. 6.be well-rounded: solve posed problems and pose difficult problems for others. 7.become famous! 9/19/201526Innovation
27
More Applications Special issue editors for problem domain X. publish top 15 submissions Professor teaching a software development class: students develop fighting agents for full-round robin tournament Teaching constructive topics etc. 9/19/2015Innovation27
28
Soundness Theorem SCG is sound: The agent with the best algorithms / knowledge wins (there is no way to cheat) – best: within the group of participating agents – issues: Does an agent win because she is good at solving? Or good at proposing, opposing and providing? Answer: proposing, opposing and providing all reduce to solving. Can an agent force a win or a tie? 9/19/2015Innovation28
29
Justifying benefits (1) Benefit: competitive – collaborative Game component: hypotheses propose-oppose : problems provide-solve How this game component brings the benefit – hypothesis by Alice: for all problems F in niche N there exists a solution J: p(F,J) – Bob opposes: F’ to Alice, Alice cannot find J’:p(F’,J’) therefore she loses reputation. – Alice lost but she now knows F’ where she cannot achieve what she claimed. F’ was harder than what Alice expected. 9/19/2015Innovation29
30
Justifying benefits (2) Benefit: competitive – collaborative Game component: hypotheses propose-oppose : problems provide-solve How this game component brings the benefit – hypothesis HA by Alice: for all problems F in niche N there exists a solution J: p(F,J) – Bob opposes by non-trivially strengthening HA to HB: HB => HA. Alice cannot discount HB. Therefore she loses reputation. – Alice lost but she now knows that her hypothesis HA might not be the strongest. 9/19/2015Innovation30
31
Benefits of SCG-driven Focus on understanding problem domain. – What are the niches where specialized algorithms perform well? – What are the hard problems in a niche? Knowledge maintenance system Control of niches to be explored 9/19/2015Innovation31
32
Reputation Gain Challenging (C) Gain for A (A supporting), Loss for A (B discounting)
33
Plan Why is it relevant, useful? – Larger context: Open Innovation, Wikinomics – Applications: Netflix in the small, teaching What is it? What is new? – Map problem domain to “second life”, find best solution there and map it back to real life. What do we improve: benefits of SCG How to use SCG Disadvantages Experience with current implementation Related work Detailed example Conclusions 9/19/2015Innovation33
34
How to use SCG(X) ABB needs new ideas about how to solve optimization problems in domain X. Define hypothesis language for X – X-problems – hypotheses, includes protocol Submit hypothesis language definition to SCG server. 9/19/201534Innovation
35
How to use SCG(X) Offer prize money for winner with conditions, e.g., performance must be at least 10% higher as performance of agent XY that ABB provides. 10 teams from 6 countries sign up, committing to 6 competitions. Player executables become known to other players after each competition. One team from ABB. The SCG server sends them the basic agent and the administrator for testing. 9/19/201535Innovation
36
How to use SCG(X) Game histories known to all. Data mining! First competition is at 23.59 on day 1. Registration starts at 18.00 on same day. The competition lasts 2.5 hours. Repeat on days 7, 14, … 42. The final winner is: Team Mumbai, winning 10000 Euro. Delivers source code and design document describing winning algorithm to ABB. 9/19/201536Innovation
37
Benefits for ABB of using SCG(X) Teams perform know-how retrieval and integration and maybe some research. – Participating teams try to find the best knowledge in the area. – Hypothesis language gives control! The non-discounted hypotheses give hints about new X-specific knowledge. A well-tested solver for X-problems that integrates the current algorithmic knowledge in field X. 9/19/201537Innovation
38
Plan Why is it relevant, useful? – Larger context: Open Innovation, Wikinomics – Applications: Netflix in the small, teaching What is it? What is new? – Map problem domain to “second life”, find best solution there and map it back to real life. What do we improve: benefits of SCG How to use SCG Disadvantages Experience with current implementation Related work Detailed example Conclusions 9/19/2015Innovation38
39
Disadvantages of SCG The game is addictive. After Bob having spent 4 hours to fix his agent and still losing against Alice, Bob really wants to know why! Overhead to learn to define and participate in competitions. The administrator for SCG(X) must perfectly supervise the game. Includes checking the legality of X-problems. – if admin does not, cheap play – watching over the admin 9/19/201539Innovation
40
How to compensate for those disadvantages Warn the scholars. Use a gentleman’s security policy: report administrator problems, don’t exploit them to win. Occasionally have a non-counting “attack the administrator” competition to find vulnerabilities in administrator. – both generic as well as X-specific vulnerabilities. 9/19/201540Innovation
41
GIGO: Garbage in / Garbage out If all agents are weak, no useful solver created. WEAK against STRONG: – STRONG accepts a hypothesis that is not discountable but WEAK cannot support it. Correct knowledge might be discounted. – STRONG strengthens a hypothesis too much that it becomes discountable, but WEAK cannot discount it. Incorrect knowledge might be supported – STRONG is discouraged to exploit WEAK by game rules 9/19/2015Innovation41
42
Plan Why is it relevant, useful? – Larger context: Open Innovation, Wikinomics – Applications: Netflix in the small, teaching What is it? What is new? – Map problem domain to “second life”, find best solution there and map it back to real life. What do we improve: benefits of SCG How to use SCG Disadvantages Experience with current implementation Related work Detailed example Conclusions 9/19/2015Innovation42
43
Experience Used for 3 years in undergraduate Software Development course. Prerequisites: 2 semesters of Introductory Programming, Object-Oriented Design, Discrete Structures, Theory of Computation. – Collect and integrate knowledge from prerequisite courses, lectures, and literature. – Teach it to the agent. 30% of grade is allocated for agent performance in weekly competitions. 9/19/201543Innovation
44
Mechanics of using current implementation We define X = MAX-CSP. We produce administrator and baby agent for X at beginning of course. Game flow: – Agents register with administrator – After deadline, administrator tells agents when it is their turn (1 minute) sending them all currently proposed hypotheses – After 1 minute, agent sends back transactions. 9/19/2015Innovation44
45
Mechanics of using current implementation 3 competitions per week. Last about 12 hours each. 75% of competitions count towards grade. 1 competition: attack the administrator. 9/19/2015Innovation45
46
Experience MAX-CSP MAX-CSP Problem Decompositions T-Ball (one relation), Softball (several relations, one implication tree), Baseball (several relations). ALL, SECRET 9/19/201546Innovation
47
Stages for SECRET T-Ball MAXCUT – R(x,y)= x!=y – fair coin ½ – maximally biased coin ½ – semi-definite programming / eigenvalue minimization 0.878 9/19/201547Innovation
48
Stages for SECRET T-Ball One-in-three – R(x,y,z) = (x+y+z=1) – fair coin: 0.375 – optimally biased coin: 0.444 9/19/201548Innovation
49
Stages for ALL Baseball Propose/Oppose/Provide/Solve – based on fair coin – optimally biased coin correctly optimize polynomials – correctly eliminate noise relations – correctly implement weights – … 9/19/201549Innovation
50
Life with SCG with SCG structured collaboration between scholars, frequent feedback motivation: propose and oppose non-trivial hypotheses to gain reputation. Drive to win knowledge accumulation in undiscounted hypotheses target scholars on a topic without SCG collaboration is unstructured, less effective motivation: reputation gain is delayed knowledge is scattered in emails, programs and minds more management effort required 9/19/201550Innovation
51
How to model a hypothesis A problem space. A discounting predicate on the problem space. A protocol to set the predicate through alternating “moves” (decisions) by Alice and Bob. If the predicate becomes true, Alice wins. 9/19/201551Innovation
52
How to model a hypothesis Proposing and challenging a hypotheses is risky: your opponent has much freedom to choose its decisions within the game rules. Alternating quantifiers. Replace “exists” by agent algorithm kept by administrator. 9/19/201552Innovation
53
Hypothesis [Example] 1in3 example. 9/19/201553Innovation
54
X = Boolean MAXCSP Given a sequence of Boolean constraints formulated using a set R of Boolean relations, find an assignment that maximizes the fraction of satisfied constraints. Niche defined by R. 9/19/201554Innovation
55
1in3 niche Only relation 1in3 is used. 1in3 problem F: v1 v2 v3 v4 v5 1in3( v1 v2 v3) 1in3( v2 v4 v5) 1in3( v1 v3 v4) 1in3( v3 v4 v5) secret 1 0 0 1 0 Truth Table 1in3 000 0 001 1 010 1 011 0 100 1 101 0 110 0 111 0 Secret quality SQ = 3/4 9/19/201555Innovation
56
1in3 Hypothesis 1in3 hypothesis H proposed by Alice: exists F in 1in3 niche so that for all S Bob that opponent Bob searches in time t (small constant) seconds: Quality(F,S Bob ) < 0.4 * Quality(F,S Alice ). H = (niche = (1in3), AR =0.4, confidence = 0.8) Bob has clever knowledge that Alice does not have. He opposes the hypothesis H by challenging it using his randomized algorithm. 9/19/201556Innovation
57
Bob’s clever knowledge 4/9 for 1in3 4/9 for 1in3: For all F in 1in3 niche, exists S so that Quality(F,S) >= 0.444 * SQ. Proof: la(p)=3*p*(1-p) 2 has the maximum 4/9. argmax p in [0,1] la(p) = 1/3. Without search, in PTIME. Derandomize Bob successfully discounts Alice gets a hint – Was Bob just lucky? Truth Table 1in3 000 0 001 1 010 1 011 0 100 1 101 0 110 0 111 0 9/19/201557Innovation
58
1in3 Hypothesis Bob does not know whether 4/9 is best possible. Should check Semidefinite Programming. Bob only knows that the set of 1in3 problems having a solution satisfying 4/9 + eps, eps > 0, is NP-complete. 9/19/201558Innovation
59
Related Work Renaissance mathematicians Various benchmark based competitions What is new? – Software that has an ego – Holistic software with introspection – Evaluating software through a game – Scientific Community Game Software Development 9/19/2015Innovation59
60
Conclusions To address a problem domain X: – “map it to second life”: define a scientific community game for X on the web: SCG(X) – let the game SCG(X) run a few times and choose the winner Benefits – Evaluates fairly, frequently, constructively and dynamically. Encourages retrieval of state-of-the-art know-how, integration and discovery. – Challenges humans, drives innovation, both competitive and collaborative. – Agents point humans to what needs attention in problem solution / software. 9/19/2015Innovation60
61
Conclusions SCG(X) provides a structured process for developing software for optimization problems. Benefits – Social Engineering: makes it fun through game. – Fair: Only hard work makes you win. – Engage a large community on one domain X. Tools 9/19/2015Innovation61
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.