Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Amplifying the Wisdom of the Crowd, Building and Measuring Expert Consensus for Nonproliferation Issues James Carroll INMM Taos Technical Meeting, May 31, 2012 LA-UR

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Abstract Scientific and cultural progress mostly takes place in the marketplace of ideas. The internet has largely changed how this marketplace functions. Although it has brought many improvements, it has also created a new set of problems. We propose a web service that can be used to crowdsource the process of knowledge acquisition and of knowledge summary, mediate disagreements, and improve discourse on controversial issues, thereby improving the efficiency of the marketplace of ideas. Our approach involves a specific technique for integrating: a forum; a wiki organized into separate camps, with camps organized into a hierarchical structure; a survey system based upon camp support; a mechanism for dynamically reorganizing the structure of the camp hierarchy while mediating user disagreements; and a customizable mechanism for determining how votes are weighted based on credentials or expert assignment. A version of this approach has been implemented, released as open source, and a beta test has gone live. Initial use seems to validate the merits of this approach. (U) Slide 2

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Outline 1. Epistemology and the Marketplace of Ideas 2. Inefficiencies in the Marketplace of Ideas (In the internet age) 3. A Potential Partial Solution Slide 3

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D 1. Epistemology and the Marketplace of Ideas Slide 4

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Logic, the quest for truth: Socrates Plato Aristotle Slide 5

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Logic, the quest for truth: Socrates Plato Aristotle Slide 6

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D The Destruction of Objectivity David Hume ( ) The Problem of Induction a question of how things behave when they go “beyond the present testimony of the senses, and the records of our memory” [Hume1748]. The problem of Generalization (no-free-lunch). Kurt Gödel ( ) for any self- consistent recursive axiomatic system powerful enough to describe the arithmetic of the natural numbers (for example Peano arithmetic), there are true propositions about the naturals that cannot be proved from the axioms. Slide 7

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D No Free Lunch Thomas Mitchell (The Futility of Bias Free Learning): There are as many hypotheses consistent with past data that predict one outcome as that predict another off training set (when generalizing). David Wolpert (The Supervised Learning No Free Lunch Theorem): When all hypotheses are equally likely, there is no “best” supervised learning algorithm off training set, for the 0-1 loss function. James Carroll (No-Free-Lunch and Bayesian Optimality): When all hypotheses are equally likely, then the “best” learning algorithm will report that all outputs are equally likely off training set. Slide 8

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Empirical Falsifiable You can’t prove which theory is true, so stop trying. Theories should be used that are falsifiable, but which have not yet been falsified after concerted effort. Such theories will be useful. Slide 9 Karl Popper ( )

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D The Marketplace of Ideas “Ideas that can get themselves accepted in a competitive market of ideas will tend to be of better quality than other ideas. The marketplace of ideas therefore improves the quality of our ideas and our thinking. If different Slide 10 ideas are presented in an “ideas-market”, and if that market is populated by a maximum number of free agents expressing themselves freely, then those competing ideas will be exposed to a maximum number of supporting and dissenting arguments, and the balance of arguments in favor of or against an idea will be compared to the same balance for counter-ideas. The idea with the best balance will “survive”, because alternative ideas will be seen as comparatively defective, given the fact that the arguments in favor of them are weaker or the arguments against them are stronger.”

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D The Marketplace of Ideas This doesn’t just lead to ideas that are true, but to hypotheses that are useful, in the sense of making the most general predictions about the most things, with the most accuracy. These survive in the marketplace. Slide 11

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Another Rout to the Marketplace: Rout 2… Induction…. Occam’s razor solves the problem of inference. We compute p(h|data)…. But we didn’t personally collect all the data, ourselves Nor can we all do the complex inference for every problem There are more problems you could be an expert on than there are seconds in your lifetime. SO… we have to specialize and trade. But some people lie, so… We have to determine who to trust, and what data to trust, and ultimately what to believe. Result: The marketplace of ideas. Slide 12

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D I, Pencil, My Family Tree as told to Leonard E. Read Slide 13

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Collective Intelligence Slide 14

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Inefficiencies in the Marketplace of Ideas: 1) the limited access to information, 2) the un-manageable volume of information, 3) the reliability of information, 4) the need to determine who to consider an expert, 5) the need to determine what the experts believe, 6) the difficulty in changing popular notions that are resistant to change, 7) the incivility, repetition, and lack of focus in discourse, and 8) the difficulty in overcoming confirmation bias. Slide 15

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Current Cultural and Internet Mechanisms to Improve Marketplace Functionality: 1. Search Engines 2. Peer Review 3. Wikis 4. Forums, Bulletin Boards, and Lists 5. Survey Systems Slide 16

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D The Canonizer Solution A wiki to leverage the wisdom of the crowd for knowledge collection and summary Wiki pages are organized into camps with differing opinions, instead of single pages for each topic Camps are organized into a hierarchical structure to encourage agreement A mechanism for dynamically reorganizing the structure of the camp hierarchy while mediating user disagreements to encourage agreement and highlight areas of continuing disagreement A survey system based upon camp support for determining consensus, and A mechanism for customizing how votes are weighted based on credentials or expert assignment to allow users to explore different demographic effects, or to allow users to give extra weight to experts in different fields if they so choose. A forum which also functions as an list for discussion Slide 17

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Slide 18

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Slide 19

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Weighting Votes (hierarchy of theories of consciousness): Theories of Mind and Consciousness Approachable Via Science Representational Qualia Theory Property Dualism Representational Functionalism MisIdeal MonismMis Slide 20

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Weighting Votes (1 person 1 vote) Theories of Mind and Consciousness (39) Approachable Via Science (36) Representational Qualia Theory (26.25) Property Dualism (20) Representational Functionalism (4.75) Mis (1.5) Ideal Monism (2.25) Mis (7.5)Mis (3) Slide 21

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Weighting Votes (PhDs) Theories of Mind and Consciousness (3) Approachable Via Science (3) Representational Qualia Theory (2) Property Dualism (1) Representational Functionalism (1) Mis (0) Ideal Monism (0)Mis (1)Mis (3) Slide 22

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Weighting Votes (“expert” philosophers of mind) Theories of Mind and Consciousness (10.41) Approachable Via Science (10.41) Representational Qualia Theory (10.41) Property Dualism (9.46) Representational Functionalism (0.94) Mis (0.01) Ideal Monism (0)Mis (0) Slide 23

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Weighting Votes (“expert” computer scientists) Theories of Mind and Consciousness (1.65) Approachable Via Science (1.65) Representational Qualia Theory (1.65) Property Dualism (0) Representational Functionalism (1.65) Mis (0.00) Ideal Monism (0)Mis (0) Slide 24

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D Deliverables/Results 1. A history of the debate and conversation 2. A hierarchy of opinions, driven towards consensus and points of mutual agreement 3. A single page for each “camp” summarizing the arguments for and against each position. 4. A survey of participants, showing which view has the most support 5. A mechanism for varying how the votes are weighted to determine who believes what Slide 25

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA U N C L A S S I F I E D References Hume, D., 1747, “An Enquiry Concerning Human Understanding,” p. 108 Thomas Mitchell, 1980, “The Need for Biases in Learning Generalizations”, Rutgers Computer Science. David Wolpert, 2001, “The Supervised Learning No Free Lunch Theorem” NASA Ames Research Center. James Carroll, 2007, “No-Free-Lunch and Bayesian Optimality” in Meta- Learning IJCNN Workshop Slide 26