The Representation of Uncertainty for Validation and Analysis of Social Simulations TRADOC Analysis Center – Monterey 21 September 2010 Unclassified Debbie Duong, Augustine Consulting David Makovoz and Hyam Singer, Impact Computing

What is the Basic Mission of Analysis? What does the commander want from analysis? –The likelihood of outcomes of plausible courses of action –How certain we are of those likelihoods What technologies can help us to find these outcomes? –Simulations that compute results based on theoretical cause (correlation alone isn’t general enough) –Integrative Frameworks that use ontologies to compose multiple simulations to recombine factors –Probabilistic Ontologies in those frameworks to properly handle uncertainty What is a “Probabilistic Ontology” ? –An “ontology” is a rich categorization of the objects of a domain, general and specific, that includes the relations between objects. It is subject to inference, and can enforce logical consistency –A “probabilistic ontology” includes categorizations with a degree of uncertainty 8/16/20102 CG Modeling

Why it is difficult to find likelihoods of plausible outcomes in Irregular Warfare Irregular Warfare(IW) is uncertain, and Integrative Frameworks are needed to take uncertainty into account –Integrative Frameworks can handle Epistemic Uncertainty, the uncertainty in what social scientists know, by testing Courses of Action against different combinations of theories by switching models in and out –Integrative Frameworks can handle Intrinsic Uncertainty of the social world, which is greater than the physical world because of “path dependencies” in Complex Adaptive Systems, and the human use of symbols These basic forms of uncertainty lead to many sub-types that probabilistic ontologies can combine, to make a single probabilistic outcome space –Uncertainty of correspondence –Uncertainty of properties –Uncertainty of subsumed relations –Social correlative studies –Dynamic descriptions –Confidence factors –Feasible Parameter Sets 8/16/20103 CG Modeling

Uncertainty of Correspondence Example: “There is a 20% chance that what a child identifies as a bug is the same as an entomologist would” Lack of correspondence between the categorization of data in simulations comes from Epistemic Uncertainty –Theoretical disagreement about the relevant categories and relations between them Social data mostly comes in different, useful-for-a-purpose, points of view and so it is hard to interoperate using logic How Probabilistic Ontologies can help –Inference may be used in detailed ontologies that break down to elements and retranslate up, in a mediation ontology -Example: “a child calls any animal with an exoskeleton a bug” –Probabilistic relations may be used to cover what logic does not -Example: “what an entomologist calls an insect, a child calls a “bug” –In multiple runs of the simulation, change the match in the interface from simulation to simulation probabilistically 8/16/20104 CG Modeling

Uncertainty of Properties Example: “There is a 70% chance that a chair has four legs” Comes from the fact that human concepts are by nature fuzzy, and ill-suited to crisp definitions that make computer concepts decidable Differences in points of view can be reflected in different beliefs in the percent chance of a category having a property Uncertainty of properties allows something to have “membership” values in many categories How Probabilistic Ontologies can help –Uncertainty of properties can be used to compute correspondence using statistics of properties -Example: Use the fact that John thinks 90% of chairs have four legs, and Kate thinks 25% of them do, to match Kate’s idea of “chair” to John’s idea of “chair” dependent on the number of legs –In multiple runs of the simulation, change the match in the interface from simulation to simulation probabilistically 8/16/20105 CG Modeling

Uncertainty of Subsumed Relations Example: “There is a 90% chance that a murder weapon is a gun” Comes from the uncertainty of multi-resolutional models –One model is at a higher (less specific) level of description, i.e. doesn’t need to know the type of weapon, and the over is at a lower level of description, and i.e. does need to know the type of weapon –If they have the same point of view, then correspondence going from lower to higher can be exact. However, correspondence from higher to lower must be probabilistic How Probabilistic Ontologies can help –Compute the conditional probability that a higher level translates to a particular lower level category –In multiple runs of the simulation, change the match in the interface from simulation to simulation probabilistically 8/16/20106 CG Modeling

Social Correlative Studies Example: 10% of the time inflation occurs, unemployment is also high Two branches of social literature are correlative studies and theoretical studies –Simulations take care of causal studies, because they have the ability to represent cause in humans through motivation towards goals –Even in causal simulations, statistics covers for what is not known, and falls under the category of “correlative” -For example, If I don’t know the causal mechanisms by which anomie is related to unemployment, I take a random variate How Probabilistic Ontologies can help –Correlative studies can help resolve conflicts between models and to turn composed simulations into a single scenario –This happens through finding a scalar correspondence to data, with the correlative studies serving as the more trusted data than that created by the simulation, to use to adjudicate between models –A probabilistic match between the concepts of the social correlative studies and the models is necessary to do this validation and adjudication –If we know what data is correlated to what, we can assign surrogate data 8/16/20107 CG Modeling

Dynamic Descriptions There is over a 90% chance that a State with serious factionalization and a partial democracy will become unstable within a year In order to compare a simulation’s data to social correlative data and to another simulation’s data, not only should a “co-occurrence” match be made, but a dynamic “sequential” match –To validate a simulation against data, dynamic sequences of states should be similar –A Markov process describes dynamic sequences of states based on variable values well, and can be a common framework for comparison of series data How Probabilistic Ontologies can help –Probabilistic Ontologies can find the correspondence between data of one simulation and data of another, so that Markov processes that describe their dynamics can be compared. –The Markov process itself may be expressed in a probabilistic ontology 8/16/20108 CG Modeling

Confidence Factors Example: John believes that the New York Times is wrong 20% of the time while the National Enquirer is wrong 80% of the time In order to compute what we believe to be the correct state space of output, we must also include the pedigree of data that we are using, our confidence in the source of the data. –Subject Matter experts have different levels of skills even within the same school of thought –Modelers and modeling techniques have different levels of trustworthiness –It may be that you would not want to trust data that did not correspond to trusted data in the past How Probabilistic Ontologies can help –Probability theory can fold confidence of source right in with the other types of uncertainty, making a wider range of parameters to test for data with less correspondence 8/16/20109 CG Modeling

Feasible Parameter Sets Example: Ninety five percent of the time that visibility is low, the speed of the boat is low Analysis using multiple models is all about data farming feasible sets of parameters that implement the study. –We data farm for likelihoods of *plausible* Courses of Action –This may include Strategic Data Farming, where motivation towards goals are used to add parameters of moves and countermoves for scripted simulations How Probabilistic Ontologies can help –Uncertainty may tell the range, but ontology tells the rules that restrict –The models should be treated like implementations of a functional specification of a conceptual model, implementing an ontology –Ontologies are rich enough to express the rules and relations of a theory, and define what is open to scientific refute and what is mere implementation details –The ontology is a “cognitive wrapper” that automates how a human would put in combinations of parameters 8/16/ CG Modeling

Summary Probability Theory can combine all the forms of uncertainty in the parameters in the interface between models, to efficiently explore the probabilistic state space Probabilistic Ontologies tell the rules of the study, and the models merely implement these rules. Ontologies restrict the sets of inputs to the models so that they are appropriate for exploring the space of likelihood Probabilistic Ontologies can compose a federation of simulations and help it to find a consensus, so that different theories may be treated as another form of uncertainty through switching models in and out 8/16/ CG Modeling