Wizer – What-If Analyzer: Validation of Large Scale Stochastic Agent Models Project Investigator: Kathleen M. Carley – CMU, ISRI, CASOS Why is Validation Hard? Complexity of multi-agent systems: the significant number of input parameters, output variables, & model parameters, and their interactions Models are a subset of reality: model assumptions may not match assumptions underlying data Cognitive bias: validation is often best not done by the modeler(s) Validation is knowledge intensive Validation consumes a significant amount of time & resources: large multi-agent systems take significant time to run, even on a supercomputer making available only a limited number of virtual experiments The amount of empirical data is limited Stochastic nature of many models adds to complexity Least developed area of the computational modeling How Validation is Currently Done Manually Human expert judgments (pitfall: implicit biases) Assisted by computer: Monte-Carlo technique and exploratory modeling In software engineering: Automated program verification and testing Theorem proving: mechanization of formal reasoning, following laws of logic Model checking: method for formally verifying finite-state concurrent systems using temporal logic. In engineering: Experiment design Response surface methodology Conceptual View of Wizer Flow Diagram of Wizer Alert Wizer System information of what parameters influence which data stream & how Which data streams are wrong/right and how Wizer Inference Engine, inferring and deciding which parameters to change and how Empirical constraints on parameters New parameters Inner Workings of Alert Wizer School absence Work absence Doctor visits ER visits ER visits (SDI) 7 Drug Types Purchases BioWar simulator EMPIRICAL DATA Minimum Bound Check Maximum Bound Check “Mean Significantly Not Different” Check AUTOMATED CHECK REPORT Simulation outputs Calculate Statistics Compare Statistics For Simulated with Real Real Statistics Alert Wizer System Diagram Experiment Designer Meta-Modeler Automated code generator Simulation model & parameters for each knowledge nugget constrained by soft knowledge Knowledge miner & causal relation extractor Trend Inference EnginePatterns, norms, constraints, culture, and other “soft” knowledge inference engine Experiment Executor Knowledge base Causal Inference Engine Causal Detector Simulation History Organizer Response Surface Comparator Simulator Simulation histories * New experiment specification New multi-agent model New codes New execution commands Empirical data from literature, journals, surveys, census, health care, sociology, epidemiology, geography, software engineering, etc. Knowledge nuggets and soft knowledge Knowledge as simulation Simulation nuggets Knowledge nuggets Soft knowledge Simulation outputs Trends and differentials Simulation happenings * Causal relations Soft knowledge Knowledge nuggets Feedback Control commands * Performance, old multi-agent architecture, old experiment specification, and results Software engineering knowledge Simulation happenings * Wizer 0.1 Result with BioWar original thresholds governing agent behaviors: th0=5, th1=20, th2=130, th3=260 (these thresholds interact to determine whether an agent goes certain places) Checking mean & std. dev. of visits to places: Work:mean , std. dev School:mean , std. dev Pharmacy:mean , std. dev Doctor:mean , std. dev Emergency_Room:mean , std. dev work is outside bound threshold th0 is the actual cause of work being too low work is too low, decrease th0 school is within bound pharmacy is within bound doctor is within bound Emergency_Room is outside bound thresholds th2 and th3 are the actual causes of ER being too high Emergency_Room is too high, increase th2, increase th3 Wizer modified thresholds: th0=3, th1=20, th2=132, th3=262 Response Surface and Wizer Response surface methodology: collection of mathematical and statistical techniques for the modeling & analysis of problems in which a response of interest is influenced by several variables. Wizer: extends response surface methodology by performing knowledge-intensive search steps via a social inference engine, utilizing laws of logic, instead of just doing mathematical & statistical calculations integration of inference engine & simulation “virtual experiments” Wizer’s testbed: Wizer is used to validate BioWar – a scalable multi-agent network simulator for disease spread in a heterogeneous population. Biowar -- which has a complex response surface -- can be viewed as a multi- dimensional numeric & symbolic optimization problem: e.g., school absenteeism is influenced by student health status and weather announcements. Student: Alex Yahja – CMU, ISRI, CASOS Wizer Wizer is a data-driven automation of validation, utilizing the integration of inference engine and simulation to perform knowledge-intensive “what-if” analyses through simulations and inferences. Hence the name Wizer: What-if Analyzer.