MIS 643 Agent-Based Modeling and Simulation (ABMS) Bertan Badur Department of Management Information Systems Boğaziçi University

Model Analysis Chapter 21-23, of Agent-Based and Individual- Based Modeling: A Practical Introduction, by S. F. Railsback and V. Grimm

Outline Chapter 21: Introduction to Part IV Chapter 22: Analyzing and Understanding ABMs Chapter 23: Sensitivity, Uncertainty and Robustness Analysis Chapter 24:

Chapter 21: Introduction to Part IV 21.1 Objectives of Part IV 21.2 Overview of Part IV

21.1 Objectives of Part IV Testing – checking whether a model or submodel is correctly implemented and does what itis supposed to do Analysing a model: trying to understand what amodel does Understanding not automatic from begining of modeling cycle –sukbmodels or simple models –POM for sturucture, theory calibration Full models –frase design at some point –understand how it works and behave

not too soon once the model –key processes –represent real system reasonably version number two or three versions is likely Programming and testing easy What is science? –relation between model and real system – POM Part III –analyse throughly – what it does simlfy or extend by adding new elements formulation few days, analysing months yearns

21.2 Overview of Part IV Chapter 22 general strategies of analyzing ABMs specific to ABMs –structural richness and realism through controled simulation experiments –change assuptions submodels... Chapter 23 –sensitivity, uncertainty and robustness Sensitivity, Uncertainty and Robustness Analysis

Chapter 22: Analyzing and Understanding ABMs 22.1 Introduction 22.2 Example Analysis: The Segregation Model 22.3 Additional Heuristics for Understanding ABMs 22.4 Statistics for Understanding 22.5 Summary and Conclusions

22.1 Introduction controlled experments –varying one factor at a time – efeects on results –establishng causal relationships – understanding how the results are affected by each factor Scientific method – reproducable experiments –compleatly dercribing the model - lab or field documenting –parameter values- input data- initial conditions –anaylsing results of experments

controlled simulation experiments –design, test and calibrate - models –understanding and analyzing what models do How to analyse –model, the system and questions addressed, –experience and problem solving heuristics Heuristics or rule of tumbs –often usefull but not always not unscientific

learning objectives Understan purpose and goals of analyzing full AMBs –finished or preliminary ten heuristics statistical anaysis for ABMs

22.2 Example Analysis: The Segregation Model ODD purpose entities, state variables and scales –turtles – households loaction, heppyness –houses - patches space 51*51 time stop – all heppy

Processes if all happy stop far all aent if lo limit move update heppyness produce output

submodels –move –update

Analysis

Heukristic: try extream values of parameters model outcomes is often easy to predict or understand Set tolernce low Set tolarance high

Heuristics: findtipping points in model behavior qualitatively diferent behavior at extream values of parameters vary the parameter try to find “tipping point” –the parameter range – model behavior suddenly changes regiems of control –process A after some point process B may dominent

Heuristics try different visual representations of the model –color size patches run the model step by srep look at striking or strange patterns at interesting points keep the parmeter and vary other parameters

22.3 Additional Heuristics for Understanding ABMs use several “currencies” for evaluating your simulation experiments analyze simplified version of your model analyze from the buttom up explore unrealistic senarios

Heuristics: use several “currencies” for evaluating your simulation experiments ABMs are rich in structure “currincies” summary statistics or observations emprical measures in the real system Ex: population modeling –measure – population size wealth –analyze time series of population size –even mena or range good currincies – observation in ODD design concept several currincies – how sensitive they are

statistical distributions –mean standard deviation, range –distribution – normal, exponential characteristics of time series –trend, autocorrolation time units to reach a state measures of spatical distributions –spatial autocorrelation, fractile dimension measures of difference among agents –how some charcetristics different, distributions stability properties network characteristics –clustering coefficient, degree

Heuristics: analyze simplified version of your model simplfy ABM so many foctors affect output reduce complexity –undertand what mechnizms what cause what results make the environment constant make space homogenuous –all patches same over time reduce stocasticity –fixed initial conditions – all agent alike –insteaad of randomness use mean values reduce the system size turn off some actions in model schedule manually create simplified initail configrations

Heuristics: analyze from the buttom up ABMs hard to understand behavior of its parts – agents and their behavior first test and undertsnd these then full model anaysis of submodels developing theory for agnet bahavior

Heuristic: explore unrealistic senarios simulate senarios – never occur in reality to see direct effect of a process or mechanizm on resutls – remove it Ex 2: How investor behavior affects double – auction markets interesting contrast: –models – unrealistically simple investor behavior –produce system level results not so unrealistic conclusion –complex agent behavior – not reasn for complex market dynamics –market rules themselfs might be important

22.4 Statistics for Understanding statistics – analysis and understanding infer causal relatinships from a limited and fixed data ABM – –generates as much data aa possible –additional mechnizms if cannot explain –add new mechanizms –change assuptions purpose and mind-set of –statistics and simulation modeling are quite different

summary sttistics –aggregagting model outputs - mean, standard deviation –extream values might be importnat so outliers are usefull Contrasting senarios –detect and quantify differences between senarios –assumptions may affect resutls – number of treatments –easier to change assuptions –t test ANOVA

Quantifying correlative relationships –regression ANOVA –statistical relationsships between inputs – outputs –inputs: paramerters, initial conditions, time series –not directly idenfy causal relations –but idenfity relavant factors –meta-models Comparing model outputs to emprical patterns

22.5 Summary and Conclusions combin –reasoning, strong inference, systematic anaysis, intiution and creativity once build an ABM or freeze it understand what is does – controlled simulation experiments heuristics publications heuristics in figure 22.3 add your own

Chapter 23: Sensitivity, Uncertainty and Robustness Analysis 23.1 Introduction and Objectives 23.2 Sensitivity Analysis 23.3 Uncertainty Analysis 23.4 Robustness Analysis 23.5 Summary and Conclusions

23.1 Introduction and Objectives Does an ABM reproduce observed patterns robustly or sensitive to change in model –parameter –structure how uncertain are model results if model reproduce patterns foır –parameters – limited range or values –key processes are modelsed one exact way unlikely to capture real mechanizm underlying hhe patterns

Basic Definitions Sensitivity analysis (SA) exokıres how sensitive model’s outputs are to changes in parameter values Uncertainty Analysis (UA) looks at how uncertainty in parameter values affect the relaibility of model results Robustness analysis (RA) explores robustness ofresults and conclusions of a model to changes in its structure

Learning objectives local SA with BehavioSpace visualizations – SA with several parameters or global SA stamdard UA methods with BehaviorSpace steps of conducting RA

23.2 Sensitivity Analysis to perform SA full version of the model “reference” parameter set one or two key outputs controled simulation conditions

Local Sensitivity Analysis Objective – how sensitive the model currency seleced parameters one at a time usually all parameters Steps –range of parameter – +or-5% –run model for reference P and p-dP p+dp – replicate –mean C values –calculate sensitivity – approximatins to partial derivative

Three types of parameters high values of S –processes imortant in the model high value of S and highuncertrainty in reference valus –little information to estimate their values –special attantion as calibration –target of emprical research to reduce uncertainty low values of S –relatively unimportant processes - removable

Alternatives only positive change C’/C absolugtechange distibuton of C – variance diferent values of P –rgression of C on P

Limitations linear response so parameter change is small parameter interractios missing around reference parameter set

Analysisof Parameter Interractions via Countour Plots contour plots – interractions of two parameters –all other parameters are kept constant Multi-panel contour figures – model sensitivity –many parameters at onces

Global Sensitivity Analysis vary all parameters over their full range look at several currencies - understanding “brute force” - analysis –for each parameter several values –replicaitons –hard to measure currencies regression analyis – respose surface methods design of simulation experiments –not all combination of parameters

23.3 Uncertainty Analysis similar to SA but to understand how –the uncertainty in parameter values and –model’s sentitivity to parameters interract to cause uncertainty in model results parameters – measurment errors steps of a UA –identify the parameters –for each parameter – define a distribution belief or measurment errors –run the modelmany times – drawing from distributions –analyze distribution of model results

23.4 Robustness Analysis Weisberg (2006) Whether the results depends on the –esentials of the model or –details of the simplfying assuptions study number of distinct similar models of the same phenomena despte different assumptions – similar results –robust theorm - free of details of the model modeling, POM –robust explanations of observed patterns

A full model – frozen two heuristics: –analyze simplified versions –explore unrelistic senarios more complex versions General steps of RA –start with a well tested model version –which elements to modify –test modified model – reproduce observed patterns

theory development – agent behavior –testing alternative submodels RA –testing alternative versions Example: Robustness Analysis of the Breeding Synchrony Model –left as an exercise

23.5 Summary and Conclusions