1. MIS 585 Special Topics in MIS: Agent-Based Modeling 2015/2016 Fall Chapter 7 Verification, Validation, and Replication

2. Outline Correctness of a Model Verification Validation Replication Summary

3. Verification, Validation, and Replication Previous chapters –importance and utility of ABM –extend an existing and build a new ABM –components of an ABM –collect and analyze results from an ABM In this chapter –correctness and usefulness of an ABM –how an implemented ABM corresponds to a conceptual model –how to assess the match between an ABM and the real world

4. Outline Correctness of a Model Verification Validation Replication Summary

5. Correctness of a Model Model accuricy can be evaluated through three modeling processes –verification, validation, and replication Validation: –whether the implemented model corresponds and explains some real world phenomena Verification: –implemented model corresponds to the target – conceptual model Replication: –implementation of a conceptual model of another researcher

6. Outline Correctness of a Model Verification Validation Replication Summary

7. Verification Communication Describing Conceptual Models Verification Testing Beyond Verification Sensitivity Analysis and Robustness Verification Benefits and Issues

8. Verification goal of verification –elimination of bugs in the code General guidline - Basic modeling principle –start simple –verificaiton is easy as well adding components incrementally towards the RQ other compleications –interaction between components

9. Voting Behavior Model (VBM) political scientists people’s social interactions determine voting in elections based on polls and election results –people inital way of wote –they talk to neigbors and friends discuss the way of voting may change their vore –may happen several times during an election period

10. Communication model author and implementer –may be different –varification is more difficult get familize –implemneters – domain –domain experts – modeling tools conceptual model – implemented

11. Describing Conceptual Models flowcharts –rounded squares – start and stop –squares – initiate processes –dimonds – decision pseudo-code Unified Modeling Language (UML) –ABM specific concepts to UML ODD

12. Pseudo-code of the VBM for each voter: set vote either 0 or 1 with equal probabiity loop until election if majority of neighbors’s votes 1 and votrers vote is 0 set vote 1 elseif majority of neighbors’s votes 0 and votrers vote is 1 set vote 0 if vote = 1, eet color blue else set color red Display count of voters with 1 Display count of voters with 0 end loop

13. Verification Testing Implementing the pseudo-code in NteLogo then, write a test procedure for testing whether the number of green and blue patches are roughly equal or not

14. setup patches-own [ vote ;; my vote (0 or 1) total ;; sum of votes around me ] to setup clear-all ask patches [ if random 2 = 0 ;; half a chance of this [ set vote 1 ]

15. setup (cont.) ask patches [ if random 2 = 0 ;; half a chance of this [ set vote 0 ] ask patches [ recolor-patch ] check-setup reset-ticks end

16. recolor-patch to recolor-patch ;; patch procedure ifelse vote = 0 [ set pcolor green ] [ set pcolor blue ] end

17. check-setup ;; This procedure checks to see if the SETUP procedure sets up the model with roughly ;; equal numbers of blue and green patches to check-setup ;; count the difference between the number of green and the number of blue patches let diff abs (count patches with [ vote = 0 ] - count patches with [ vote = 1 ]) if diff >.1 * count patches [ print "Warning: Difference in initial voters is greater than 10%." ] end

18. corrected code patches-own [ vote ;; my vote (0 or 1) total ;; sum of votes around me ] to setup clear-all ask patches [ set vote random 2 ;; set vote to either 0 or 1 recolor-patch ] reset-ticks check-setup end

19. Unit Testing (component testing) writing small tests to check whether individual units are working correctly in the code “in model” unit testing – “online” unit testing “offline” unit testing: seperate piece of code with input values

20. go to go ask patches [ set total (sum [ vote ] of neighbors) ] ;; use two ask patches blocks so all patches compute "total" ;; before any patches change their votes ask patches [ ifelse vote = 0 and total >= 4 [ set vote 1 ] [ if vote = 1 and total <= 4 [ set vote 0 ] ] check-setup recolor-patch ] tick end

21. Beyond Verification no equilibrium jaggled edges and cyclses tied votes voters on the edge cycles back and forth add a switch CHANGE-VOTE-IF-TİED? when voters neighbors are closely divided, vote underdog –not for majority

22. go to go ask patches [ set total (sum [ vote ] of neighbors) ] ;; use two ask patches blocks so all patches compute "total" ;; before any patches change their votes ask patches [ if total > 5 [ set vote 1 ] if total < 3 [ set vote 0 ] if total = 4 [ if change-vote-if-tied? [ set vote (1 - vote) ] ] ;; invert the vote

23. go (cont.) if total = 5 [ ifelse award-close-calls-to-loser? [ set vote 0 ] [ set vote 1 ] ] if total = 3 [ ifelse award-close-calls-to-loser? [ set vote 1 ] [ set vote 0 ] ] recolor-patch ] tick end

24. Sensitivity and Robustness how sensitive the outputs results of a model to changes in –parameters, initail conditions, environments, behaviorsl asumptions examination of the impact of varying model parameters on results

25. Sensitivity and Robustness how sensitive the outputs results of a model to changes in –parameters, initail conditions or other asumptions E.g.: in the VM –how the final voting behavior sensitive to initial distrfibution of votes change initial distribution of green and red agents by a parameter

26. stop the model –no voter change votes or –100 ticks verification of new initialization 10 repitations graph initial distrbution vs. average count of votes –initial green patches vs. final green patches Figure 7.6 nonlinear effect in either side

27. depends on the output a qualitative measure – –solid islands of both colors form insensitive to change in initial distribution Chagne environments –square to hex latice, network

28. Verification: Benefits and Issues understanding unexpected outcomes impact of changing parameters or rules omceptual model – implemented model

29. Verification is difficult chalanging to determine – surprising reults –a bug in code –outcome of a low level rule –conceptual model is wrong

30. not just verified or not but continuous less or more verified –more tests –more sensitivity analysis –replication of the model

31. Outline Correctness of a Model Verification Validation Replication Summary

32. Validation Macrovalidation vs. Microvalidation Face Validation vs. Emprical Validation Validation Benefits and Questions

33. Validation process of ensuring that there is a correspondence between the the implemented model and the reality by its nature –complex, multilevel and relative RQ in mind validate aspects of the model related to RQs

34. Two Axes of Validation two axes Rand and Rust 2011 1 - level at which validation process is occuring –microvalidation: behviors and mechanisms of agents are match up with real world analogs –macrovalidation: ensuring that the aggreage, emergent preperties of the model is corresopeding with that ofreal world’s

35. Two Axes of Validation 2 – level of detail of the validation process –face validation: shows that mechansms and preperties of the model looks like the same as that of the real world –Emprical validation: make sure that the model generates data that can be demonstraed to similar to the corresponding pateterns in the real world

36. Illustration Flocking model (FM) in biology section of NetLogo library –regenerate patterns of birds focking –an ABM based on Boids models of Reynolds 1987 –birds flocks arise without being led by a special leader –each bird following exactly same set of rules from which flocks emerge –alignment, seperation, cohesion

37. the three rules alignment: –a bird turns so that it is moving in the same direction as nearby birds seperation: –a bird turns to avoid hitting another bird –overrides the other two: if two birds are approaching each other, they seperate cohesion: –a bird turns other nearby birds attect heading, speed of birds constant robust: swarming of insects, schooling of fish, V shaped flocking patterns of geeses

38. setup turtles-own [ flockmates ;; agentset of nearby turtles nearest-neighbor ;; closest one of our flockmates ] to setup clear-all crt population [ set color yellow - 2 + random 7 ;; random shades look nice set size 1.5 ;; easier to see setxy random-xcor random-ycor ] reset-ticks end

39. go to go ask turtles [ flock ] ;; the following line is used to make the turtles ;; animate more smoothly. repeat 5 [ ask turtles [ fd 0.2 ] display ] ;; for greater efficiency, at the expense of smooth ;; animation, substitute the following line instead: ;; ask turtles [ fd 1 ] tick end

40. flock to flock ;; turtle procedure find-flockmates if any? flockmates [ find-nearest-neighbor ifelse distance nearest-neighbor < minimum-separation [ separate ] [ align cohere ] ] end to find-flockmates ;; turtle procedure set flockmates other turtles in-radius vision end to find-nearest-neighbor ;; turtle procedure set nearest-neighbor min-one-of flockmates [distance myself] end

41. alignment to align ;; turtle procedure turn-towards average-flockmate-heading max-align-turn end to find-flockmates ;; turtle procedure set flockmates other turtles in-radius vision end to-report average-flockmate-heading ;; turtle procedure ;; We can't just average the heading variables here. ;; For example, the average of 1 and 359 should be 0, ;; not 180. So we have to use trigonometry. let x-component sum [dx] of flockmates let y-component sum [dy] of flockmates ifelse x-component = 0 and y-component = 0 [ report heading ] [ report atan x-component y-component ] end

42. seperation to separate ;; turtle procedure turn-away ([heading] of nearest- neighbor) max-separate-turn end

43. coherence to cohere ;; turtle procedure turn-towards average-heading-towards-flockmates max-cohere- turn end to-report average-heading-towards-flockmates ;; turtle procedure ;; "towards myself" gives us the heading from the other turtle ;; to me, but we want the heading from me to the other turtle, ;; so we add 180 let x-component mean [sin (towards myself + 180)] of flockmates let y-component mean [cos (towards myself + 180)] of flockmates ifelse x-component = 0 and y-component = 0 [ report heading ] [ report atan x-component y-component ] end

44. help procedures to turn-towards [new-heading max-turn] ;; turtle procedure turn-at-most (subtract-headings new-heading heading) max-turn end to turn-away [new-heading max-turn] ;; turtle procedure turn-at-most (subtract-headings heading new-heading) max-turn end ;; turn right by "turn" degrees (or left if "turn" is negative), ;; but never turn more than "max-turn" degrees to turn-at-most [turn max-turn] ;; turtle procedure ifelse abs turn > max-turn [ ifelse turn > 0 [ rt max-turn ] [ lt max-turn ] ] [ rt turn ] end

45. Macrovalidation vs. Microvalidation comparing actual and model generated data at different levels –agent level – microvalidation –macro/system level – macrovalidation E.g.: in FM ask agents has same behavior or properties as real birds –loocation, heading, flockmade, vision real birds many other properties –age, hungryk or not,.. –not relevant to the RQ

46. lRQ: whether agent fly like real birds so –lccation, heading, vision –2D or 3D Global properties of model and real birds in ABM agent interactions leads to global or emergent properties does the flocking patterns of model corresponds to reality

47. micro, macro or many levels in beteen –individual birs –flockmades reacts to each other –flocks as a whole in equation-based modeling only macrolevel

48. Face Validation vs. Empirical Validation ABM – reality not unique – many good models Face validation –look the model “on face” –conatins components behavior mechanisms corresponding real onces

49. Empirical validation –data generated by model corresponds to the data obtained from raal phenomena –measures and numerical data Statistical data analysis problem: defining inputs and outputs in real world may be dificult

50. Example FM for Face validation birds –micro level: move in streight line, change directions, decisions based on local information no complex calculations –macro level: flocks at the macro level are compared Emprical validation –angular momentum, size, velocity of flocks –comparable with real measures –depending on parameters and initial conditions model can generate data similar to real ones

51. Calibration is the process of Finding the parameters and initial conditions that generate data that are matched up with real world dafta use emprical data adjust parameters and mechaisms Emprical validation –micro and macro levels

52. Pattern Oriented Modeling matching patterns of real world at different levels with the onces generated by model many patterns at each level

53. Stochasticity, Variant and Invariabt Results and Path Dependence ABMs stochastic each repitition different ouputs compare raal data with distributions sometimes real wolrd has distributions use t or Kolmogrov Simirnov test for comparing ABM procuduces –variant and invariant results –E.g.: city development; south and morth sides may have differently developed

54. Variant results path dependent process –history greatly affects its final states E.g.: in city development western part is udervalued at the begining of a run –people migrate there –not path dependence – always happens if northen and soutern sides have equla chance of valueation –half of the runs north and south are migrated

55. Validation Benefits and Questions general principles about the real world changing parameters and mechanisms –experiments hard to do on real systems –easyly performend on validated models Validation –more or less relative make sense only with RQ – context of trhe question

56. Outline Correctness of a Model Verification Validation Replication Summary

57. Replication Replication of Computational Models: Dimensions and Standards Benefits of Replication Recommendations for Model Replicators Recommendations for Model Authors

58. Replication of Computational Models Experiments in scinece has to be replicated Important in computational modeling

59. Dimensions of Replication Wilensky and Rand 2007 time harware languages toolkits Algorithms Authors

60. Replication Strndards Axtell et al 1996 numerical identity distributional equivalence relational alignment only data cenral to the conceptual model is to be measured and compared how often results are compaired –at the end of run or at certain ticks

61. Benefits of Replication verification validation development of a shaired understaning of modeling

62. Recomandations for Model Replicators replication standards focal measures level of communication familiarity with language/toolkit of the orignal model Examination of the source code Expossure to original implemented model Exploration of parameter space

63. Recomandations to Model Authors level of detail of conceptual model specifications of details of the model model authorship/implementation availability of the model sensitivity analysis –none, –key parameters –all prameters varied

64. Outline Correctness of a Model Verification Validation Replication Summary

65. Verification –conceptual model – implemented model validation –our model – real world replication All increases confinece to the model