1 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Model Validation as an Integrated Social Process George P. Richardson Rockefeller College of Public Affairs and Policy University at Albany - State University of New York

2 Rockefeller College of Public Affairs and Policy University at Albany State University of New York What do we mean by ‘validation’? No model has ever been or ever will be thoroughly validated. …‘Useful,’ ‘illuminating,’ or ‘inspiring confidence’ are more apt descriptors applying to models than ‘valid’ (Greenberger et al. 1976). Validation is a process of establishing confidence in the soundness and usefulness of a model. (Forrester 1973, Forrester and Senge 1980).

3 Rockefeller College of Public Affairs and Policy University at Albany State University of New York The classic questions Not ‘Is the model valid,’ but Is the model suitable for its purposes and the problem it addresses? Is the model consistent with the slice of reality it tries to capture? (Richardson & Pugh 1981)

4 Rockefeller College of Public Affairs and Policy University at Albany State University of New York The system dynamics modeling process Adapted from Saeed 1992

5 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Processes focusing on system structure

6 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Processes focusing on system behavior

7 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Two kinds of validating processes

8 Rockefeller College of Public Affairs and Policy University at Albany State University of New York The classic tests Focusing on STRUCTURE Focusing on BEHAVIOR Testing SUITABILITY for PURPOSES Dimensional consistency Extreme conditions Boundary adequacy Parameter insensitivity Structure insensitivity Testing CONSISTENCY with REALITY Face validity Parameter values Replication of behavior Surprise behavior Statistical tests Contributing to UTILITY & EFFECTIVENESS Appropriateness for audience Counterintuitive behavior Generation of insights Forrester 1973, Forrester & Senge 1980, Richardson and Pugh 1981

9 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Validation is present at every step Conceptualizing: Do we have the right people? The right dynamic problem definition? The right level of aggregation? Mapping: Developing promising dynamic hypotheses Formulating: Clarity, logic, and extremes Simulating: Right behavior for right reasons Deciding: Implementable conclusions Implementing: Requires conviction!

10 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Do we have the right people?

11 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Problem frame stakeholder map Weak opponentsStrong opponents Weak supportersStrong supporters Weak Strong Stakeholder Power High Low High Opposition Support Problem Frame Bryson, Strategic Planning for Public and Nonprofit Organizations

12 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Power versus Interest grid SubjectsPlayers CrowdContext setters Weak Strong Power High Low Interest Eden & Ackerman 1998

13 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Pursuing validity in mapping Think causally, not correlationally Think stocks and flows, even if you don’t draw them Use units to make the causal logic plausible, even if you don’t write them down Be able to tell a story for every link and loop Move progressively from less precise to more precise -- from informal map to formal map

14 Rockefeller College of Public Affairs and Policy University at Albany State University of New York The standard cautions

15 Rockefeller College of Public Affairs and Policy University at Albany State University of New York These arrows mean ‘and then’ We start with some understandings of the problem and its systemic context, and then we conceptualize (map) the system. Then we build the beginnings of a model, which we then test to understand it. Then we reformulate, or reconceptualize, or revise our understandings, or do some of all three, and then continue…

16 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Arrows here are flows of material The words here represent stocks. This is not a causal diagram.

17 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Only this one is a causal loop No explicit stocks or flows, no clear units, but it tells a compelling story – It’s a good start.

18 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Project modeling core structure

19 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Identical structure without explicit stocks and flows

20 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Pursuing validity writing equations Recognizable parameters Robust equation forms Phase relations Richardson’s Rule: Every complicated, ugly, excessively mathematical equation and every equation flaw saps confidence in the model.

21 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Modeling conflict within & between nations

22 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Complexity & flaws destroy confidence P of int'l conflict = DELAY FIXED ((Lateral pressure/10*Military force effect/Trade and bargaining leverage + International conflict)/Lateral conflict break point, 1, 0) Flaws Complexity, discreteness, units confusion and disagreement, disembodied parameter, confusion of the effect of a concept [leverage] with the concept itself, and the wonder what keeps this probability between 0 and 1?

23 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Robust equation forms

24 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Causal mish-mash

25 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Robust equation formulations

26 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Robust equation formulations

27 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Robust equation formulations

28 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Robust equation formulations

29 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Pursuing validity in equations: Phasing

30 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Phase relations Constant Perceived Value suggests continually rising Resources, but that doesn’t seem correct

31 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Phase relations Here, the Perceived Value of Integrated Information sets a planned level of resources

32 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Pursuing validity fitting to data Generally, a weak test of model validity Whole-model procedures Optimization Partial-model procedures Reporting results Graphically Numerically: Theil statistics

33 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Example of weakness of fitting to data Logistic curve dx/dt = ax - bx 2 Gompertz curve dx/dt = ax - bx ln(x)

34 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Fitting global petroleum with Logistic

35 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Fitting global petroleum with Gompertz

36 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Presenting model fit visually

37 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Presenting model fit numerically Theil statistics, for example Based on a breakdown of the mean squared error: 1 = Bias + Variation + Covariation

38 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Presenting model fit numerically

39 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Learning from surprise model behavior Have clear a priori expectations Follow up all unanticipated behavior to appropriate resolution Confirm all behavioral hypotheses through appropriate model tests (Mass 1991/1981)

40 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Tests to reveal and resolve surprise behavior Testing the symmetry of policy response (up and down) Testing large amplitude versus small amplitude response Testing policies entering at different points Testing different patterns of behavior Isolating uniqueness of equilibrium or steady state Understanding forces producing equilibrium positions (Mass 1991/1981)

41 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Summary Modelers, stakeholders, problem experts, and others in the modeling process pursue validity at every step along the way. We have rigorous traditions guiding model creation, formulation, exploration, and implications. We have a powerful, intimidating battery of tests of model structure and behavior. Model-based conclusions that make it through all this deserve the confidence of everyone in the process.

42 Rockefeller College of Public Affairs and Policy University at Albany State University of New York Epilog Reason is itself a matter of faith. It is an act of faith to assert that our thoughts have any relation to reality. (G.K. Chesterton) I have no exquisite reason for’t, but I have reason good enough. (Sir Andrew, Twelfth Night)

43 Rockefeller College of Public Affairs and Policy University at Albany State University of New York References Greenberger, Crensen and Crissy (1976). Models in the Policy Process. New York: Russell Sage Foundation. Forrester, J. W. (1973). Confidence in Models of Social Behavior--With Emphasis on System Dynamics Models., M. I. T. System Dynamics Group. Forrester, J. W. and P. M. Senge (1980). Tests for Building Confidence in System Dynamics Models. System Dynamics. A. A. Legasto, Jr. et al., New York, North-Holland. 14: Richardson, G. P. and A. L. Pugh, III (1981). Introduction to System Dynamics Modeling with DYNAMO. Cambridge MA, Productivity Press. Reprinted by Pegasus Communications. Saeed, K. (1992). "Slicing a complex problem for systems dynamics modeling." System Dynamics Review 8(3): Bryson, J. (199x). Strategic Planning for Public and Nonprofit Organizations, citing Eden and Ackerman, Making Strategy (1998) and Anderson, Bryson, and Crosby (1999). Eden, C. and F. Ackerman (1998). Making Strategy. Mass, N. J. (1991/1981). "Diagnosing surprise model behavior: a tool for evolving behavioral and policy insights (1981)." System Dynamics Review 7(1):