MODEL ASSESSMENT AND VALIDATION

Model Validation Models should be based on objective research, expressed mathematically and should be shown to be useful and pass critical tests. Declaration of Independence: Modelers are independent of the system?

Validation Errors Null Hypothesis True Null Hypothesis False Reject Null HypothesisType I ErrorCorrect Fail to Reject Null Hypothesis CorrectType II Error

Example Defendant InnocentDefendant Guilty Guilty VerdictType I ErrorCorrect Not Guilty VerdictCorrectType II Error

Type III Error The modeler sets out on the wrong track altogether. The result is that the model does completely the wrong thing and is of no value at all.

Open Box (White-box) Validation We know the inner workings of our model – compare to Black-box

Validation Difficulties The theory we are assuming for the real application is complex When model is designed to look at modifications and new circumstances, we do not know how things will react in the real world. Prediction issues

Interpretive Models 3 R’s Reductionism: complex systems can be reduced to simpler component parts without loss of information. Repeatability: Can we achieve repeatability – the scientific principle. Refutation: no experiment can prove conclusively that a theory is true. (Karl Popper)

Cognitive Maps Validation Cognitive Maps (A person’s mental image of a situation) If people cannot have common concepts in their respective Cognitive Maps – are the maps invalid?