Thoughts on Model Validation for Engineering Design George A. Hazelrigg

Why bother? Mathematical models are all we have to support engineering design They had better be up to the task We need to have confidence that they are This demands “validation”

Two different kinds of modeling Descriptive models –They replicate experimental results –They look at the past –They are “validated” using data Predictive models –They predict the future –There are no data to validate them

Descriptive models We teach them We validate them We work with them We develop them They are generally deterministic We tend to understand them

Predictive models We don’t teach them We don’t know how to validate them They are always probabilistic We don’t know how to use them We really don’t understand them But all design is based on them

What is not validation? Data points X t

Some notions Law of nature—a fundamental understanding of causality, d(mV)/dt=F Symbolic or mathematical model—a construct that comprises an abstract representation of something Rational—self consistent Laws of nature are invariant and fixed over all time and space

More notions We use predictive models in support of decision making A predictive model is good if it enables us to make decisions that get us the “best” outcomes Predictive model validation must be in terms of decisions Events are fleeting, symbolic models are case-specific

Models produce information Information relates to a specific decision— it is what the decision is based on. Information can be measured as the probability that the preferred choice in a specific decision with specific alternatives will lead to the outcome most desired from among the outcomes actually achievable from the available alternatives

Information

Information in large alternative sets H O =1 H O <1

“Valid” defined valid (val ! id) adj. 1. Well grounded; just: a valid objection. 2. Producing the desired results; efficacious: valid methods. 3. Having legal force; effective or binding: a valid title. 4. Logic. a. Containing premises from which the conclusion may logically be derived: a valid argument. b. Correctly inferred or deduced from a premise: a valid conclusion. 5. Archaic. Of sound health; robust. [French valide, from Old French, from Latin validus, strong, from valre, to be strong. We will define a model to be valid if, using the model, in a choice between two alternatives, we are guaranteed of selecting the alternative that will yield the better outcome

Notion of model validity Region of model validity H O <1 H O =1

So what? So, what we are interested in are the statistical properties of ( (x)

An example—M&Ms in a jar We might feel that, because of edge effects, : * / : is slightly greater than one if the data were obtained from a jar that is smaller than the “design” jar. So, perhaps, we would assign it a distribution with a mean of, say, 1.01 and a standard deviation of, say, We might take the ratio V c /V c * to have mean 1.0 and standard deviation 0.005, and the ratio V j * /V j is a random variable with mean equal to the estimated ratio of the volumes of the jars and a standard deviation reflective of our ability to estimate this ratio.

More example The ratio N/N * is a trickier quantity to understand. It reflects our expectation on how the models for n and n * apply to their respective cases. In this case, the models for n and n * are mathematically identical, and they are precise models. All the model error is the result of inaccuracy in our ability to precisely estimate the input data. Thus, N/N * is deterministic and takes on the value of our estimated ratio V j /V j *. This model thus allows estimation of the distribution on (. We must subjectively estimate how good our model is

Conclusions Validity of predictive models is determined in the case of specific decisions Validity and information take on meaning only in the context of decisions Validation is a down and dirty personal thing—it is subjective But we still need to be consistent—i.e., rational

An insight for engineering education The validity of all predictive models is estimated only subjectively Therefore, good judgment is the key to good modeling for design Engineering education needs to have a focus on good judgment in modeling