Using uncertainty to test model complexity Barry Croke

Concept Possible to test whether a model has an appropriate complexity through analysis of scatter in residuals Take the standard deviation of the uncertainty distribution as a measure of uncertainty Divide each residual by its uncertainty (need uncertainty in both observed and modelled values)

Concept Result will be a distribution of values: If model has appropriate complexity, then the standard deviation of the distribution of normalised residuals should be near 1. If standard deviation of normalised residuals >> 1 then model is too simple. If standard deviation << 1 then model is too complex (has fitted to noise) Result depends on the accuracy and precision of the uncertainty estimates

Complications assumption is that the estimates of the uncertainties in the observed and modelled values are sufficiently well known. A systematic over-estimation of the uncertainties will result in a reduction in the standard deviation of the normalised residuals leading to a bias toward simpler models. an under-estimation of the uncertainties will bias the result to more complex models.

Complications Further, any random noise in the uncertainty estimates will tend to increase the standard deviation of the normalised residuals, biasing the result towards more complex models. Any structured noise in the uncertainty estimates may lead to an over- or under-estimation in the standard deviation of the normalised residuals. Must be evaluated based on the confidence in the estimated uncertainties in the observed and modelled values.