1. Calibration of Computer Simulators using Emulators

2. Recap –Emulators We are concerned with complex, non-linear simulators In this session we will look at calibration of such simulators We will heavily depend on emulators An emulator is a Gaussian process (or second order process) that interpolates the simulator output Emulators are fast EGU short course - session 42

3. Calibration Simulator users often want to tune the simulator using observations of the real system Adjust the input parameters so that the simulator output matches observations as well as possible Two very important points 1. Calibration will reduce uncertainty about x but will not eliminate it 2. It is necessary to understand how the simulator relates to reality Model discrepancy EGU short course - session 43

4. Calibration and Assimilation Calibration is concerned with the values of the inputs that are consistent with the data. Assimilation is concerned with producing the best forecast/hindcast Calibration changes the simulator inputs Assimilation changes the simulator state variables EGU short course - session 44

5. Model discrepancy Simulator output y = f(x) will not equal the real system value z Even with best/correct inputs x Model discrepancy is the difference z – f(x) As discussed in Session 1, model discrepancy is due to Wrong or incomplete science Programming errors, rounding errors Inaccuracy in numerically solving systems of equations Ignoring model discrepancy leads to poor calibration Over-fitting of parameter estimates Over-confidence in the fitted values EGU short course - session 45

6. History matching History matching (a term taken from the petroleum industry) means finding sets of inputs that given simulator outputs that are ‘compatible’ with data Calibration means finding a best value (or a distribution) for the inputs given the data EGU short course - session 46

7. Implausibility Define a measure of implausibility ( I mp ) If the implausibility is greater then ±3 those values of the inputs are deemed implausible Because this is a function of the emulator not the original simulator runs we calculate it everywhere in input space EGU short course - session 47

8. Waves of Implausibility Wave 1: Apply the implausibility measure. Mark part of input space as implausible Wave 2: Add extra points in the not implausible region and rebuild the emulator. Repeat the implausibility measure Wave 3+: Repeat until the implausible region ceases to grow EGU short course - session 4 8

9. A 1-d example EGU short course - session 49

10. 10

11. EGU short course - session 411

12. EGU short course - session 412

13. EGU short course - session 413

14. EGU short course - session 414

15. EGU short course - session 415

16. Example -Galform EGU short course - session 416

17. Example - Galform Galform is a simulator of Galaxy formation It has 17 inputs The amount of not implausible space in each wave is None of the original 1000 member LHC was an acceptable fit to the data EGU short course - session 417 Wave 114.9% Wave 25.9% Wave 31.6% Wave 40.26% Wave 50.036%

18. Calibration In history matching we were simply looking for regions of input space that were not implausible given the data. In calibration we want to find the ‘best input’ x (and it associated uncertainty) EGU short course - session 418

19. Kennedy and O’Hagan(2001) ζ is the real system z=ζ+ε is data on the real system ( ε~N(0,σ 2 ) ) y=f(x) is the simulator output d=ζ-y is the model discrepancy ζ=f(x)+d Build an emulator for f and simultaneously model the discrepancy as a GP ζ=f * (x)+d * z=f * (x)+d * +ε EGU short course - session 419

20. Kennedy and O’Hagan (2001) -2 We can now perform an uncertainty analysis This shows how much we have learned about the simulator inputs from the data The mean/mode of the posteriors give us our estimate of the best value for the inputs EGU short course - session 420

21. Model Discrepancy Revisited We have seen that we can use the model discrepancy to calibrate/history match the simulator We can also look at the discrepancy between different simulators This is particularly interesting if we have hierarchies of simulators EGU short course - session 421

22. Hierarchies of Simulators Often we have hierarchies of simulators Usually the resolution is increasing but additional processes could be added EGU short course - session 422

23. Hierarchies of Simulators Rather than emulate each simulator separately Emulate simulator 1 and then emulate the difference between each level Need to have some runs at common inputs Need few runs of expensive complex simulators EGU short course - session 423

24. Reified Simulators EGU short course - session 424

25. Reified Simulators EGU short course - session 425

26. Reified Simulators EGU short course - session 426

27. Reified Simulators Reified simulators are ‘imaginary’ simulators that we impose between our simulators and reality They are the ‘best’ simulator we could produce Model discrepancy is split into two: 1. The discrepancy between the current simulator and the reified simulator 2. The discrepancy between the reified simulator and reality Reification does not reduce the discrepancy. It might make it easier to elicit. EGU short course - session 427

28. Overview Emulators are useful tools in the calibration of complex simulators Two methods have been described: History Matching – ruling out implausible regions of input space Calibration – Finding ‘best fit’ input values Reification may be useful in eliciting the relationship between simulators and reality EGU short course - session 428