Gaussian process emulation of multiple outputs Tony O’Hagan, MUCM, Sheffield
Outline Gaussian process emulators Simulators and emulators GP modelling Multiple outputs Covariance functions Independent emulators Transformations to independence Convolution Outputs as extra dimension(s) The multi-output (separable) emulator The dynamic emulator Which works best? An example
Simulators and emulators A simulator is a model of a real process Typically implemented as a computer code Think of it as a function taking inputs x and giving outputs y y = f(x) An emulator is a statistical representation of the function Expressing knowledge/beliefs about what the output will be at any given input(s) Built using prior information and a training set of model runs The GP emulator expresses f as a GP Conditional on hyperparameters
GP modelling Mean function Regression form h(x) T β Used to model broad shape of response Analogous to universal kriging Covariance function Stationary Often use the Gaussian form σ 2 exp{-(x-x ′ ) T D -2 (x-x ′ )} D is diagonal with correlation lengths on diagonal Hyperparameters β, σ 2 and D Uninformative priors
The emulator Then the emulator is the posterior distribution of f After integrating out β and σ 2, we have a t process conditional on D Mean function made up of fitted regression h T β* plus smooth interpolator of residuals Covariance function conditioned on training data Reproduces training data exactly Important to validate Using a validation sample of additional runs Check that emulator predicts these runs to within stated accuracy No more and no less Bastos and O’Hagan paper on MUCM website
Multiple outputs Now y is a vector, f is a vector function Training sample Single training sample for all outputs Probably design for one output works for many Mean function Modelling essentially as before, h i (x) T β i for output i Probably more important now Covariance function Much more complex because of correlations between outputs Ignoring these can lead to poor emulation of derived outputs
Covariance function Let f i (x) be i-th output Covariance function c((i,x), (j,x ′) ) = cov[f i (x), f j (x ′ )] Must be positive definite Space of possible functions does not seem to be well explored Two special cases Independence: c((i,x), (j,x ′) ) = 0 if i ≠ j No correlation between outputs Separability: c((i,x), (j,x ′) ) = σ ij c x (x, x ′ ) Covariance matrix Σ between outputs, correlation c x between inputs Same correlation function c x for all outputs
Independence Strong assumption, but... If posterior variances are all small, correlations may not matter How to achieve this? Good mean functions and/or Large training sample May not be possible in practice, but... Consider transformation to achieve independence Only linear transformations considered as far as I’m aware z(x) = A y(x) y(x) = B z(x) c((i,x), (j,x ′) ) is linear mixture of functions for each z
Transformations to independence Principal components Fit and subtract mean functions (using same h) for each y Construct sample covariance matrix of residuals Find principal components A (or other diagonalising transform) Transform and fit separate emulators to each z Dimension reduction Don’t emulate all z Treat unemulated components as noise Linear model of coregionalisation (LMC) Fit B (which need not be square) and hyperparameters of each z simultaneously
Convolution Instead of transforming outputs for each x separately, consider y(x) = ∫ k(x,x*) z(x*) dx* Kernel k Homogeneous case k(x-x*) General case can model non-stationary y But much more complex
Outputs as extra dimension(s) Outputs often correspond to points in some space Time series outputs Outputs on a spatial or spatio-temporal grid Add coordinates of the output space as inputs If output i has coordinates t then write f i (x) = f*(x,t) Emulate f* as single output simulator In principle, places no restriction on covariance function In practice, for single emulator we use restrictive covariance functions Almost always assume separability -> separable y Standard functions like Gaussian correlation may not be sensible in t space
The multi-output emulator Assume separability Allow general Σ Use same regression basis h(x) for all outputs Computationally simple Joint distribution of points on multivariate GP have matrix normal form Can integrate out β and Σ analytically
The dynamic emulator Many simulators produce time series output by iterating Output y t is function of state vector s t at time t Exogenous forcing inputs u t, fixed inputs (parameters) p Single time-step simulator f* s t+1 = f*(s t, u t+1, p) Emulate f* Correlation structure in time faithfully modelled Need to emulate accurately Not much happening in single time step but need to capture fine detail Iteration of emulator not straightforward! State vector may be very high-dimensional
Which to use? Big open question! This workshop will hopefully give us lots of food for thought MUCM toolkit v3 scheduled to cover these issues All methods impose restrictions on covariance function In practice if not in theory Which restrictions can we get away with in practice? Dimension reduction is often important Outputs on grids can be very high dimensional Principal components-type transformations Outputs as extra input(s) Dynamic emulation Dynamics often driven by forcing
Example Conti and O’Hagan paper On my website: Time series output from Sheffield Global Dynamic Vegetation Model (SDGVM) Dynamic model on monthly timestep Large state vector, forced by rainfall, temperature, sunlight 10 inputs All others, including forcing, fixed 120 outputs Monthly values of NBP for ten years
Multi-output emulator on left, outputs as input on right For fixed forcing, both seem to capture dynamics well Outputs as input performs less well, due to more restrictive/unrealistic time series structure
Conclusions Draw your own!