Getting started with GEM-SA

GEM-SA course - session 32 This talk Starting GEM-SA program Creating input and output files Explanation of the menus, toolbars, etc. Description of the project window

GEM-SA course - session 33 Starting GEM-SA Double-click the GEM-SA icon to start The main window appears, with Menu Toolbar Main results area with three tabs Sensitivity Analysis, Main Effects and Results Summary Initially all empty Log window

GEM-SA course - session 34 menu Log window toolbar Sensitivity analysis output grid The main GEM-SA window

GEM-SA course - session 35 Toolbar icons New project Open project Save project Print output report Edit project Generate input design points Rescale an input Standardise design Copy input design to clipboard Convert input to integer Run the analysis Help

GEM-SA course - session 36 Output tabs When an emulator has been fitted, the contents of these tabs will provide the main results Sensitivity Analysis. This will report the SA variance decompositions One line for each input parameter One line for each pair of inputs, if joint effects are selected Main effects. This will plot the main effects of the various inputs Results Summary. This will present numerical summaries of emulator fit and uncertainty analysis

GEM-SA course - session 37 Log Window output Tells us Which training data are being loaded/saved Transformations applied to the data Fitted Gaussian process parameters Summary of cross-validation analysis Summary of the uncertainty analysis

GEM-SA course - session 38 Creating a GEM project To build the emulator we first need 3 files: Data file of code inputs Data file of code outputs GEM-SA project file

GEM-SA course - session 39 Restrictions on input/output data Single output Multiple outputs must be treated individually GEM can read multiple outputs file, but a single column is specified within a project Max 30 input parameters Max 400 training points The data files are plain text files One row for each point Rows can be space or tab delimited

GEM-SA course - session 310 Generating a new input design Designs can be generated using the toolbar icon or the menu: Input  Generate… The design dialog appears

GEM-SA course - session 311 Generating a new input design Click OK and fill in the required range for each input Click OK again

GEM-SA course - session 312 Editing input designs If you select a column, you can rescale values of that input or round values to be integers Designs can be loaded into or saved from this window using the Inputs menu. Use to copy the points to the clipboard for use in other programs

GEM-SA course - session 313 Types of design GEM-SA can generate 2 types of design LP-  Maximin Latin Hypercube designs Both have good space-filling properties Ensure all regions of the input space are well represented

GEM-SA course - session 314 LP-  design Very quick to generate Deterministic set of uniform points Increasing the sample size just adds points to the smaller design Making it useful for sequential analysis Only have to generate the extra runs

GEM-SA course - session 315 Maximin Latin hypercube design Maximin Latin Hypercube designs Maximise the minimum distance amongst all pairs of points Can take a long time to generate Projections also generally space-filling Lower dimensional projections are also latin hypercubes Good when only a few inputs are active

GEM-SA course - session 316 Creating output points Each row from the input design must be used to generate outputs by running the computer code One run for each row Various methods to automate this: Spreadsheet Simple, but requires functional form Script Only need executable code Loop through inputs, modify code input file Modify code to loop through the points Messy, need source code

GEM-SA course - session 317 Example: using a spreadsheet Copy the input design to the clipboard using Open Excel and paste inputs Create formula in final column Copy formula for all rows of the design Cut and paste special (values) in a new sheet Save as text file

GEM-SA course - session 318 Example: using a script Read simulator’s base input file Read training inputs file Loop through training file lines Replace target inputs using training line Write new base input file Run code Calculate output(s) and add to training output file

GEM-SA course - session 319 my $pftchangeline = 21; # change line 21 within the input file for each run = (11,14,23,19); # columns within pftchangeline to modify = (0,1,2,3); # ordering of these parameters within training inputs open(BASEINFILE, "input.dat"); # getinitial (fixed) input file used by sdgvmd = ; # and store the input lines close BASEINFILE; open(LHFILE, "training_inputs.txt"); my $newpftline = $lines[$pftchangeline]; = split(" ", $newpftline); while ( ){ # assigns each line in turn to $_ chomp; split; open(INFILE, "> inputfile.dat"); = # modify lines $lines[$pftchangeline] = join(' print close INFILE; `sdgvm0 input.dat`; # run sdgvm0 with modified input # now do something with the output files }

GEM-SA course - session 320 The project window Appears whenever you Load a project Edit a project Create a new project This window also has 3 tabs Files Options Simulations

GEM-SA course - session 321 Names for the input files Names for the output files

GEM-SA course - session 322 How many inputs? What are the input names? Which column from output file?

GEM-SA course - session 323 Which joint effects should be calculated? What should be calculated, and how?

GEM-SA course - session 324 How are the inputs uncertain? What prior mean for the output?

GEM-SA course - session 325 What kind of prediction? What kind of cross validation?

GEM-SA course - session 326 MCMC control parameters How many points used to calculate main effects, joint effects How many realisations of predictions, main and joint effects to generate

The options tab

GEM-SA course - session 328 Input parameter names This window appears if you press the Names… button Giving names is optional, but useful later when looking at GEM-SA output Ordering can be changed using the arrows

GEM-SA course - session 329 Selecting joint effects Select calculate joint effects if in sensitivity analysis you want to see the joint effects (interactions) of pairs of inputs as well as their individual effects Use Inputs to include in joint effects panel to select which ones Default All inputs computes joint effects for all pairs Can take a lot of computation To compute only the joint effects between selected inputs, deselect All inputs and select the two or more inputs whose joint effects are of interest

GEM-SA course - session 330 Other checkboxes Sum effects There are two ways to plot the joint effect of two inputs: A combined effect in which the value plotted is the mean output value at that combination of input values A pure interaction, in which with the main effects of those inputs are subtracted from the combined effect Select sum effects if you want to see combined effects, and deselect it to see interactions This selection is ignored if you don’t ask for joint effects to be computed

GEM-SA course - session 331 Other checkboxes Code has numerical error We generally assume that the model output is computed exactly every time So the meta-model passes exactly through all the training points There are two situations in which this assumption is not right Your code has numerical errors which you want to smooth out Your code is stochastic and the output values have random noise Selecting code has numerical error turns the assumption off The variance of the error will be estimated as part of the fitting process The meta-model will smooth out the training points to a degree depending on the estimated error variance Can make the fitting process quite unstable, so beware!

GEM-SA course - session 332 Other checkboxes Use MCMC for emulator parameters By default, GEM-SA estimates the underlying smoothness parameters and then pretends that the estimates are exact Selecting use MCMC for emulator parameters takes into account uncertainty in the fitting of the emulator Slows down the computation substantially, often with minimal effect on the results Auto-tune Metropolis algorithm Use only with MCMC If not selected, you must supply a tuning file

GEM-SA course - session 333 Input uncertainty options These options are for specifying what kind of distribution each uncertain input has There are a limited range of options All unknown, product normal/uniform Inputs are independent, with either normal or uniform distributions All known No uncertainty analysis required Some known, rest product normal/uniform Some input values will be fixed (in the dialog window or in a prediction file) Others will be given independent distributions, either normal or uniform

GEM-SA course - session 334 Input uniform ranges If you say that some or all have uniform distributions, a window appears (when you click OK) to specify ranges Option to use ranges in input data file All uniform Some fixed, rest uniform

GEM-SA course - session 335 Input normal parameters If you say that some or all have normal distributions, a window appears (when you click OK) to specify the mean and variance of each distribution Option to use ranges in input data file All normal Some fixed, rest normal

GEM-SA course - session 336 Prior mean options The emulator will fit better if it knows roughly how the output is expected to respond to the inputs You have just two choices If you expect to see a trend in the output in response to changes in its inputs, select linear term for each input Otherwise, selecting constant mean results in no overall trends being expected or fitted

GEM-SA course - session 337 Selecting prediction type Having fitted the Gaussian process emulator, GEM-SA can predict what the output would be if the computer code were run at new input sets These are specified in a prediction file If there is no prediction file, selecting the prediction type has no effect Predictions can be Simulated realisations of outputs at the prediction inputs Similar to main effect outputs Takes account of correlation between predictions Marginal means and variances of outputs at the prediction inputs Faster to compute, especially with many prediction points Easy to interpret

GEM-SA course - session 338 Selecting cross validation type Cross-validation is a way of checking the validity of the predictions made by GEM-SA The idea is to fit the emulator leaving out some of the training data points, then predict the missing points and see how well the predictions do Choice of none, leave-one-out or leave final 20% out Leave-one-out Hyper-parameters use all data and are then fixed when prediction is carried out for each omitted point Leave final 20% out Hyper-parameters are estimated using the reduced (80%) data subset

The files and simulations tabs

GEM-SA course - session 340 GEM-SA files You always have to specify an Inputs File and an Outputs File You only need to specify a Prediction Inputs File if you want to generate predictions You only need to specify a Metropolis-Hastings Tuning File if you select MCMC for computation and deselect auto-tuning The Main effects file will always be created when you do sensitivity analysis The Joint Effects file will be created if you ask for joint effects to be computed The Predictions File will be created if you ask for predictions (by specifying a Prediction Inputs File) It will contain simulated predictions or prediction means The Predictions Variance File is created if you ask for predictions and specify prediction means and variances

GEM-SA course - session 341 Simulations The first three of these settings apply only if you select MCMC computation For expert users only! You could choose the number of simulations that are computed for each main effect and interaction But the default is generally plenty You might want to increase the number of points on each main effect To get more detail in the plots But at the cost of longer computations