1. Cost of surrogates
In linear regression, the process of fitting involves solving a set of linear equations once.
For moving least squares, we need to form and solve the system at every prediction point.
With radial basis neural networks we have to optimize the selection of neurons, which will again entail multiple solutions of the linear system. We may find the best spread by minimizing cross-validation errors.
Kriging, our next surrogate is even more expensive, we have a spread constant in every direction and we have to perform optimization to calculate the best set of constants.
With many hundreds of data points this can become significant computational burden.

2. Kriging philosophy
We assume that the data is sampled from an unknown function that obeys simple correlation rules.
The value of the function at a point is correlated to the values at neighboring points based on their separation in different directions.
The correlation is strong to nearby points and weak with far away points, but strength does not change based on location.
This is often grossly wrong because a function may be fast undulating in one corner of the design space and vary slowly in another corner.
Still, Kriging is a good surrogate, and it may be the most popular surrogate in academia.
Normally Kriging is used with the assumption that there is no noise so that it interpolates exactly the function values.
It works out to be a local surrogate, and it uses functions that are very similar to the radial basis functions.

3. Reminder: Covariance and Correlation
Covariance of two random variables X and Y
The covariance of a random variable with itself is the square of the standard deviation
Covariance matrix for a vector contains the covariances of the components
Correlation
The correlation matrix has 1 on the diagonal.

4. Correlation between function values at nearby points
x=10*rand(1,10)
xnear=x+0.1; xfar=x+1;
ynear=sin(xnear)
y=sin(x)
yfar=sin(xfar)
rfar=corrcoef(y,yfar)
0.4229
r=corrcoef(y,ynear)
0.9894

5. Gaussian correlation function
Correlation between point x and point s
y10=sin(10*x); y10near=sin(10*xnear)
r10=corrcoef(y10,y10near)
0.4264
For the function we would like to estimate

6. Universal Kriging
Linear trend model
Systematic departure x y
Kriging
Sampling data points
Systematic Departure
Linear Trend Model
Named after a South African mining engineer D. G. Krige
Assumption: Systematic departures Z(x) are correlated
Gaussian correlation function C(x,s,θ) is most popular

7. Simple Kriging
Kriging started without the trend, and it is not clear that one cannot get by without it.
Simple Kriging is uses a covariance structure with a constant standard deviation.
The most popular correlation structure is Gaussian
The standard deviation measures the uncertainty in function values. If we have dense data, that uncertainty will be small, and if the data is sparse the uncertaty will be large.
How do you decide whether the data is sparse or dense?

8. Prediction and shape functions
Simple Kriging prediction formula
R is the correlation matrix of the data points.
The equation is linear in r, which means that the basis functions are the exponentials
The equation is linear in y, which is in common with linear regression.

9. Prediction variance
Square root of variance is still called standard error
The uncertainty at any x is still normally distributed. 9

10. Finding the thetas The thetas and sigma must be found by optimization.
Maximize the likelihood of the data.
For a given curve, we can calculate the probability that if the curve is exact, we would have sampled the data.
Minimize the cross-validation error.
Each set of theta acts like a different surrogate.
Both problems are ill-conditioned and expensive for large number of data points.
Watch for thetas reaching their higher bounds!
Prediction variance equation does not count for the uncertainty in the theta values.