1. Lecture 10: Gaussian Processes
CS480/680: Intro to ML
Lecture 10: Gaussian Processes
24/02/2019
Yao-Liang Yu

2. Outline Gaussian Distribution Gaussian Process
Gaussian Linear Regression
Advanced
24/02/2019
Yao-Liang Yu

3. Announcement
Assignment 3 due on Nov 08.
24/02/2019
Yao-Liang Yu

4. Gaussian distribution
covariance, PSD
mean
dimension
24/02/2019
Yao-Liang Yu

5. Carl Friedrich Gauss (1777 – 1855)
24/02/2019
Yao-Liang Yu

6. Important facts
marginal
joint
conditional
24/02/2019
Yao-Liang Yu

7. Derivation
24/02/2019
Yao-Liang Yu

8. Outline Gaussian Distributions Gaussian Processes
Gaussian Linear Regression
Advanced
24/02/2019
Yao-Liang Yu

9. What is a random variable?
A random variable is a function Z(ω) ω
Z(ω)
24/02/2019
Yao-Liang Yu

10. Gaussian process
A collection of Gaussian random variables {Zt : t in T} such that for any finite N, {Zt : t in N} is jointly Gaussian
A Gaussian process is a function of two variables Z(t,ω)
For any finite N, {Zt := Z(t,ω) | t in N} is a Gaussian random vector
For any ω, Zω := Z(t, ω) : T  R is a function of one variable t (sample path)
Does Gaussian process exist?
24/02/2019
Yao-Liang Yu

11. Example
24/02/2019
Yao-Liang Yu

12. Mean and covariance function
For each t, Z(t,ω) is a Gaussian random variable hence has mean
For each s and t, the covariance between Zs and Zt:
Say
mean function
covariance function
24/02/2019
Yao-Liang Yu

13. Recap: verifying a kernel
For any n, for any x1, x2, …, xn, the kernel matrix K with
is symmetric and positive semidefinite ( )
Symmetric: Kij = Kji
Positive semidefinite (PSD): for all
24/02/2019
Yao-Liang Yu

14. What is a covariance function?
For t1, t2, …, tn, by definition is the covariance between and
K is symmetric and PSD
Thus, the covariance function κ is a kernel!
24/02/2019
Yao-Liang Yu

15. Conversely Theorem. Given any function and kernel function , exist
This may not hold for other distributions !!!
24/02/2019
Yao-Liang Yu

16. Andrey Kolmogorov (1903 - 1987) Foundations of the theory probability
24/02/2019
Yao-Liang Yu

17. Effect of kernel
24/02/2019
Yao-Liang Yu

18. Example: Linear Regression
unknown but deterministic
independent of each other
for different x Z t
Y is a Gaussian process
24/02/2019
Yao-Liang Yu

19. Maximum Likelihood Having observed Need to estimate w
Choose w that explains the observations best
i.i.d.
likelihood of data
24/02/2019
Yao-Liang Yu

20. Example: Bayesian Linear Regression
independent of w
independent of each other
for different x Z t
Y is a Gaussian process
24/02/2019
Yao-Liang Yu

21. Maximum A Posteriori
likelihood of data
prior
posterior
normalization
Different prior on w leads to different regularization
w ~ N(0, I) is equivalent as ridge regression
w ~ Lap(0, I) is equivalent as Lasso
24/02/2019
Yao-Liang Yu

22. Outline Gaussian Distributions Gaussian Process
Gaussian Linear Regression
Advanced
24/02/2019
Yao-Liang Yu

23. Abstract View Let be a Gaussian process Having observed
Need to predict
24/02/2019
Yao-Liang Yu

24. Familiar view Equivalent in finite dimensions
Feature map of k
Equivalent in finite dimensions
Incorrect but “intuitive” in infinite dimensions
24/02/2019
Yao-Liang Yu

25. Back to abstract is jointly Gaussian What is the distribution of ?
24/02/2019
Yao-Liang Yu

26. Recap: Important facts
marginal
joint
conditional
24/02/2019
Yao-Liang Yu

27. Recap: Example
24/02/2019
Yao-Liang Yu

28. Application t = (position, velocity, acceleration) Z = torque
24/02/2019
Yao-Liang Yu

29. Outline Gaussian Distributions Gaussian Process
Gaussian Linear Regression
Advanced
24/02/2019
Yao-Liang Yu

30. Gaussian process classification
Binary label Y generated by
Zt is a Gaussian process (real-valued)
Given observations
Need to predict
integrate out
latent variable
24/02/2019
Yao-Liang Yu

31. Questions?
24/02/2019
Yao-Liang Yu