Fast N-Body Learning Nando de Freitas University of British Columbia

Historical Perspective Non-iterative or “direct” methods for eigenvalue problems and linear systems of equations require O(N 3 ) operations. Let's look at the history of what has been regarded as large N: 1950: N= : N= : N= : N=20000 So over the course of 45 years N has increased by a factor of However, the speed of computers has increased by a factor of From this the O(N 3 ) bottleneck is evident. If only we could reduce the cost to O(N) – sigh!

Krylov for Eigen-Problems

Krylov for Systems of Equations

N-Body Problems in Learning Sum-kernel problem: Max-kernel problem:

N-Body Problems

Obvious applications of N-body Learning Exact and approximate message propagation. Markov chain Monte Carlo Gaussian processes, Wishart processes and Laplace processes. Spectral learning: eigenmaps, SNE, NCUTS, ranking on manifolds, … (even if using Nystrom) Reinforcement learning. The E step.

Kernel-(fill in your favourite name). Rao-Blackwellised Monte Carlo. Nearest neighbour methods. Some types of boosting. Computer graphics. EM, fluid dynamics, gravitation, quantum systems. … and much more ! Obvious applications of N-body Learning

Illustrative Example: Zhu, Lafferty & Zoubin

Illustrative Example Energy function using the graph Laplacian: Easy, but … a big linear system: Naïve iterative solution:

Illustrative Example We have solved a Gaussian process (where the covariance is the inverse graph Laplacian in O(N).

Illustrative Example

Message propagation Whether it’s exact: … or approximate:

Fast Methods in this Workshop

Fast Multipole Methods

Recursive Tree Structures

Distance Transform m(j) = min ( w(i) + d(i,j) ) i

In this workshop You’ll encounter tutorials on fast methods from the people who’ve been developing them. You’re likely to see encounter people arguing over error bounds, implementation strategies, applications and many more things. You’ll see statistics, learning, data structures and numerical computation come together. You’ll dream of the powder up on the hill.