1. aims: - evaluate typical properties in controlled model situations - gain general insights into machine learning problems - compare algorithms in controlled environments - optimize and develop novel training algorithm Theory of Learning - analysis of model scenarios specify… - learning problem and student complexity (student / teacher scenarios) - statistics of observed data - cost function and/or training algorithm complements other approaches - e.g. “assumption free” bounds on generalization behavior in VC-Theory etc.

2. essential ingredients: - large systems with many adaptive parameters - high-dimensional input data - perform average over stochastic training procedures - perform average over randomized data set Statistical Physics ? history: - equilibrium/dynamics of recurrent networks (Hopfield 1982) - physics of interactions between neurons (Gardner 1988) - learning a rule with a perceptron (Vallet 1989) - on-line learning dynamics (Kinzel/Rujan 1990) → obtain typical (average behavior of large systems in controlled model scenarios

3. The Central Limit Theorem (in its simplest form) consider: independent identically distributed (i.i.d.) random numbers x i The sum of many random numbers is a Gaussian with mean and variance The sum becomes for M→∞ a Gaussian random quantity with density

4. possible extensions: - non-zero mean values (trivial) - i-dependent variance (condition: finite, same order of magnitude) - sums of weakly correlated random numbers most important point: - details of the statistics of x i are irrelevant - could be Gaussians themselves, binary x i =±1, uniform in [-a,a] … CLT applies effectively already very quite small M (“>12”) important in the following: correlated sums, e.g. with given coefficients a i,b i