Math Primer

Outline Calculus: derivatives, chain rule, gradient descent, taylor expansions Bayes Rule Fourier Transform Dynamical linear systems

Calculus Derivatives Derivative=slope

Calculus Derivative: a few common functions (x n )’=nx n-1 (x -1 )’=-1/x 2 = x -2 exp(x)’=exp(x) log(x)’=1/x cos(x)’=-sin(x)

Calculus Derivative: Chain rule Ex: gaussian z=h(x)=exp(-x 2 )’,f(y)=exp(y), y=g(x)=-x 2 f’(y)=exp(y)= exp(-x 2 ), g’(x)=-2x z’= f’(y)g’(x) =exp(-x 2 )(-2x)

Calculus Finding minima: gradient descent f(x) f’(x 0 ) < 0 x0x0 x* x0+xx0+x  x > 0 f’(x*)=0  x = -  f’(x 0 )

Calculus Example: minimizing an error function

Calculus Taylor expansion

Calculus

Bayes rule Example: drawing from 2 boxes 2 boxes (B1,B2) P(B1)=0.2,P(B2)=0.8 (Prior) Balls with two colors (R,G) B1=(16R,8G), B2=(8R,16G) P(R|B1)=2/3, P(G|B1)=1/3 (Conditional) P(R|B2)=1/3, P(G|B2)=2/3 (Conditional)

Bayes rule Joint distributions P(G,B1)=P(B1)P(G|B1)=0.2*0.33=0.066 P(G,B2)=P(B2)P(G|B2)=0.8*0.66=0.528 P(X,Y)=P(X|Y)P(Y) P(Y,X)=P(Y|X)P(X) P(Y|X)P(X)=P(X|Y)P(Y)

Bayes rule P(Y|X)P(X)=P(X|Y)P(Y) P(Y|X)=P(X|Y)P(Y)/P(X) If you draw G, what is the probability that it came from box1? P(B1|G)=P(G|B1)P(B1)/P(G) How do you get this? Marginalize

Bayes rule Marginalization P(G)=P(G,B1)+P(G,B2)= =0.6 P(G)=P(G|B1)P(B1)+P(G|B2)P(B2) P(Y)=  x P(Y,X) P(Y)=  x P(Y|X)P(X) P(Y|X)=P(X|Y)P(Y)/  Y P(X|Y)P(Y)

Bayes rule If you draw G, what is the probability that it came from Box1 or Box2? P(B1|G)=P(G|B1)P(B1)/P(G) =(0.33*0.2)/0.6=0.11 P(B2|G)=P(G|B2)P(B2)/P(G) =(0.66*0.8)/0.6=0.89 Sum to one

Bayes rule P(A,B|C)=P(A|B,C)P(B|C) P(B|A,C)=P(A|B,C)P(B|C)/P(A|C)

Fourier transform Basis in linear algebra Basis function: dirac Basis function: sin

Fourier Transform Decomposition in sum of sin and cosine Power: first term is the DC Phase Fourier transform for Dirac Sin Gaussian (inverse relationship)

Fourier Transform Convolution and products

Fourier transform Fourier transform of a Gabor

Fourier transform Eigenspace for liner dynamical system…

Dynamical systems Stable if <0, unstable otherwise

Dynamical systems Fixed Point

Dynamical systems Stable if f’(x 0 )<0, unstable otherwise.

Dynamical systems

go into eigen space Equations decouple Stable if  <0, unstable otherwise.

Dynamical systems

go into eigen space Equations decouple

Dynamical systems go into eigen space Equations decouple Stable is f’(x 0 )<0, unstable otherwise.

Dynamical systems Fixed point Saddle point Unstable point Stable and unstable oscillations: complex eigenvalues

Nonlinear Networks Discrete case: Stable if | |<1, unstable otherwise

Nonlinear Networks Discrete case:

Nonlinear Networks Dynamics around attractor:

Nonlinear Networks Stable Fixed point: | 1 |<1, | 2 |<1

Nonlinear Networks Saddle Point: | 1 |>1, | 2 |<1

Nonlinear Networks Unstable Fixed point: | 1 |>1, | 2 |>1

Nonlinear Networks Line Attractor: 1 =1, | 2 |<1

Nonlinear Networks Oscillation: complex ’s

Nonlinear Networks: global stability Lyapunov Function: function of the state of the system which is bounded below and goes down over time. If such a function exists, the system is globally stable. Ex: Hopfield network, Cohen-Grossberg network