1. Math Primer

2. Outline Calculus: derivatives, chain rule, gradient descent, taylor expansions Bayes Rule Fourier Transform Dynamical linear systems

3. Calculus Derivatives Derivative=slope

4. Calculus Derivative: a few common functions (x n )’=nx n-1 (x -1 )’=-1/x 2 = x -2 exp(x)’=exp(x) log(x)’=1/x cos(x)’=-sin(x)

5. Calculus Derivative: Chain rule Ex: gaussian z=h(x)=exp(-x 2 )’,f(y)=exp(y), y=g(x)=-x 2 f’(y)=exp(y)= exp(-x 2 ), g’(x)=-2x z’= f’(y)g’(x) =exp(-x 2 )(-2x)

6. Calculus Finding minima: gradient descent f(x) f’(x 0 ) < 0 x0x0 x* x0+xx0+x  x > 0 f’(x*)=0  x = -  f’(x 0 )

7. Calculus Example: minimizing an error function

8. Calculus Taylor expansion

9. Calculus

10. Bayes rule Example: drawing from 2 boxes 2 boxes (B1,B2) P(B1)=0.2,P(B2)=0.8 (Prior) Balls with two colors (R,G) B1=(16R,8G), B2=(8R,16G) P(R|B1)=2/3, P(G|B1)=1/3 (Conditional) P(R|B2)=1/3, P(G|B2)=2/3 (Conditional)

11. Bayes rule Joint distributions P(G,B1)=P(B1)P(G|B1)=0.2*0.33=0.066 P(G,B2)=P(B2)P(G|B2)=0.8*0.66=0.528 P(X,Y)=P(X|Y)P(Y) P(Y,X)=P(Y|X)P(X) P(Y|X)P(X)=P(X|Y)P(Y)

12. Bayes rule P(Y|X)P(X)=P(X|Y)P(Y) P(Y|X)=P(X|Y)P(Y)/P(X) If you draw G, what is the probability that it came from box1? P(B1|G)=P(G|B1)P(B1)/P(G) How do you get this? Marginalize

13. Bayes rule Marginalization P(G)=P(G,B1)+P(G,B2)=0.066+0.528=0.6 P(G)=P(G|B1)P(B1)+P(G|B2)P(B2) P(Y)=  x P(Y,X) P(Y)=  x P(Y|X)P(X) P(Y|X)=P(X|Y)P(Y)/  Y P(X|Y)P(Y)

14. Bayes rule If you draw G, what is the probability that it came from Box1 or Box2? P(B1|G)=P(G|B1)P(B1)/P(G) =(0.33*0.2)/0.6=0.11 P(B2|G)=P(G|B2)P(B2)/P(G) =(0.66*0.8)/0.6=0.89 Sum to one

15. Bayes rule P(A,B|C)=P(A|B,C)P(B|C) P(B|A,C)=P(A|B,C)P(B|C)/P(A|C)

16. Fourier transform Basis in linear algebra Basis function: dirac Basis function: sin

17. Fourier Transform Decomposition in sum of sin and cosine Power: first term is the DC Phase Fourier transform for Dirac Sin Gaussian (inverse relationship)

18. Fourier Transform Convolution and products

19. Fourier transform Fourier transform of a Gabor

20. Fourier transform Eigenspace for liner dynamical system…

21. Dynamical systems Stable if <0, unstable otherwise

22. Dynamical systems Fixed Point

23. Dynamical systems Stable if f’(x 0 )<0, unstable otherwise.

24. Dynamical systems

26. go into eigen space Equations decouple Stable if  <0, unstable otherwise.

27. Dynamical systems

28. go into eigen space Equations decouple

29. Dynamical systems go into eigen space Equations decouple Stable is f’(x 0 )<0, unstable otherwise.

30. Dynamical systems Fixed point Saddle point Unstable point Stable and unstable oscillations: complex eigenvalues

31. Nonlinear Networks Discrete case: Stable if | |<1, unstable otherwise

32. Nonlinear Networks Discrete case:

33. Nonlinear Networks Dynamics around attractor:

34. Nonlinear Networks Stable Fixed point: | 1 |<1, | 2 |<1

35. Nonlinear Networks Saddle Point: | 1 |>1, | 2 |<1

36. Nonlinear Networks Unstable Fixed point: | 1 |>1, | 2 |>1

37. Nonlinear Networks Line Attractor: 1 =1, | 2 |<1

38. Nonlinear Networks Oscillation: complex ’s

39. Nonlinear Networks: global stability Lyapunov Function: function of the state of the system which is bounded below and goes down over time. If such a function exists, the system is globally stable. Ex: Hopfield network, Cohen-Grossberg network