1. Random Matrix Theory and Numerical Linear Algebra: A story of communication Alan Edelman Mathematics Computer Science & AI Labs ILAS Meeting June 3, 2013

2. Page 2

3. Page 3

4. Page 4

5. An Intriguing Thesis Page 5 The results/methodologies from NUMERICAL LINEAR ALGEBRA would be valuable even if computers had not been built. … but I’m glad we have computers …& I’m especially grateful for the Julia computing system

6. Eigenvalues of GOE (β=1 means reals) Naïve Way in four languages: A=randn(n); S=(A+A’)/sqrt(2*n);eig(S) A=randn(n,n);S=(A+A’)/sqrt(2*n);eigvals(S) A=matrix(rnorm(n*n),ncol=n);S=(a+t(a))/sqrt(2*n); eigen(S,symmetric=T,only.values=T)$values; A=RandomArray[NormalDistribution[],{n,n}]; S=(A+Transpose[A])/Sqrt[2*n];Eigenvalues[s] Page 6

7. Eigenvalues of GOE (β=1 means reals) Naïve Way in four languages: A=randn(n); S=(A+A’)/sqrt(2*n);eig(S) A=randn(n,n);S=(A+A’)/sqrt(2*n);eigvals(S) A=matrix(rnorm(n*n),ncol=n);S=(a+t(a))/sqrt(2*n); eigen(S,symmetric=T,only.values=T)$values; A=RandomArray[NormalDistribution[],{n,n}]; S=(A+Transpose[A])/Sqrt[n];Eigenvalues[s] Page 7 If for you: simulation is just intuition, a check, a figure, or a student project, then software like this is written quickly and victory is declared.

8. Eigenvalues of GOE (β=1 means reals) Naïve Way in four languages: A=randn(n); S=(A+A’)/sqrt(2*n);eig(S) A=randn(n,n);S=(A+A’)/sqrt(2*n);eigvals(S) A=matrix(rnorm(n*n),ncol=n);S=(a+t(a))/sqrt(2*n); eigen(S,symmetric=T,only.values=T)$values; A=RandomArray[NormalDistribution[],{n,n}]; S=(A+Transpose[A])/Sqrt[n];Eigenvalues[s] Page 8 For me: Simulation is a chance to research efficiency, Numerical Linear Algebra Style, and this research cycles back to the mathematics! Ψ=Ψ=

9. Tridiagonal Model More Efficient (β=1: Same eigenvalue distribution!) (Silverstein, Trotter, general β Dumitriu&E. etc) g i ~N(0,2) Julia: eig(SymTridiagonal(d,e)) Storage: O(n) (vs O(n 2 )) Time: O(n 2 ) (vs O(n 3 )) Page 9 n

10. Histogram without Histogramming: Sturm Sequences Count #eigs < s: Count sign changes in Det( (A-s*I)[1:k,1:k] ) Count #eigs in [x,x+h]: Take difference in number of sign changes at x+h and x Speed comparison vs. naïve way Mentioned in Dumitriu and E 2006, Used theoretically in Albrecht, Chan, and E 2008 Page 10

11. Page 11 A good computational trick is a good theoretical trick! Finite Semi-Circle Laws for Any Beta! Finite Tracy-Widom Laws for Any Beta!

12. Efficient Tracy Widom Simulation Naïve Way: A=randn(n); S=(A+A’)/sqrt(2*n);max(eig(S)) Better Way: Only create the 10n 1/3 initial segment of the diagonal and off-diagonal as the “Airy” function tells us that the max eig hardly depends on the rest Lead to the notion of the stochastic operator limit Page 12

13. Google Translate Page 13

14. Google Translate (in my dreams) Lost In Translation: eigs = svd’s squared Page 14

15. The Jacobi Random Matrix Ensemble (Constantine 1963) Suppose A and B are randn(m1,n) and randn(m2,n) –(iid standard normals) The eigenvalues of or in symmetric form has joint density Page 15

16. The Jacobi Ensemble Geometrically Take random n-hyperplane in R m (m>n) (uniformly) Take reference hyperplane (any dimension) Orthogonal projection of the unit ball in the random hyperplane onto the reference hyperplane is an ellipsoid Semi-axes lengths are Jacobi cosines: Page 16

17. The GSVD Usual Presentation Underlying Geometry Take hyperplane spanned by [ ] X=span( 1 st m 1 columns of I m ) Y=span(last m 2 columns of I m ) ABAB A: m 1 x n B: m 2 x n [ ]: m x n ABAB Page 17

18. GSVD Mathematics (Cosine,Sine) Thus any hyperplane is the span of with U’U=V’V=I and C 2 +S 2 =I (diagonal) One can directly verify by taking Jacobians projecting out the W’dW directions that we do not care about and further understand C J = 18/77 Page 18

19. Circular Ensembles A random complex unitary matrix has eigenvalues distributed on the unit circle with density It is desriable to construct random matrices that we can compute with for general beta: (Killip and Nenciu 2004 product of tridiagonal solution) Page 19

20. Numerical linear algebra Ammar, Gragg, Reichel (1991) Hessenberg Unitary Matrices G j = Recalled in Forrester and Rains (2005) Converts Killip and Nenciu to AmmarGraggReichel format Classical orthogonal polynomials on unit circle! Story relates to CMV matrices Verblunsky Coefficients = Schur Parameters Page 20

21. Implementation Needed facts 1)AmmarGraggReichel format 2)Generating the variables requires some thinking |  j | ~ sqrt(1-rand()^ ((2/β)/j))  j = |  j |*exp(2*pi*i*rand()) Remark: Authors would rightly feel all the mathematical information is in their papers. Still this presenter had some considerable trouble gathering enough of the pieces to produce the ultimate arbiter of understanding the result: a working code! Plea: Random Matrix Theory is so valuable to so many, that a pseudocode or code available is a great technology transfer mechanism. Page 21

22. Julia Code: (similar to MATLAB etc.) really useful! Page 22

23. The method of Ghosts and Shadows for Beta Ensembles Page 23

24. So far: I tried to hint about β Introduction to Ghosts G 1 is a standard normal N(0,1) G 2 is a complex normal (G 1 +iG 1 ) G 4 is a quaternion normal (G 1 +iG 1 +jG 1 +kG 1 ) G β (β>0) seems to often work just fine  “Ghost Gaussian” Page 24

25. Chi-squared Defn: χ β is the sum of β iid squares of standard normals if β=1,2,… Generalizes for non-integer β as the “gamma” function interpolates factorial χ β is the sqrt of the sum of squares (which generalizes) (wikipedia chi-distriubtion) |G 1 | is χ 1, |G 2 | is χ 2, |G 4 | is χ 4 So why not |G β | is χ β ? I call χ β the shadow of G β 2 Page 25

26. Page 26

27. Working with Ghosts Real quantity Page 27

28. Singular Values of a Ghost Gaussian by a real diagonal Page 28 (see related work by Forrester 2011) (Dubbs, E, Koev, Venkataramana 2013)

29. The Algorithm Page 29

30. Removing U and V Page 30

31. Algorithm cont. Page 31

32. Completion of Recursion Page 32

33. Monte Carlo Trials Parallel Histograms No tears! Page 33

34. Julia: Parallel Histogram 3 rd eigenvalue, pylab plot, 8 seconds! 75 processors Page 34

35. Linear Algebra too limited in Lets me put together what I need: e.g.:Tridiagonal Eigensolver Fast rank one update Arrow matrix eigensolver can surgically use LAPACK without tears Page 35

36. Weekend Julia Run Histogram of smallest singular value of randn(200) –spiked by doubling the first column –Julia: A=randn(200,200); A(:,1)*=2; –Reduced mathematically to bidiagonal form –Used julia’s bidiagonal svd –Ran 2.25 billion trials in 20 hours on 75 processors –Naïve serial algorithm would take 16 years Page 36

37. Weekend Julia Run Histogram of smallest singular value of randn(200) –spiked by doubling the first column –Julia: A=randn(200,200); A(:,1)*=2; –Reduced mathematically to bidiagonal form –Used julia’s bidiagonal svd –Ran 2.25 billion trials in 20 hours on 75 processors –Naïve serial algorithm would take 16 years I knew the limiting histogram from my thesis, universality, etc. With this kind of power, I could obtain the first order correction for finite n! Semilogy plot of abs correction and a prediction: This is like having an electron microscope to see small details that would be invisible with conventional tools! Page 37

38. Conclusion If you already held Numerical Linear Algebra with high esteem, thanks to random matrix theory, now there are even more reasons! Try julia. (Google: julia) Page 38

39. Page 39

40. Wishart Matrices (arbitrary covariance) G=mxn matrix of Gaussians Σ=mxn semidefinite matrix G’G Σ is similar to A=Σ ½ G’GΣ -½ For β=1,2,4, the joint eigenvalue density of A has a formula: Known for β=2 in some circles as Harish-Chandra-Itzykson-Zuber Page 40

41. Eigenvalue density of G’G Σ ( similar to A=Σ ½ G’GΣ -½ ) Present an algorithm for sampling from this density Show how the method of Ghosts and Shadows can be used to derive this algorithm Further evidence that β=1,2,4 need not be special Page 41

42. Scary Ideas in Mathematics Zero Negative Radical Irrational Imaginary Ghosts: Something like a sometimes commutative algebra of random variables that generalizes random Reals, Complexes, and Quaternions and inspires theoretical results and numerical computation Page 42

43. Did you say “commutative”?? Quaternions don’t commute. Yes but random quaternions do! If x and y are G 4 then x*y and y*x are identically distributed! Page 43

44. RMT Densities Hermite: c ∏|λ i - λ j | β e -∑λ i 2 /2 (Gaussian Ensemble) Laguerre: c ∏|λ i -λ j | β ∏λ i m e -∑λ i (Wishart Matrices) Jacobi: c ∏|λ i -λ j | β ∏λ i m 1 ∏(1-λ i ) m 2 (Manova Matrices) Fourier: c ∏|λ i -λ j | β (on the complex unit circle) (Circular Ensembles) (orthogonalized by Jack Polynomials) Page 44

45. Wishart Matrices (arbitrary covariance) G=mxn matrix of Gaussians Σ=mxn semidefinite matrix G’G Σ is similar to A=Σ ½ G’GΣ -½ For β=1,2,4, the joint eigenvalue density of A has a formula: Page 45

46. Joint Eigenvalue density of G’G Σ The “ 0 F 0 ” function is a hypergeometric function of two matrix arguments that depends only on the eigenvalues of the matrices. Formulas and software exist. Page 46

47. Generalization of Laguerre Laguerre: Versus Wishart: Page 47

48. General β? The joint density : is a probability density for all β>0. Goals: Algorithm for sampling from this density Get a feel for the density’s “ghost” meaning Page 48

49. Main Result An algorithm derived from ghosts that samples eigenvalues A MATLAB implementation that is consistent with other beta- ized formulas –Largest Eigenvalue –Smallest Eigenvalue Page 49

50. More practice with Ghosts Page 50

51. Bidiagonalizing Σ=I Z’Z has the Σ=I density giving a special case of Page 51

52. Numerical Experiments – Largest Eigenvalue Analytic Formula for largest eigenvalue dist E and Koev: software to compute Page 52

53. 53

54. Page 54

55. 55

56. Smallest Eigenvalue as Well The cdf of the smallest eigenvalue, Page 56

57. Cdf’s of smallest eigenvalue 57

58. Goals Continuum of Haar Measures generalizing orthogonal, unitary, symplectic Place finite random matrix theory “β”into same framework as infinite random matrix theory: specifically β as a knob to turn down the randomness, e.g. Airy Kernel –d 2 /dx 2 +x+(2/β ½ )dW  White Noise Page 58

59. Formally Let S n =2π/Γ(n/2)=“surface area of sphere” Defined at any n= β>0. A β-ghost x is formally defined by a function f x (r) such that ∫ ∞ f x (r) r β-1 S β-1 dr=1. Note: For β integer, the x can be realized as a random spherically symmetric variable in β dimensions Example: A β-normal ghost is defined by f(r)=(2π) -β/2 e -r 2 /2 Example: Zero is defined with constant*δ(r). Can we do algebra? Can we do linear algebra? Can we add? Can we multiply? r=0 Page 59

60. Understanding ∏|λ i -λ j | β Define volume element (dx)^ by (r dx)^=r β (dx)^ (β-dim volume, like fractals, but don’t really see any fractal theory here) Jacobians: A=QΛQ’ (Sym Eigendecomposition) Q’dAQ=dΛ+(Q’dQ)Λ- Λ(Q’dQ) (dA)^=(Q’dAQ)^= diagonal ^ strictly-upper diagonal = ∏dλ i =(dΛ)^ off-diag = ∏ ( (Q’dQ) ij (λ i -λ j ) ) ^=(Q’dQ)^ ∏|λ i -λ j | β Page 60