1. Machine Learning on Massive Datasets Alexander Gray Georgia Institute of Technology College of Computing FASTlab: Fundamental Algorithmic and Statistical Tools Laboratory

2. The FASTlab Fundamental Algorithmic and Statistical Tools Laboratory www.fast-lab.org www.fast-lab.org 1.Alexander Gray: Assoc Prof, Applied Math + CS; PhD CS 2.Arkadas Ozakin: Research Scientist, Math + Physics; PhD Physics 3.Dong Ryeol Lee: PhD student, CS + Math 4.Ryan Riegel: PhD student, CS + Math 5.Sooraj Bhat: PhD student, CS 6.Wei Guan: PhD student, CS 7.Nishant Mehta: PhD student, CS 8.Parikshit Ram: PhD student, CS + Math 9.William March: PhD student, Math + CS 10.Hua Ouyang: PhD student, CS 11.Ravi Sastry: PhD student, CS 12.Long Tran: PhD student, CS 13.Ryan Curtin: PhD student, EE 14.Ailar Javadi: PhD student, EE 15.Abhimanyu Aditya: Affiliate, CS; MS CS + 5-10 MS students and undergraduates

3. Exponential growth in dataset sizes 1990 COBE 1,000 2000 Boomerang 10,000 2002 CBI 50,000 2003 WMAP 1 Million 2008 Planck 10 Million Data: CMB Maps Data: Local Redshift Surveys 1986 CfA 3,500 1996 LCRS 23,000 2003 2dF 250,000 2005 SDSS 800,000 Data: Angular Surveys 1970 Lick 1M 1990 APM 2M 2005 SDSS 200M 2010 LSST 2B Instruments [Science, Szalay & J. Gray, 2001]

4. 1993-1999: DPOSS 1999-2008: SDSS Coming: Pan-STARRS, LSST

5. Happening everywhere! Molecular biology (cancer) microarray chips Particle events (LHC) particle colliders microprocessors Simulations (Millennium) Network traffic (spam) fiber optics 300M/day 1B 1M/sec

6. 1.How did galaxies evolve? 2.What was the early universe like? 3.Does dark energy exist? 4.Is our model (GR+inflation) right? Astrophysicist Machine learning/ statistics guy R. Nichol, Inst. Cosmol. Gravitation A. Connolly, U. Pitt Physics C. Miller, NOAO R. Brunner, NCSA G. Djorgovsky, Caltech G. Kulkarni, Inst. Cosmol. Gravitation D. Wake, Inst. Cosmol. Gravitation R. Scranton, U. Pitt Physics M. Balogh, U. Waterloo Physics I. Szapudi, U. Hawaii Inst. Astronomy G. Richards, Princeton Physics A. Szalay, Johns Hopkins Physics

7. 1.How did galaxies evolve? 2.What was the early universe like? 3.Does dark energy exist? 4.Is our model (GR+inflation) right? Astrophysicist Machine learning/ statistics guy O(N n ) O(N 2 ) O(N 3 ) O(c D T(N)) R. Nichol, Inst. Cosmol. Grav. A. Connolly, U. Pitt Physics C. Miller, NOAO R. Brunner, NCSA G. Djorgovsky, Caltech G. Kulkarni, Inst. Cosmol. Grav. D. Wake, Inst. Cosmol. Grav. R. Scranton, U. Pitt Physics M. Balogh, U. Waterloo Physics I. Szapudi, U. Hawaii Inst. Astro. G. Richards, Princeton Physics A. Szalay, Johns Hopkins Physics Kernel density estimator n-point spatial statistics Nonparametric Bayes classifier Support vector machine Nearest-neighbor statistics Gaussian process regression Hierarchical clustering

8. R. Nichol, Inst. Cosmol. Grav. A. Connolly, U. Pitt Physics C. Miller, NOAO R. Brunner, NCSA G. Djorgovsky, Caltech G. Kulkarni, Inst. Cosmol. Grav. D. Wake, Inst. Cosmol. Grav. R. Scranton, U. Pitt Physics M. Balogh, U. Waterloo Physics I. Szapudi, U. Hawaii Inst. Astro. G. Richards, Princeton Physics A. Szalay, Johns Hopkins Physics Kernel density estimator n-point spatial statistics Nonparametric Bayes classifier Support vector machine Nearest-neighbor statistics Gaussian process regression Hierarchical clustering 1.How did galaxies evolve? 2.What was the early universe like? 3.Does dark energy exist? 4.Is our model (GR+inflation) right? Astrophysicist Machine learning/ statistics guy O(N n ) O(N 2 ) O(N 3 ) O(N 2 ) O(N 3 )

9. R. Nichol, Inst. Cosmol. Grav. A. Connolly, U. Pitt Physics C. Miller, NOAO R. Brunner, NCSA G. Djorgovsky, Caltech G. Kulkarni, Inst. Cosmol. Grav. D. Wake, Inst. Cosmol. Grav. R. Scranton, U. Pitt Physics M. Balogh, U. Waterloo Physics I. Szapudi, U. Hawaii Inst. Astro. G. Richards, Princeton Physics A. Szalay, Johns Hopkins Physics Kernel density estimator n-point spatial statistics Nonparametric Bayes classifier Support vector machine Nearest-neighbor statistics Gaussian process regression Hierarchical clustering 1.How did galaxies evolve? 2.What was the early universe like? 3.Does dark energy exist? 4.Is our model (GR+inflation) right? But I have 1 million points Astrophysicist Machine learning/ statistics guy O(N n ) O(N 2 ) O(N 3 ) O(N 2 ) O(N 3 )

10. State-of-the-art statistical methods… Best accuracy with fewest assumptions with orders-of-mag more efficiency. Large N (#data), D (#features), M (#models) The challenge D N M Reduce data? Use simpler model? Approximation with poor/no error bounds?  Poor results

11. What I like to think about… 1.The statistical problems and methods needed for answering scientific questions 2.The computational problems (bottlenecks) and methods involved in scaling all of them up to big datasets 3.The infrastructure-tailored software to make the application happen

12. How to do Stats/Machine Learning on Very Large Survey Datasets? 1.Choose the appropriate statistical task and method for the scientific question 2.Use the fastest algorithm and data structure for the statistical method 3.Apply method to the data!

13. How to do Stats/Machine Learning on Very Large Survey Datasets? 1.Choose the appropriate statistical task and method for the scientific question 2.Use the fastest algorithm and data structure for the statistical method 3.Apply method to the data!

14. 10 data analysis problems, and scalable tools we’d like for them 1.Querying (e.g. characterizing a region of space): –spherical range-search O(N) –orthogonal range-search O(N) –spatial join O(N 2 ) –k-nearest-neighbors O(N) –all-k-nearest-neighbors O(N 2 ) [red: of high interest in astronomy] 2.Density estimation (e.g. comparing galaxy types): –mixture of Gaussians –kernel density estimation O(N 2 ) –kernel conditional density estimation O(N 3 ) –submanifold kernel density estimation O(N 2 ) [Ozakin and Gray, NIPS 2009] –hyperkernel density estimation O(N 4 ) [Sastry and Gray 2010, to be submitted] –L 2 density estimation trees [Ram and Gray, under review]

15. 10 data analysis problems, and scalable tools we’d like for them 3. Regression (e.g. photometric redshifts): –linear regression –kernel regression O(N 2 ) –Gaussian process regression/kriging O(N 3 ) 4. Classification (e.g. quasar detection, star-galaxy separation): –decision tree –k-nearest-neighbor classifier O(N 2 ) –nonparametric Bayes classifier O(N 2 ), aka KDA –support vector machine (SVM) O(N 3 ) –non-negative SVM O(N 3 ) [Guan and Gray 2010, under review] –isometric separation map O(N 3 ) [Vasiloglou, Gray, and Anderson, MLSP 2008 (Nominated for Best Paper)]

16. 10 data analysis problems, and scalable tools we’d like for them 5.Dimension reduction (e.g. galaxy or spectra characterization): –principal component analysis –non-negative matrix factorization –kernel PCA (including LLE, IsoMap, et al.) O(N 3 ) –maximum variance unfolding O(N 3 ) –rank-based manifolds O(N 3 ) [Ouyang and Gray, ICML 2008] –isometric non-negative matrix factorization O(N 3 ) [Vasiloglou, Gray, and Anderson, SDM 2009] –density-preserving maps O(N 3 ) [Ozakin and Gray 2010, to be submitted] 6.Outlier detection (e.g. new object types, data cleaning): –by density estimation, by dimension reduction –by robust L p estimation [Ram, Riegel and Gray 2010, to be submitted]

17. 10 data analysis problems, and scalable tools we’d like for them 7. Clustering (e.g. automatic Hubble sequence) –by dimension reduction, by density estimation –k-means –mean-shift segmentation O(N 2 ) –hierarchical clustering (“friends-of-friends”) O(N 3 ) 8. Time series analysis (e.g. asteroid tracking, variable objects): –Kalman filter –hidden Markov model –trajectory tracking O(N n ) –functional independent component analysis [Mehta and Gray, SDM 2009] –Markov matrix factorization [Tran, Wong, and Gray 2010, under review] –generative mean map kernel [Mehta and Gray 2010, to be submitted]

18. 10 data analysis problems, and scalable tools we’d like for them 9. Feature selection and causality (e.g. which features predict star/galaxy) –LASSO regression –L 1 SVM –Gaussian graphical model inference and structure search –discrete graphical model inference and structure search –fractional-norm SVM [Guan and Gray 2010, under review] –mixed-integer SVM [Guan and Gray 2010, to be submitted] 10. 2-sample testing and matching (e.g. cosmological validation, multiple surveys): –minimum spanning tree O(N 3 ) –n-point correlation O(N n ) –bipartite matching/Gaussian graphical model inference O(N 3 ) [Riegel and Gray, in preparation]

19. Core methods of statistics / machine learning / mining Querying: spherical range-search O(N), orthogonal range-search O(N), spatial join O(N 2 ), nearest-neighbor O(N), all-nearest-neighbors O(N 2 ) Density estimation: mixture of Gaussians, kernel density estimation O(N 2 ), kernel conditional density estimation O(N 3 ) Regression: linear regression, kernel regression O(N 2 ), Gaussian process regression O(N 3 ) Classification: decision tree, nearest-neighbor classifier O(N 2 ), nonparametric Bayes classifier O(N 2 ), support vector machine O(N 3 ) Dimension reduction: principal component analysis, non-negative matrix factorization, kernel PCA O(N 3 ), maximum variance unfolding O(N 3 ) Outlier detection: by density estimation or dimension reduction Clustering: by density estimation or dimension reduction, k-means, mean- shift segmentation O(N 2 ), hierarchical clustering O(N 3 ) Time series analysis: Kalman filter, hidden Markov model, trajectory tracking O(N n ) Feature selection and causality: LASSO, L 1 SVM, Gaussian graphical models, discrete graphical models 2-sample testing and matching: bipartite matching O(N 3 ), n-point correlation O(N n )

20. Now pretty fast (2011)… made fast by us: fastest, fastest in some settings Querying: spherical range-search O(logN)*, orthogonal range-search O(logN)*, spatial join O(N)*, nearest-neighbor O(logN), all-nearest- neighbors O(N) Density estimation: mixture of Gaussians, kernel density estimation O(N), kernel conditional density estimation O(N log3 )* Regression: linear regression, kernel regression O(N), Gaussian process regression O(N)* Classification: decision tree, nearest-neighbor classifier O(N), nonparametric Bayes classifier O(N)*, support vector machine O(N)/O(N 2 ) Dimension reduction: principal component analysis, non-negative matrix factorization, kernel PCA O(N)*, maximum variance unfolding O(N)* Outlier detection: by density estimation or dimension reduction Clustering: by density estimation or dimension reduction, k-means, mean- shift segmentation O(N), hierarchical (FoF) clustering O(NlogN) Time series analysis: Kalman filter, hidden Markov model, trajectory tracking O(N logn )* Feature selection and causality: LASSO, L 1 SVM, Gaussian graphical models, discrete graphical models 2-sample testing and matching: bipartite matching O(N)**, n-point correlation O(N logn )*

21. How to do Stats/Machine Learning on Very Large Survey Datasets? 1.Choose the appropriate statistical task and method for the scientific question 2.Use the fastest algorithm and data structure for the statistical method 3.Apply method to the data!

22. Core computational problems What are the basic mathematical operations, or bottleneck subroutines, can we focus on developing fast algorithms for?

23. Core computational problems Generalized N-body problems Graphical model inference Linear algebra Optimization

24. 4 main computational bottlenecks: N-body, graphical models, linear algebra, optimization Querying: spherical range-search O(N), orthogonal range-search O(N), spatial join O(N 2 ), nearest-neighbor O(N), all-nearest-neighbors O(N 2 ) Density estimation: mixture of Gaussians, kernel density estimation O(N 2 ), kernel conditional density estimation O(N 3 ) Regression: linear regression, kernel regression O(N 2 ), Gaussian process regression O(N 3 ) Classification: decision tree, nearest-neighbor classifier O(N 2 ), nonparametric Bayes classifier O(N 2 ), support vector machine O(N 3 ) Dimension reduction: principal component analysis, non-negative matrix factorization, kernel PCA O(N 3 ), maximum variance unfolding O(N 3 ) Outlier detection: by density estimation or dimension reduction Clustering: by density estimation or dimension reduction, k-means, mean- shift segmentation O(N 2 ), hierarchical clustering O(N 3 ) Time series analysis: Kalman filter, hidden Markov model, trajectory tracking O(N n ) Feature selection and causality: LASSO, L 1 SVM, Gaussian graphical models, discrete graphical models 2-sample testing and matching: bipartite matching O(N 3 ), n-point correlation O(N n )

25. Mathematical/practical challenges Return answers with error guarantees Rigorous theoretical runtimes Ensure small constants (small actual runtimes, not just good asymptotic runtimes)

26. Generalized N-body Problems I How it appears: nearest-neighbor, sph range-search, ortho range-search Common methods: locality sensitive hashing, kd-trees, metric trees, disk-based trees Mathematical challenges: high dimensions, provable runtime, distribution- dependent analysis, parallel indexing Mathematical topics: computational geometry, randomized algorithms

27. kd-trees: most widely-used space- partitioning tree [Bentley 1975], [Friedman, Bentley & Finkel 1977],[Moore & Lee 1995] First example: spherical range search. How can we compute this efficiently?

28. A kd-tree: level 1

29. A kd-tree: level 2

30. A kd-tree: level 3

31. A kd-tree: level 4

32. A kd-tree: level 5

33. A kd-tree: level 6

34. Range-count recursive algorithm

38. Pruned! (inclusion) Range-count recursive algorithm

46. Pruned! (exclusion) Range-count recursive algorithm

49. fastest practical algorithm [Bentley 1975] our algorithms can use any tree Range-count recursive algorithm

50. Generalized N-body Problems I Interesting approach: Cover-trees [Beygelzimer et al 2004] –Provable runtime –Consistently good performance, even in higher dimensions Interesting approach: Learning trees [Cayton et al 2007] –Learning data-optimal data structures –Improves performance over kd-trees Interesting approach: MapReduce [Dean and Ghemawat 2004] –Brute-force –But makes HPC automatic for a certain problem form Interesting approach: approximation using spill-trees [Liu, Moore, Gray, and Yang, NIPS 2004] –Faster/more accurate than LSH Interesting approach: approximation in rank [Ram, Ouyang and Gray, NIPS 2009] –Approximate NN in terms of distance conflicts with known theoretical results –Is approximation in rank feasible? –Faster/more accurate than LSH

51. Generalized N-body Problems II How it appears: kernel density estimation, mixture of Gaussians, kernel regression, Gaussian process regression, nearest-neighbor classifier, nonparametric Bayes classifier, support vector machine, kernel PCA, hierarchical clustering, trajectory tracking, n-point correlation Common methods: FFT, Fast Gauss Transform, Well- Separated Pair Decomposition Mathematical challenges: high dimensions, query- dependent relative error guarantee, parallel, beyond pairwise potentials Mathematical topics: approximation theory, computational physics, computational geometry

52. Generalized N-body Problems II Interesting approach: Generalized Fast Multipole Method, aka multi-tree methods [Gray and Moore, NIPS 2001; Ram, Lee, March, and Gray, NIPS 2009; Riegel, Boyer and Gray 2010, to be submitted] –Fastest practical algorithms for the problems to which it has been applied –Hard query-dependent relative error bounds –Automatic parallelization (THOR: Tree-based Higher-Order Reduce)

53. Generalized N-body Problems II Interesting approach (Gaussian kernel): dual-tree with multipole/Hermite expansions [Lee, Gray and Moore, NIPS 2005; Lee and Gray, UAI 2006] –Ultra-accurate fast kernel summations Interesting approach (near-arbitrary kernel): finite- difference [Gray and Moore, AISTATS 2003; Gray and Moore, SDM 2003 (SDM Best Paper Award, ASA Best Paper Award)], automatic derivation of multipole methods [Lee and Gray 2010, to be submitted]

54. Generalized N-body Problems II Interesting approach (summative forms): multi-scale Monte Carlo [Holmes, Gray, Isbell, NIPS 2006; Holmes, Gray, Isbell, UAI 2007] –Very fast bandwidth learning Interesting approach (high-dimensional evaluation): Monte Carlo multipole methods [Lee and Gray, NIPS 2008] –Uses SVD tree

55. Generalized N-body Problems II Interesting approach (N-body decision problems: NN clsf, KDA): dual-tree bounding with hybrid tree expansion [Liu, Moore, and Gray, NIPS 2004; Gray and Riegel, CompStat 2004; Riegel and Gray, SDM 2007] –An exact classification algorithm Interesting approach (for EMST): dual-tree Boruvka algorithm [March and Gray KDD 2010] –Note this is a cubic problem

56. Generalized N-body Problems II Interesting approach (for n-point): n- tree algorithms [Gray and Moore, NIPS 2001; Moore et al., Mining the Sky 2001] –First efficient exact algorithm for n-point correlations Interesting approach (for n-point): Monte Carlo n-tree [March and Gray 2010, to be submitted] –Orders of magnitude faster

57. Generalized N-body Problems II Interesting approach (for multi-body potentials in physics): higher-order multipole methods [Lee and Gray 2010, to be submitted; Waters, Ozakin, Gray, et al. 2010, to be submitted] –First fast algorithm for higher-order potentials Interesting approach (for quantum-level simulation): 4-body treatment of Hartree- Fock [March and Gray 2010, in preparation]

58. 2-point correlation r Characterization of an entire distribution? “How many pairs have distance < r ?” 2-point correlation function

59. The n-point correlation functions Spatial inferences: filaments, clusters, voids, homogeneity, isotropy, 2-sample testing, … Foundation for theory of point processes [Daley,Vere-Jones 1972], unifies spatial statistics [Ripley 1976] Used heavily in biostatistics, cosmology, particle physics, statistical physics 2pcf definition: 3pcf definition:

60. 3-point correlation “How many triples have pairwise distances < r ?” r3r3 r1r1 r2r2 Standard model: n>0 terms should be zero!

61. How can we count n-tuples efficiently? “How many triples have pairwise distances < r ?”

62. Use n trees! [Gray & Moore, NIPS 2000]

63. “How many valid triangles a-b-c (where ) could there be? A B C r count{A,B,C} = ?

64. “How many valid triangles a-b-c (where ) could there be? count{A,B,C} = count{A,B, C.left } + count{A,B,C.right} A B C r

65. “How many valid triangles a-b-c (where ) could there be? A B C r count{A,B,C} = count{A,B,C.left} + count{A,B, C.right }

66. “How many valid triangles a-b-c (where ) could there be? A B C r count{A,B,C} = ?

67. “How many valid triangles a-b-c (where ) could there be? A B C r Exclusion count{A,B,C} = 0!

68. “How many valid triangles a-b-c (where ) could there be? AB C count{A,B,C} = ? r

69. “How many valid triangles a-b-c (where ) could there be? AB C Inclusion count{A,B,C} = |A| x |B| x |C| r Inclusion

70. 3-point runtime (biggest previous: 20K) VIRGO simulation data, N = 75,000,000 naïve: 5x10 9 sec. (~150 years) multi-tree: 55 sec. (exact) n=2: O(N) n=3: O(N log3 ) n=4: O(N 2 )

71. Graphical model inference How it appears: hidden Markov models, bipartite matching, Gaussian and discrete graphical models Common methods: belief propagation, expectation propagation Mathematical challenges: large cliques, upper and lower bounds, graphs with loops, parallel Mathematical topics: variational methods, statistical physics, turbo codes

72. Graphical model inference Interesting method (for discrete models): Survey propagation [Mezard et al 2002] –Good results for combinatorial optimization –Based on statistical physics ideas Interesting method (for discrete models): Expectation propagation [Minka 2001] –Variational method based on moment-matching idea Interesting method (for Gaussian models): L p structure search, solve linear system for inference

73. Linear algebra How it appears: linear regression, Gaussian process regression, PCA, kernel PCA, Kalman filter Common methods: QR, Krylov, … Mathematical challenges: numerical stability, sparsity preservation, … Mathematical topics: linear algebra, randomized algorithms, Monte Carlo

74. Linear algebra Interesting method (for probably-approximate k-rank SVD): Monte Carlo k-rank SVD [Frieze, Drineas, et al. 1998-2008] –Sample either columns or rows, from squared length distribution –For rank-k matrix approx; must know k Interesting method (for probably-approximate full SVD): QUIC-SVD [Holmes, Gray, Isbell, NIPS 2008] –Sample using cosine trees and stratification –Builds tree as needed –Full SVD: automatically sets rank based on desired error Interesting method (for kernel PCA and Gaussian process regression): matrix-free multigrid [Lee and Gray 2010, in preparation]

75. QUIC-SVD speedup 38 days  1.4 hrs, 10% rel. error 40 days  2 min, 10% rel. error

76. Optimization How it appears: support vector machine, maximum variance unfolding, robust L 2 estimation Common methods: interior point, Newton’s method Mathematical challenges: ML-specific objective functions, large number of variables / constraints, global optimization, parallel Mathematical topics: optimization theory, linear algebra, convex analysis

77. Optimization Interesting method: Sequential minimization optimization (SMO) [Platt 1999] –Much more efficient than interior-point, for SVM QPs Interesting method: Stochastic quasi-Newton [Schraudolf 2007] –Does not require scan of entire data Interesting method: Sub-gradient methods [Vishwanathan and Smola 2006] –Handles kinks in regularized risk functionals Interesting method: Stochastic Frank-Wolfe [Ouyang and Gray, SDM 2010 (ASA Best Paper Award)] –Fastest method for nonlinear SVMs so far

78. Optimization Interesting method: L-BFGS for SDP-based manifold learning [Vasiloglou, Anderson, and Gray, MLSP 2008] Interesting method: DC programming for fractional- norm SVMs [Guan and Gray 2010, under review] Interesting method: Perspective cuts and big-M for mixed-integer SVMs [Guan and Gray 2010, to be submitted]

79. Ex: support vector machine Data: IJCNN1 [DP01a] 2 classes 49,990 training points 91,701 testing points 22 features SMO: 12,831 SV’s, 84,360 iterations, 98.3% accuracy, 765 sec SFW: 4,145 SV’s, 4,500 iterations, 98.1% accuracy, 21 sec SMO: O(N 2 -N 3 ) SFW: O(N/e + 1/e 2 )

80. How to do Stats/Machine Learning on Very Large Survey Datasets? 1.Choose the appropriate statistical task and method for the scientific question 2.Use the fastest algorithm and data structure for the statistical method 3.Apply method to the data!

81. Some science enabled –Dark energy evidence (very large N) Science 2003, Top Scientific Breakthrough of the year New methods: very fast spatial correlations –Cancer detection by mass spectrometry metabolomics (very large D) Cancer Epidemiology, Biomarkers, & Prevention 2010 Most accurate to date (99%), including early stage New methods: functional support vector machines, L q feature selection, nonlinear dimension reduction

82. Some other projects –Medical health record search, clustering New method: Very fast exact/approx nearest-neighbor New method: Very fast hierarchical clustering –Classification of uncertain, temporal objects New method: Very fast nonparametric classification New method: Deconvolution of measurement errors New method: Kernel between HMMs; can combine iid, spatial, and temporal features New method: Minimize human labeling: active learning –Image matching and segmentation New method: Very fast matching between images New method: Affine-invariant NMF segmentation

83. Software MLPACK –First scalable comprehensive C++ ML library, in-RAM –Parallelism coming in 2011 MLPACK-db –Fast data analytics in relational databases (SQL Server; on-disk) Skytree Server (commercial) –Industry-targeted version of the above two

84. The end You can now use many of the state-of-the- art data analysis methods on very large datasets Talk to me… I have a lot of problems (ahem) but always want more! Alexander Gray agray@cc.gatech.eduagray@cc.gatech.edu www.fast-lab.org