1. Big Data, Computation and Statistics Michael I. Jordan February 23, 2012 1

2. What Is the Big Data Phenomenon? Big Science is generating massive datasets to be used both for classical testing of theories and for exploratory science Measurement of human activity, particularly online activity, is generating massive datasets that can be used (e.g.) for personalization and for creating markets Sensor networks are becoming pervasive

3. What Is the Big Data Problem? Computer science studies the management of resources, such as time and space and energy

4. What Is the Big Data Problem? Computer science studies the management of resources, such as time and space and energy Data has not been viewed as a resource, but as a “workload”

5. What Is the Big Data Problem? Computer science studies the management of resources, such as time and space and energy Data has not been viewed as a resource, but as a “workload” The fundamental issue is that data now needs to be viewed as a resource –the data resource combines with other resources to yield timely, cost-effective, high-quality decisions and inferences

6. What Is the Big Data Problem? Computer science studies the management of resources, such as time and space and energy Data has not been viewed as a resource, but as a “workload” The fundamental issue is that data now needs to be viewed as a resource –the data resource combines with other resources to yield timely, cost-effective, high-quality decisions and inferences Just as with time or space, it should be the case (to first order) that the more of the data resource the better

7. What Is the Big Data Problem? Computer science studies the management of resources, such as time and space and energy Data has not been viewed as a resource, but as a “workload” The fundamental issue is that data now needs to be viewed as a resource –the data resource combines with other resources to yield timely, cost-effective, high-quality decisions and inferences Just as with time or space, it should be the case (to first order) that the more of the data resource the better –is that true in our current state of knowledge?

8. No, for two main reasons: –query complexity grows faster than number of data points the more rows in a table, the more columns the more columns, the more hypotheses that can be considered indeed, the number of hypotheses grows exponentially in the number of columns so, the more data the greater the chance that random fluctuations look like signal (e.g., more false positives)

9. No, for two main reasons: –query complexity grows faster than number of data points the more rows in a table, the more columns the more columns, the more hypotheses that can be considered indeed, the number of hypotheses grows exponentially in the number of columns so, the more data the greater the chance that random fluctuations look like signal (e.g., more false positives) –the more data the less likely a sophisticated algorithm will run in an acceptable time frame and then we have to back off to cheaper algorithms that may be more error-prone or we can subsample, but this requires knowing the statistical value of each data point, which we generally don’t know a priori

10. A Formulation of the Problem Given an inferential goal and a fixed computational budget, provide a guarantee (supported by an algorithm and an analysis) that the quality of inference will increase monotonically as data accrue (without bound)

11. A Formulation of the Problem Given an inferential goal and a fixed computational budget, provide a guarantee (supported by an algorithm and an analysis) that the quality of inference will increase monotonically as data accrue (without bound) –this is far from being achieved in the current state of the literature! –a solution will blend computer science and statistics, and fundamentally change both fields

12. A Formulation of the Problem Given an inferential goal and a fixed computational budget, provide a guarantee (supported by an algorithm and an analysis) that the quality of inference will increase monotonically as data accrue (without bound) –this is far from being achieved in the current state of the literature! –a solution will blend computer science and statistics, and fundamentally change both fields I believe that it is an achievable goal, but one that will require a major, sustained effort

13. An Example The bootstrap is a generic statistical method for computing error bars (and other measures of risk) for statistical estimates

14. An Example The bootstrap is a generic statistical method for computing error bars (and other measures of risk) for statistical estimates It requires resampling the data with replacement a few hundred times, and computing the estimator independently on each resampled data set

15. An Example The bootstrap is a generic statistical method for computing error bars (and other measures of risk) for statistical estimates It requires resampling the data with replacement a few hundred times, and computing the estimator independently on each resampled data set –seemingly a wonderful connection with modern multi-core and cloud computing architectures

16. An Example The bootstrap is a generic statistical method for computing error bars (and other measures of risk) for statistical estimates It requires resampling the data with replacement a few hundred times, and computing the estimator independently on each resampled data set –seemingly a wonderful connection with modern multi-core and cloud computing architectures But if we have a terabyte of data, each resampled data set is 632 gigabytes… –and error bars computed on subsamples have the wrong scale

17. An Example The bootstrap is a generic statistical method for computing error bars (and other measures of risk) for statistical estimates It requires resampling the data with replacement a few hundred times, and computing the estimator independently on each resampled data set –seemingly a wonderful connection with modern multi-core and cloud computing architectures But if we have a terabyte of data, each resampled data set is 632 gigabytes… –and error bars computed on subsamples have the wrong scale So the classical bootstrap is dead in the water for massive data

18. An Example The bootstrap is a generic statistical method for computing error bars (and other measures of risk) for statistical estimates It requires resampling the data with replacement a few hundred times, and computing the estimator independently on each resampled data set –seemingly a wonderful connection with modern multi-core and cloud computing architectures But if we have a terabyte of data, each resampled data set is 632 gigabytes… –and error bars computed on subsamples have the wrong scale So the classical bootstrap is dead in the water for massive data –this issue hasn’t been considered before in the statistics literature

19. The Big Data Bootstrap Ariel Kleiner, Purnamrita Sarkar and Ameet Talwalkar and Michael I. Jordan University of California, Berkeley

20. Assessing the Quality of Inference Observe data X 1,..., X n Form a parameter estimate  n =  (X 1,..., X n ) Want to compute an assessment  of the quality of our estimate  n (e.g., a confidence region)

21. The Goal A procedure for quantifying estimator quality which is accurate automatic scalable

22. The Unachievable Frequentist Ideal Ideally, we would ① Observe many independent datasets of size n. ② Compute  n on each. ③ Compute  based on these multiple realizations of  n.

23. The Unachievable Frequentist Ideal Ideally, we would ① Observe many independent datasets of size n. ② Compute  n on each. ③ Compute  based on these multiple realizations of  n.

24. The Unachievable Frequentist Ideal Ideally, we would ① Observe many independent datasets of size n. ② Compute  n on each. ③ Compute  based on these multiple realizations of  n. But, we only observe one dataset of size n.

25. The Underlying Population

26. Sampling

27. Approximation

28. Pretend The Sample Is The Population

29. The Bootstrap Use the observed data to simulate multiple datasets of size n: ① Repeatedly resample n points with replacement from the original dataset of size n. ② Compute  * n on each resample. ③ Compute  based on these multiple realizations of  * n as our estimate of  for  n. (Efron, 1979)

30. The Bootstrap: Computational Issues Expected number of distinct points in a bootstrap resample is ~ 0.632n. Resources required to compute  generally scale in number of distinct data points. –this is true of many commonly used algorithms (e.g., SVM, logistic regression, linear regression, kernel methods, general M-estimators, etc.). –use weighted representation of resampled datasets to avoid physical data replication. –example: If original dataset has size 1 TB, then expect resample to have size ~ 632 GB.

31. Subsampling n (Politis, Romano & Wolf, 1999)

32. Subsampling n b

33. b

34. Compute  only on smaller subsets of the data of size b(n) < n, and analytically correct our estimates of  : ① Repeatedly subsample b(n) < n points without replacement from the original dataset of size n. ② Compute   b(n) on each subsample. ③ Compute  based on these multiple realizations of   b(n). ④ Analytically correct to produce final estimate of  for  n. Much more favorable computational profile than bootstrap.

35. Empirical Results: Bootstrap and Subsampling Multivariate linear regression with d = 100 and n = 50,000 on synthetic data. x coordinates sampled independently from StudentT(3). y = w T x + , where w in R d is a fixed weight vector and  is Gaussian noise. Estimate  n = w n in R d via least squares. Compute a marginal confidence interval for each component of w n and assess accuracy via relative mean (across components) absolute deviation from true confidence interval size. For subsampling, use b(n) = n  for various values of . Similar results obtained with Normal and Gamma data generating distributions, as well as if estimate a misspecified model.

36. Empirical Results: Bootstrap and Subsampling

37. Subsampling n b

38. b

39. Approximation

40. Pretend the Subsample is the Population

41. And bootstrap the subsample! This means resampling n times with replacement, not b times as in subsampling

42. The Bag of Little Bootstraps (BLB) ① Repeatedly subsample b(n) < n points without replacement from the original dataset of size n. ② For each subsample do: a)Repeatedly resample n points with replacement from the subsample. b)Compute  * n on each resample. c)Compute an estimate of  based on these multiple resampled realizations of  * n. ③ We now have one estimate of  per subsample. Output their average as our final estimate of  for  n.

43. The Bag of Little Bootstraps (BLB)

44. Bag of Little Bootstraps (BLB) Computational Considerations A key point: Resources required to compute  generally scale in number of distinct data points. This is true of many commonly used estimation algorithms (e.g., SVM, logistic regression, linear regression, kernel methods, general M-estimators, etc.). Use weighted representation of resampled datasets to avoid physical data replication. Example: if original dataset has size 1 TB with each data point 1 MB, and we take b(n) = n 0.6, then expect subsampled datasets to have size ~ 4 GB resampled datasets to have size ~ 4 GB (in contrast, bootstrap resamples have size ~ 632 GB)

45. Empirical Results: Bag of Little Bootstraps (BLB)

46. BLB: Theoretical Results Higher-Order Correctness Then:

47. BLB: Theoretical Results BLB is asymptotically consistent and higher- order correct (like the bootstrap), under essentially the same conditions that have been used in prior analysis of the bootstrap. Theorem (asymptotic consistency): Under standard assumptions (particularly that  is Hadamard differentiable and  is continuous), the output of BLB converges to the population value of  as n, b approach ∞.

48. BLB: Theoretical Results Higher-Order Correctness Assume:  is a studentized statistic.  (Q n (P)), the population value of  for  n, can be written as where the p k are polynomials in population moments. The empirical version of  based on resamples of size n from a single subsample of size b can also be written as where the are polynomials in the empirical moments of subsample j. b ≤ n and

49. BLB: Theoretical Results Higher-Order Correctness Also, if BLB’s outer iterations use disjoint chunks of data rather than random subsamples, then

50. Conclusions An initial foray into the evaluation of estimators for Big Data Many further directions to pursue –BLB for time series and graphs –streaming BLB –m out of n / BLB hybrids –super-scalability Technical Report available at: www.cs.berkeley.edu/~jordan/publications.html