Yi Wu (CMU) Joint work with Parikshit Gopalan (MSR SVC) Ryan O’Donnell (CMU) David Zuckerman (UT Austin) Pseudorandom Generators for Halfspaces TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AAAA

Outline Introduction Pseudorandom Generators Halfspaces Pseudorandom Generators for Halfspaces Our Result Proof Conclusion 2

Deterministic Algorithm Program InputOutput The algorithm deterministically outputs the correct result. 3

Randomized Algorithm Program Input Output Random Bits. The algorithm outputs the correct result with high probability. 4

Randomized Algorithms Primality testing ST-connectivity Order statistics Searching Polynomial and matrix identity verification Interactive proof systems Faster algorithms for linear programming Rounding linear program solutions to integer Minimum spanning trees shortest paths minimum cuts Counting and enumeration Matrix permanent Counting combinatorial structures Primality testing ST-connectivity Order statistics Searching Polynomial and matrix identity verification Interactive proof systems Faster algorithms for linear programming Rounding linear program solutions to integer Minimum spanning trees shortest paths minimum cuts Counting and enumeration Matrix permanent Counting combinatorial structures Primality testing ST-connectivity Order statistics Searching Polynomial and matrix identity verification Interactive proof systems Faster algorithms for linear programming Rounding linear program solutions to integer Minimum spanning trees shortest paths minimum cuts Counting and enumeration Matrix permanent Counting combinatorial structures Primality testing ST-connectivity Order statistics Searching Polynomial and matrix identity verification Interactive proof systems Faster algorithms for linear programming Rounding linear program solutions to integer Minimum spanning trees shortest paths minimum cuts Counting and enumeration Matrix permanent Counting combinatorial structures 5

Is Randomness Necessary? Open Problem: Can we simulate every randomized polynomial time algorithm by a deterministic polynomial time algorithm (the “BPP P” cojecture)? Derandomization of randomized algorithms. Primality testing [AKS] ST-connectivity [Reingold] Quadratic residues [?] 6

How to generate randomness? Question: How togenerate randomness for every randomized algorithm? Simpler Question: How to generate “pseudorandomness” for some class of programs? 7

Pseudorandom Generator (PRG) PRG n random bit k<<n random bit Yes /No Both program Answer Yes/No with almost the same probability n “pseudorandom” bit Program Input Program 8

Why study PRGs? Algorithmic Applications When k = log (n), we can derandomize the algorithm in polynomial time. Streaming Algorithm. Complexity Theoretic Implications Lower Bound of Circuit Class. Learning Theory. 9

PRG for Classes of Program Space Bounded Program [Nis92] Constant-depth circuits [Nis91, Baz07, Bra09] Halfspaces [DGJSV09, MZ09] 10

Outline Introduction Pseudorandom Generators Halfspaces Pseudorandom Generators for Halfspaces Our Result Proof Conclusion 11

Halfspaces Halfspaces: Boolean functions h:R n → {-1,1} of the form h(x) = sgn(w 1 x 1 +…+w n x n - θ ) where w 1,…, w n, θ  R Well-studied in complexity theory Widely used in Machine Learning: Perceptron, Winnow, boosting, Support Vector Machines, Lasso, Liner Regression. 12

Product Distribution For halfspace h(x), x is sampled from some product distribution; i.e., each x i is independently sampled from distribution D i. For example, each D i can be 1. Uniform distribution on {-1,1} 2. Uniform distribution on [-1,1] 3. Gaussian Distribution 13

Index Introduction Pseudorandom Generators Halfspaces Pseudorandom Generators for Halfspaces Main Result Proof Conclusion 14

PRG for halfspaces h(x) = sign(w 1 x 1 +…+w n x n - θ ) PRG x 1, x 2 …x n from some product distribution k<<n random bit Yes/No Both program Answer Yes/No with almost the same probability Pseudorandom Variable x 1, x 2 …x n 15

Geometric Interpretation, PRG for uniform distribution over [-1,1] 2 16

Geometric Interpretation, PRG for uniform distribution over [-1,1] 2 Total Number of points = poly(dim) Number of points in the halfspace is proportional to area. 17

Application to Machine Learning How many testing points is it enough to estimate the accuracy of the N dimensional linear classifier? Good PRG implies we only need deterministically check the accuracy on a set of poly(N) points! 18

Other Theoretical Applications Discrepancy Set for Convex Polytopes Circuit Lower bound on functions of halfspaces Counting the Solution of Knapsacks 19

Outline Introduction Pseudorandom Generator Halfspace Pseudorandom Generators for Halfspaces Our Results Proof Conclusion 20

Previous Result [DiGoJaSeVi,MeZu] PRG For Halfspace over uniform distribution on boolean cube ({-1,1} n ) with seed length O(log n). 21

Our Results: Arbitrary Product Distributions PRG for halfspaces under arbitrary product distribution over R n with the same seed length. Only requirement: E[x i 4 ] is a constant. 1. Gaussian Distribution 2. Uniform distribution on the solid cube. 3. Uniform distribution on the hypercube. 4. Biased distribution on the hypercube. 5. Almost any “natural distribution” 22

Our Results Functions of k-Halfspaces PRG for the intersections of k-halfspaces with seed length k log (n). PRG for arbitrary functions of k-halfspaces with seed length k 2 log (n). 23

Outline Introduction Pseudorandom Generator Halfspace Pseudorandom Generators for Halfspaces Our Result Proof Conclusion 24

Key Observation: Dichotomy of Halfspaces Under product distributions, every halfspace is close to one of the following: “Dictator” (halfspaces depending on very few variables, e.g. f(x) = sgn(x 1 )) “majority”(no variables has too much weight, e.g. f(x) = sgn(x 1 +x 2 +x 3 +…+x n ). 25

Dichotomy of weight distribution Weights are stable after certain index. Weights decreasing fast (Geometrically) 26

Weights Decrease fast (Geometrically) Intuition: for sign(2 n x n-1 x n-2 x 3 +…x n ) If each x i is from {-1,1}, it is just sign(x 1 ). 27

Weights are stable Intuition: for sign(100 x 1 + x 2 + x 3 +…x n ) Then by for every fixing of x 1, it is a majority on the rest of the variables. 28

Our PRG for Halfspace (Rough) 1. Randomly hashing all the coordinate into groups. 2. Use 4-wise independent distribution within each group. If it is “Dictator-like”: All the important variables are in different groups. If it is “Majority-like” (x 1 + x x n ) is close to Gaussian. 4-wise independent Distribution (somehow) can handle Gaussian. 29

Outline Introduction Pseudorandom Generator Halfspace Pseudorandom Generators for Halfspaces Our Result Proof Conclusion 30

Conclusion We construct PRG for halfspaces under arbitrary product distribution and functions of k halfspaces with small seed length. Future Work Building PRG for larger classes of program; e.g., Polynomial Threshold function (SVM with polynomial kernel). 31

32