Eric Allender Rutgers University Curiouser and Curiouser: The Link between Incompressibility and Complexity CiE Special Session, June 19, 2012

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity < 2 >< 2 > Today’s Goal:  To present new developments in a line of research dating back to 2002, presenting some unexpected connections between – Kolmogorov Complexity (the theory of randomness), and – Computational Complexity Theory  Which ought to have nothing to do with each other!

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity < 3 >< 3 > Complexity Classes P NP BPP PSPACE NEXP EXPSPACE P/poly

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity < 4 >< 4 > A Jewel of Derandomization  [Impagliazzo, Wigderson, 1997]: If there is a problem computable in time 2 n that requires circuits of size 2 εn, then P = BPP.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity < 5 >< 5 > Kolmogorov Complexity  C(x) = min{|d| : U(d) = x} – U is a “universal” Turing machine  K(x) = min{|d| : U(d) = x} – U is a “universal” prefix-free Turing machine  Important property – Invariance: The choice of the universal Turing machine U is unimportant (up to an additive constant).  x is random if C(x) ≥ |x|.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity < 6 >< 6 > Kolmogorov Complexity  C(x) = min{|d| : U(d) = x} – U is a “universal” Turing machine  K(x) = min{|d| : U(d) = x} – U is a “universal” prefix-free Turing machine  Important property – Invariance: The choice of the universal Turing machine U is unimportant (up to an additive constant).  x is random if C(x) ≥ |x|, or K(x) ≥ |x|.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity < 7 >< 7 > K, C, and Randomness  K(x) and C(x) are “close”: – C(x) ≤ K(x) ≤ C(x) + 2 log |x|  Two notions of randomness: – R C = {x : C(x) ≥ |x|} – R K = {x : K(x) ≥ |x|}  …actually, infinitely many notions of randomness: – R C U = {x : C U (x) ≥ |x|}

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity < 8 >< 8 > K, C, and Randomness  K(x) and C(x) are “close”: – C(x) ≤ K(x) ≤ C(x) + 2 log |x|  Two notions of randomness: – R C = {x : C(x) ≥ |x|} – R K = {x : K(x) ≥ |x|}  …actually, infinitely many notions of randomness: – R C U = {x : C U (x) ≥ |x|}, R K U = {x : K U (x) ≥ |x|}

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity < 9 >< 9 > K, C, and Randomness  When it makes no difference, we’ll write “R” instead of R C or R K.  Basic facts: – R is undecidable – …but it is not “easy” to use it as an oracle. – R is not NP-hard under poly-time ≤ m reductions, unless P=NP. – Things get more interesting when we consider more powerful types of reducibility.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Three Bizarre Inclusions  NEXP is contained in NP R. [ABK06]  PSPACE is contained in P R. [ABKMR06]  BPP is contained in {A : A is poly-time ≤ tt R}. [BFKL10] – A ≤ tt reduction is a “non-adaptive” reduction. – On input x, a list of queries is formulated before receiving any answer from the oracle.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Three Bizarre Inclusions  NEXP is contained in NP R. [ABK06]  PSPACE is contained in P R. [ABKMR06]  BPP is contained in P tt R. [BFKL10] “Bizarre”, because a non-computable “upper bound” is presented on complexity classes! We have been unable to squeeze larger complexity classes inside. Are these containments optimal?

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Three Bizarre Inclusions  NEXP is contained in NP R. [ABK06]  PSPACE is contained in P R. [ABKMR06]  BPP is contained in P tt R. [BFKL10] “Bizarre”, because a non-computable “upper bound” is presented on complexity classes! If we restrict attention to R K, then we can do better…

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Three Bizarre Inclusions  NEXP is contained in NP R K. – The decidable sets that are in NP R K for every U are in EXPSPACE. [AFG11]  PSPACE is contained in P R K.  BPP is contained in P tt R K. – The decidable sets that are in P tt R K for every U are in PSPACE. [AFG11]

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Three Bizarre Inclusions  NEXP is contained in NP R K (for every U). – The decidable sets that are in NP R K for every U are in EXPSPACE. [AFG11]  PSPACE is contained in P R K (for every U).  BPP is contained in P tt R K (for every U). – The decidable sets that are in P tt R K for every U are in PSPACE. [AFG11] – [CELM] The sets that are in P tt R K for every U are decidable.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Three Bizarre Inclusions  NEXP is contained in NP R K (for every U). – The decidable sets that are in NP R K for every U are in EXPSPACE. [AFG11]  PSPACE is contained in P R K (for every U).  BPP is contained in P tt R K (for every U). – The sets that are in P tt R K for every U are in PSPACE. [AFG11]

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Three Bizarre Inclusions  NEXP is contained in NP R K (for every U). – The sets that are in NP R K for every U are in EXPSPACE. [AFG11]  PSPACE is contained in P R K (for every U).  BPP is contained in P tt R K (for every U). – The sets that are in P tt R K for every U are in PSPACE. [AFG11]

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Three Bizarre Inclusions  NEXP is contained in NP R K (for every U). – The sets that are in NP R K for every U are in EXPSPACE. [AFG11]  Conjecture: This should hold for R C, too.  BPP is contained in P tt R K (for every U). – The sets that are in P tt R K for every U are in PSPACE. [AFG11]

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Three Bizarre Inclusions  NEXP is contained in NP R K (for every U). – The sets that are in NP R K for every U are in EXPSPACE. [AFG11]  This holds even for sets in EXP tt R K for all U!  BPP is contained in P tt R K (for every U). – The sets that are in P tt R K for every U are in PSPACE. [AFG11]

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Three Bizarre Inclusions  NEXP is contained in NP R K (for every U). – The sets that are in NP R K for every U are in EXPSPACE. [AFG11]  Conjecture: This class is exactly NEXP.  BPP is contained in P tt R K (for every U). – The sets that are in P tt R K for every U are in PSPACE. [AFG11] Conjecture: This class is exactly BPP.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Three Bizarre Inclusions  NEXP is contained in NP R K (for every U). – The sets that are in NP R K for every U are in EXPSPACE. [AFG11]  Conjecture: This class is exactly NEXP.  BPP is contained in P tt R K (for every U). – The sets that are in P tt R K for every U are in PSPACE. [AFG11] Conjecture: This class is exactly BPP P.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity K-Complexity and BPP vs P  BPP is contained in P tt R K (for every U). – The sets that are in P tt R K for every U are in PSPACE. Conjecture: This class is exactly P.  Some support for this conjecture [ABK06]: – The decidable sets that are in P dtt R C for every U are in P. – The decidable sets that are in P parity-tt R C for every U are in P.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity K-Complexity and BPP vs P  BPP is contained in P tt R K (for every U). – The sets that are in P tt R K for every U are in PSPACE. Conjecture: This class is exactly P.  New results support a weaker conjecture:  Conjecture: This class is contained in PSPACE ∩ P/poly.  More strongly: Every decidable set in P tt R is in P/poly.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity K-Complexity and BPP vs P  BPP is contained in P tt R K (for every U). – The sets that are in P tt R K for every U are in PSPACE. Conjecture: This class is exactly P.  New results support a weaker conjecture :  Conjecture: This class is contained in PSPACE ∩ P/poly.  More strongly: Every decidable set in P tt R is in P/poly (i.e., for every U, and for both C and K).

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity The Central Conjecture  Conjecture: Every decidable set in P tt R is in P/poly.  What can we show?

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity The Central Conjecture  Conjecture: Every decidable set in P tt R is in P/poly.  What can we show?  We show that a similar statement holds in the context of time-bounded K-complexity.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Time-Bounded K-complexity  Let t be a time bound. (Think of t as being large, such as Ackermann’s function.)  Define K t (x) to be min{|d| : U(d) = x in at most t(|x|) steps}.  Define R K t to be {x : K t (x) ≥ |x|}.  Define TTRT = {A : A is in P tt R K t for all large enough time bounds t}.  Vague intuition: Poly-time reductions should not be able to distinguish between R K t and R K, for large t.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity The Central Conjecture  Conjecture: Every decidable set in P tt R is in P/poly.  We show that a similar statement holds in the context of time-bounded K-complexity: – TTRT is contained in P/poly [ABFL12].  If t(n) = 2 2 n, then R K t is NOT in P/poly.  …which supports our “vague intuition”, because this set is not reducible to the time-t’- random strings for t’ >> t.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity The Central Conjecture  Conjecture: Every decidable set in P tt R is in P/poly.  We show that a similar statement holds in the context of time-bounded K-complexity: – TTRT is contained in P/poly [ABFL12].  BUT – The same P/poly bound holds, even if we consider P R K t instead of P tt R K t.  …and recall PSPACE is contained in P R.  So the “vague intuition” is wrong!

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity The Central Conjecture: An Earlier Approach  Conjecture: Every decidable set in P tt R is in P/poly.  We give a proof of a statement of the form: A n A j Ψ( n,j ) such that: if for each n and j there is a proof in PA of Ψ(n,j) then the conjecture holds.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Basic Proof Theory  Recall that Peano Arithmetic cannot prove the statement “PA is consistent”.  Let PA 1 be PA + “PA is consistent”.  Similarly, one can define PA 2, PA 3, …  “PA is consistent” can be formulated as “for all j, there is no length j proof of 0=1”.  For each j, PA can prove “there is no length j proof of 0=1”.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity The Central Conjecture: An Earlier Approach  Conjecture: Every decidable set in P tt R is in P/poly.  We give a proof (in PA 1 ) of a statement of the form: A n A j Ψ( n,j ) such that: if for each n and j there is a proof in PA of Ψ(n,j) then the conjecture holds.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity The Central Conjecture: The Earlier Approach Fails  The connections to proof theory were unexpected and intriguing, and seemed promising…  But unfortunately, it turns out that many of the statements Ψ(n,j) are independent of PA (and a related approach yields statements Ψ(n,j,k) that are independent of each system PA r ).

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity A High-Level View of the “Earlier Approach”  Let A be decidable, and let M be a poly-time machine computing a ≤ tt -reduction from A to R. Let Q(x) be the set of queries that M asks on input x. Let the size of Q(x) be at most f(|x|). Then there is a d such that for all x, there is a V containing only strings of length at most d+log f(|x|), such that M V (x) = A(x). Note: V says “long queries are non-random”.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity A Warm-Up  Let A be decidable, and let M be a poly-time machine computing a ≤ tt -reduction from A to R. Let Q(x) be the set of queries that M asks on input x. Let the size of Q(x) be at most f(|x|). Then there is a d such that for all x, there is a V containing only strings of length at most d+log f(|x|), such that M V (x) = A(x). Note: If some V works for all x of length n, then A is in P/poly.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Proof  Assume that for each d there is some x such that, for all V containing strings of length at most d+log f(|x|), M V (x)≠A(x).  Consider the machine that takes input (d,r) and finds x (as above) and outputs the r th element of Q(x).  This shows that each element y of Q(x) has C(y) ≤ log d + log f(|x|) + O(1)

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Proof  Assume that for each d there is some x such that, for all V containing strings of length at most d+log f(|x|), M V (x)≠A(x).  Consider the machine that takes input (d,r) and finds x (as above) and outputs the r th element of Q(x).  This shows that each element y of Q(x) has C(y) ≤ log d + log f(|x|) + O(1) < d + log f(|x|).  Thus if we pick V* to be R∩{0,1} d+log f(|x|), we see that M V* (x) = M R (x)

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Proof  Assume that for each d there is some x such that, for all V containing strings of length at most d+log f(|x|), M V (x)≠A(x).  Consider the machine that takes input (d,r) and finds x (as above) and outputs the r th element of Q(x).  This shows that each element y of Q(x) has C(y) ≤ log d + log f(|x|) + O(1) < d + log f(|x|).  Thus if we pick V* to be R∩{0,1} d+log f(|x|), we see that M V* (x) = M R (x) = A(x). Contradiction!

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Cleaning Things Up  Let A be decidable, and let M be a poly-time machine computing a ≤ tt -reduction from A to R. Let Q(x) be the set of queries that M asks on input x. Let the size of Q(x) be at most f(|x|). Then there is a d such that for all x, there is a V containing only strings of length at most d+log f(|x|) g A (|x|), such that M V (x) = A(x).

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Cleaning Things Up  Let A be decidable, and let M be a poly-time machine computing a ≤ tt -reduction from A to R. Let Q(x) be the set of queries that M asks on input x. Let the size of Q(x) be at most f(|x|). Then there is a d such that for all x, there is a V containing only strings of length at most d+log f(|x|) g A (|x|), such that M V (x) = A(x).

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Cleaning Things Up  Let A be decidable, and let M be a poly-time machine computing a ≤ tt -reduction from A to R. Let Q(x) be the set of queries that M asks on input x. Let the size of Q(x) be at most f(|x|). Then for all x, there is a V containing only strings in R of length at most g A (|x|) such that M V (x) = A(x).

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity A Refinement  Let A be decidable, and let M be a poly-time machine computing a ≤ tt -reduction from A to R. Let Q(x) be the set of queries that M asks on input x. Let the size of Q(x) be at most f(|x|). Then for all x, there is a V containing only strings in R of length at most g A (|x|) such that M V (x) = A(x).

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Approximating R  We can obtain a series of approximations to R (up to length g A (n)) as follows:  R n,0 = all strings of length at most g A (n).  R n,i+1 = R n,i minus the i+1 st string of length at most g A (n) that is found, in an enumeration of non-random strings.  R n,0, R n,1, R n,2, … R n,i* = R∩{0,1} g A (n)

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity A Refinement  Let A be decidable, and let M be a poly-time machine computing a ≤ tt -reduction from A to R. Let Q(x) be the set of queries that M asks on input x. Let the size of Q(x) be at most f(|x|). Then for all xє{0,1} n, for all i, there is a V containing only strings in R n,i such that M V (x) = A(x).

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Proof  Assume that for each d there is some x,i such that, for all V containing strings in R n,i of length at most d+log f(|x|), M V (x)≠A(x).  Consider the machine that takes input (d,r) and finds x,i (as above) and outputs the r th element of Q(x).  This shows that each element y of Q(x) has C(y) ≤ log d + log f(|x|) + O(1) < d + log f(|x|).  Thus if we pick V* to be R∩{0,1} d+log f(|x|), we see that M V* (x) = M R (x) = A(x). Contradiction!

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Where does PA come in??  Let A be decidable, and let M be a poly-time machine computing a ≤ tt -reduction from A to R. Let Q(x) be the set of queries that M asks on input x. Let the size of Q(x) be at most f(|x|). Then for all xє{0,1} n, for all i, there is a V containing only strings in R n,i such that M V (x) = A(x) and there is not a length-k proof that “for all i, V is not equal to R n,i ”.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity What went wrong with the earlier approach.  We have shown: For all xє{0,1} n, for all i, there is a V containing only short strings in R n,i such that M V (x) = A(x).  We were aiming at showing that one can swap the quantifiers, so that for all n, there is a V containing only short strings in R n,i such that, for all x of length n, M V (x) = A(x).  But there is a (useless) reduction M for which this is false. (M already knows the outcome of its queries, assuming that the oracle is R.)

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Open Questions:  Decrease the gap (NEXP vs EXPSPACE) between the lower and upper bounds on the complexity of the problems that are in NP R K for every U.  Some of our proofs rely on using R K. Do similar results hold also for R C ? – Disprove: The halting problem is in P tt R C.  Can the PSPACE ∩ P/poly bound (in the time- bounded setting) be improved to BPP?  Is this approach relevant at all to the P=BPP question?

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity P vs BPP  Our main intuition for P=BPP comes from [Impagliazzo, Wigderson]. Circuit lower bounds imply derandomization.  Note that this provides much more than “merely” P=BPP; it gives a recipe for simulating any probabilistic algorithm.  Goldreich has argued that any proof of P=BPP actually yields pseudorandom generators (and hence a “recipe” as above)… – …but this has only been proved for the “promise problem” formulation of P=BPP.

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity P vs BPP  Recall that TTRT sits between BPP and PSPACE ∩ P/poly.  A proof that TTRT = P would show that BPP = P – but it is not clear that this would yield any sort of recipe for constructing useful pseudorandom generators.  Although it would be a less “useful” approach, perhaps it might be an easier approach?

Eric Allender: Curiouser & Curiouser: The Link between Incompressibility and Complexity Thank you!