Lecture 5. The Incompressibility Method

Slides:



Advertisements
Similar presentations
Completeness and Expressiveness
Advertisements

Lecture 9. Resource bounded KC K-, and C- complexities depend on unlimited computational resources. Kolmogorov himself first observed that we can put resource.
Theory of Computing Lecture 23 MAS 714 Hartmut Klauck.
Circuit and Communication Complexity. Karchmer – Wigderson Games Given The communication game G f : Alice getss.t. f(x)=1 Bob getss.t. f(y)=0 Goal: Find.
QuickSort Average Case Analysis An Incompressibility Approach Brendan Lucier August 2, 2005.
Theory of Computing Lecture 3 MAS 714 Hartmut Klauck.
Theory of Computing Lecture 16 MAS 714 Hartmut Klauck.
Lecture 12: Lower bounds By "lower bounds" here we mean a lower bound on the complexity of a problem, not an algorithm. Basically we need to prove that.
The number of edge-disjoint transitive triples in a tournament.
1 Introduction to Computability Theory Lecture12: Reductions Prof. Amos Israeli.
1 Introduction to Computability Theory Lecture4: Non Regular Languages Prof. Amos Israeli.
1 Introduction to Computability Theory Lecture13: Mapping Reductions Prof. Amos Israeli.
Randomized Algorithms and Randomized Rounding Lecture 21: April 13 G n 2 leaves
1 Introduction to Computability Theory Lecture4: Non Regular Languages Prof. Amos Israeli.
DAST 2005 Week 4 – Some Helpful Material Randomized Quick Sort & Lower bound & General remarks…
Lecture 3. Relation with Information Theory and Symmetry of Information Shannon entropy of random variable X over sample space S: H(X) = ∑ P(X=x) log 1/P(X=x)‏,
Lecture 2 We have given O(n 3 ), O(n 2 ), O(nlogn) algorithms for the max sub-range problem. This time, a linear time algorithm! The idea is as follows:
Kolmogorov complexity and its applications Ming Li School of Computer Science University of Waterloo CS 898.
Theory of Computing Lecture 15 MAS 714 Hartmut Klauck.
February 18, 2015CS21 Lecture 181 CS21 Decidability and Tractability Lecture 18 February 18, 2015.
The Complexity of Optimization Problems. Summary -Complexity of algorithms and problems -Complexity classes: P and NP -Reducibility -Karp reducibility.
Computational Complexity Theory Lecture 2: Reductions, NP-completeness, Cook-Levin theorem Indian Institute of Science.
Theory of Computing Lecture 17 MAS 714 Hartmut Klauck.
Elementary Sorting Algorithms Many of the slides are from Prof. Plaisted’s resources at University of North Carolina at Chapel Hill.
Lecture 5. The Incompressibility Method  A key problem in computer science: analyze the average case performance of a program.  Using the Incompressibility.
Lecture 5. The Incompressibility Method  A key problem in computer science: analyze the average case performance of a program.  Using the Incompressibility.
Kolmogorov complexity and its applications Ming Li School of Computer Science University of Waterloo CS860,
Lecture 5-1. The Incompressibility Method, continued  We give a few more examples using the incompressibility method. We avoid ones with difficult and.
Prof. Amr Goneid, AUC1 Analysis & Design of Algorithms (CSCE 321) Prof. Amr Goneid Department of Computer Science, AUC Part 1. Complexity Bounds.
Lecture 5. The Incompressibility Method  A key problem in computer science: analyze the average case performance of a program.  Using the Incompressibility.
Sorting Lower Bounds n Beating Them. Recap Divide and Conquer –Know how to break a problem into smaller problems, such that –Given a solution to the smaller.
Lecture. Today Problem set 9 out (due next Thursday) Topics: –Complexity Theory –Optimization versus Decision Problems –P and NP –Efficient Verification.
CSCI 2670 Introduction to Theory of Computing December 2, 2004.
CSCI 2670 Introduction to Theory of Computing December 7, 2005.
Lecture 3. Symmetry of Information In Shannon information theory, the symmetry of information is well known. The same phenomenon is also in Kolmogorov.
Theory of Computational Complexity Probability and Computing Chapter Hikaru Inada Iwama and Ito lab M1.
Advanced Algorithms Analysis and Design
Chapter 10 NP-Complete Problems.
Fundamental Data Structures and Algorithms
Andreas Klappenecker [partially based on the slides of Prof. Welch]
Lecture 3: Parallel Algorithm Design
CSE322 PUMPING LEMMA FOR REGULAR SETS AND ITS APPLICATIONS
Analysis of Algorithms
The Incompressibility Method using Kolmogorov Complexity
Computability and Complexity
PDAs Accept Context-Free Languages
Introduction to Algorithms
Lecture 3: Divide-and-conquer
Chapter 5. Optimal Matchings
Lecture 6. Prefix Complexity K
ICS 353: Design and Analysis of Algorithms
Complexity 6-1 The Class P Complexity Andrei Bulatov.
CS 583 Analysis of Algorithms
Lecture 5-1. The Incompressibility Method, continued
The Satisfiability Problem
Data Structures Review Session
Deterministic PDAs - DPDAs
Non-regular languages
Data Structures Sorting Haim Kaplan & Uri Zwick December 2014.
Lecture 10: Query Complexity
CS21 Decidability and Tractability
Netzer & Miller 1990: On the Complexity of Event Ordering for Shared-Memory Parallel Program Executions.
NP-Completeness Reference: Computers and Intractability: A Guide to the Theory of NP-Completeness by Garey and Johnson, W.H. Freeman and Company, 1979.
Elementary Sorting Algorithms
Mathematical Analysis of Algorithms
CSE 373 Sorting 2: Selection, Insertion, Shell Sort
Switching Lemmas and Proof Complexity
CS151 Complexity Theory Lecture 5 April 16, 2019.
Sorting We have actually seen already two efficient ways to sort:
Turing Machines Everything is an Integer
Presentation transcript:

Lecture 5. The Incompressibility Method Talk at U of Maryland Lecture 5. The Incompressibility Method A key problem in computer science: analyze the average case performance of a program. Using the Incompressibility Method: Give the program a random input (with high Kolmogorov complexity) Analyze the program with respect to this single and fixed input. This is usually easier using the fact this input is incompressible. The running time for this single input is the average case running time of all inputs!

Formal language theory Talk at U of Maryland Formal language theory Example: Show L={0k1k | k>0} not regular. By contradiction, assume that DFA M accepts L. Choose k so that C(k) >> 2|M|. Simulate M: 000 … 0 111 … 1 C(k) < |M| + q + O(1) < 2|M|. Contradiction. Remark. Generalize to iff condition: more powerful & easier to use than “pumping lemmas”. k k stop here M q

Talk at U of Maryland Combinatorics Example There is a tournament (complete directed graph) T of n players that contains no large transitive subtournaments (>1 + 2 log n). Proof by Picture: Choose a random T. One bit codes an edge. C(T) ³ n(n-1)/2. If there is a large transitive subtournament, then a large number of edges are given for free! C(T)< n(n - 1)/2 - subgraph-edges + overhead Linearly ordered subgraph. Easy to describe T

Fast adder S= xy; C= carry sequence; Talk at U of Maryland Fast adder Example. Fast addition on average. Ripple-carry adder: n steps adding n-bit numbers. Carry-lookahead adder: 2 log n steps. Burks-Goldstine-von Neumann (1946): logn expected steps. S= xy; C= carry sequence; while (C≠0) { S= SC; C= new carry sequence; } Average case analysis: Fix x, take random y s.t. C(y|x)≥|y| x = … u1 … (Max such u is carry length) y = … û1 …, û is complement of u If |u| > log n, then C(y|x)<|y|. Average over all y, get logn. QED

Talk at U of Maryland Sorting Given n elements (in an array). Sort them into ascending order. This is the most studied fundamental problem in computer science. Shellsort (1959): P passes. In each pass, move the elements “in some stepwise fashion” (Bubblesort) Open for over 40 years: a nontrivial general average case complexity lower bound of Shellsort?

Shellsort Algorithm Using p increments h1, … , hp, with hp=1 Talk at U of Maryland Shellsort Algorithm Using p increments h1, … , hp, with hp=1 At k-th pass, the array is divided in hk separate sublists of length n/hk (taking every hk-th element). Each sublist is sorted by insertion/bubble sort. ------------- Application: Sorting networks --- nlog2 n comparators.

Talk at U of Maryland Shellsort history Invented by D.L. Shell [1959], using pk= n/2k for step k. It is a Θ(n2) time algorithm Papernow&Stasevitch [1965]: O(n3/2) time. Pratt [1972]: O(nlog2n) time. Incerpi-Sedgewick, Chazelle, Plaxton, Poonen, Suel (1980’s) – worst case, roughly,Θ(nlog2n / (log logn)2). Average case: Knuth [1970’s]: Θ(n5/3) for p=2 Yao [1980]: p=3 Janson-Knuth [1997]: Ω(n23/15) for p=3. Jiang-Li-Vitanyi [J.ACM, 2000]: Ω(pn1+1/p) for any p.

Shellsort Average Case Lower bound Talk at U of Maryland Shellsort Average Case Lower bound Theorem. p-pass Shellsort average case T(n) ≥ pn1+1/p Proof. Fix a random permutation Π with Kolmogorov complexity nlogn. I.e. C(Π)≥ nlogn. Use Π as input. For pass i, let mi,k be the number of steps the kth element moves. Then T(n) = Σi,k mi,k From these mi,k's, one can reconstruct the input Π, hence Σ log mi,k ≥ C(Π) ≥ n logn Maximizing the left, all mi,k must be the same. Call it m. Σ log m = pn log m ≥ Σ log mi,k ≥ nlogn  mp ≥ n. So T(n) = pnm > pn1+1/p. ■ Corollary: p=1: Bubblesort Ω(n2) average case lower bound. p=2: n1.5 lower bound. p=3, n4/3 lower bound

Talk at U of Maryland Heapsort 1964, JWJ Williams [CACM 7(1964), 347-348] first published Heapsort algorithm Immediately it was improved by RW Floyd. Worst case O(nlogn). Open for 40 years: Which is better in average case: Williams or Floyd? R. Schaffer & Sedgewick (1996). Ian Munro provided the solution here.

Heapsort average analysis (I. Munro) Talk at U of Maryland Heapsort average analysis (I. Munro) Average-case analysis of Heapsort. Heapsort: (1) Make Heap. O(n) time. (2) Deletemin, restore heap, repeat. Williams Floyd log n d d 2 log n - 2d log n + d comparisons/round Fix random heap H, C(H) > n log n. Simulate Step (2). Each round, encode the red path in log n -d bits. The n paths describe the heap! Hence, total n paths, length ³ n log n, d must be a constant. Floyd takes n log n comparisons, and Williams takes 2n log n.

A selected list of results proved by the incompressibility method Talk at U of Maryland A selected list of results proved by the incompressibility method Ω(n2) for simulating 2 tapes by 1 (20 years) k heads > k-1 heads for PDAs (15 years) k one-ways heads can’t do string matching (13 yrs) 2 heads are better than 2 tapes (10 years) Average case analysis for heapsort (30 years) k tapes are better than k-1 tapes. (20 years) Many theorems in combinatorics, formal language/automata, parallel computing, VLSI Simplify old proofs (Hastad Lemma). Shellsort average case lower bound (40 years) Lovazs Local Lemma short proof.

More on formal language theory Talk at U of Maryland More on formal language theory Lemma (Li-Vitanyi) Let L  V*, and Lx={y: xy  L}. Then L is regular implies there is c for all x,y,n, let y be the n-th element in Lx, we have C(y) ≤ C(n)+c. Proof. Like example. QED. Example 2. {1p : p is prime} is not regular. Proof. Let pi, i=1,2 …, be the list of primes. Then pk+1 is the first element in LPk, hence by Lemma, C(pk+1)≤O(1). Impossible. QED

Characterizing regular sets Talk at U of Maryland Characterizing regular sets For any enumeration of *={y1,y2, …}, define characteristic sequence X of Lx={yi : xyi L} by Xi = 1 iff xyi L Theorem. L is regular iff there is a c for all x,n, C(Xn|n) < c

Low probability events -- LLL Robin Moser Low probability events -- LLL So far, we have used Kolmogorov complexity for “very high probability events” hence we obtained “average case” complexity of our problems. But there are times when things happen with low probabilities (and do not hold for average case), can the incompressibility method still be applied? A typical such case is captured by the Lovasz Local Lemma. Moser gave such a simple (and contructive – traditional probabilistic proof is not constructive) proof at STOC’2009. We follow Lance Fortnow’s proof here.

The Lovasz Local Lemma (Symmetric version, all events with same probability) Consider a k-CNF formula Ф with n variable and n clauses. Assume each clause shares a variable with at most r other clauses. Then there is a constant d such that if r < 2k-d , then Ф is satisfiable. Moreover, we can find that assignment in time polynomial in m and n. Interpretation: if we have n independent events, each with probability <1, then there is a positive probability that none of the events will occur. LLL relaxes this to “almost independent”

Incompressibility Proof of the LLL Lance Fortnow Incompressibility Proof of the LLL Proof. Fix a Kolmogorov random string x of length n+sk (s determined later) such that C(x | Ф, k, s, r, m, n) ≥ |x| = n+sk. Algorithm Solve(Ф) 1. Use the first n bits as assignment for Ф 2. While there is an unsatisfied clause C 3. Fix(C); Algorithm Fix(C) 4. Replace the variables of C with next k bits in x; 5. While there is a unsatisfied clause D that shares a variable with C 6. Fix(D); Suppose the algorithm makes s Fix calls. If we know which clause is being fixed, we know the clause is not satisfied, hence we know all k bits corresponding to this clause. Hence we can describe x by its first n bits, plus the description of the clauses being fixed. For each of the m Fix calls from Solve, it takes log m bits to indicate the clause. For other (recursive) Fix calls from Fix, it takes log r + O(1)$ bits since there are only r other clauses that share a variable with the current clause. Thus, we must have m log m + s(log r + O(1)) + n ≥ n + sk. We must have r ≤ 2k-d, and letting s ≥ m log m finishes the proof.