CSCI-256 Data Structures & Algorithm Analysis Lecture Note: Some slides by Kevin Wayne. Copyright © 2005 Pearson-Addison Wesley. All rights reserved. 4

Computational Tractability A major focus of algorithm design is to find efficient algorithms for computational problems. What does it mean for an algorithm to be efficient? As soon as an Analytic Engine exists, it will necessarily guide the future course of the science. Whenever any result is sought by its aid, the question will arise - By what course of calculation can these results be arrived at by the machine in the shortest time? - Charles Babbage Charles Babbage (1864) Analytic Engine (schematic)

Some Initial Attempts at Defining Efficiency Proposed Definition of Efficiency (1): An algorithm is efficient if, when implemented, it runs quickly on real input instances –Where does it run? Even bad algorithms can run quickly when applied to small test cases on extremely fast processors –How is it implemented? Even good algorithms can run slowly when they are coded sloppily –What is a “real” input instance? Some instances can be much harder than others –How well, or badly, does the running time scale as problem sizes grow to unexpected levels? –We need a concrete definition that is platform-independent, instance-independent, and of predictive value with respect to increasing input sizes

Worst-Case Analysis Worst case running time: Obtain bound on largest possible running time of algorithm on input of a given size N –Draconian view, but hard to find effective alternative –Generally captures efficiency in practice Average case running time: Obtain bound on running time of algorithm on random input as a function of input size N –Hard (or impossible) to accurately model real instances by random distributions –Algorithm tuned for a certain distribution may perform poorly on other inputs

Brute-Force Search Ok, but what is a reasonable analytical benchmark that can tell us whether a running time bound is impressive or weak? –For many non-trivial problems, there is a natural brute force search algorithm that checks every possible solution (i.e., try all possibilities, see if any one works, N! for Stable Matching) –Note that this is an intellectual cop-out; it provides us with absolutely no insight into the problem structure –Thus, a first simple guide is by comparison with brute-force search Proposed Definition of Efficiency (2): An algorithm is efficient if it achieves qualitatively better worst-case performance than brute-force search –Still vague, what is “qualitatively better performance”?

Polynomial Time as a Definition of Efficiency Desirable scaling property: Algorithms with polynomial run time have the property that increasing the problem size by a constant factor increases the run time by at most a constant factor An algorithm is polynomial-time if the above scaling property holds There exists constants c > 0 and d > 0 such that on every input of size N, its running time is bounded by cN d steps.

Polynomial Time as a Definition of Efficiency Proposed Definition of Efficiency (3): An algorithm is efficient if it has a polynomial running time Justification: It really works in practice! –Generally, polynomial time seems to capture the algorithms which are efficient in practice –Although 6.02   N 20 is technically polynomial-time, it would be useless in practice –In practice, the polynomial-time algorithms that people develop almost always have low constants and low exponents –Breaking through the exponential barrier of brute force typically exposes some crucial structure of the problem

Polynomial Time as a Definition of Efficiency One further reason why the mathematical formalism and the empirical evidence seem to line up well in the case of polynomial-time solvability is that the gulf between the growth rates of polynomial and exponential functions is enormous

Asymptotic Order of Growth We could give a very concrete statement about the running time of an algorithm on inputs of size N such as: On any input of size N, the algorithm runs for at most 1.62N N + 8 steps –Finding such a precise bound may be an exhausting activity, and more detail than we wanted anyway –Extremely detailed statements about the number of steps an algorithm executes are often meaningless. Why?

Why Ignore Constant Factors? Constant factors are arbitrary –Depend on the implementation –Depend on the details of the model Determining the constant factors is tedious and provides little insight

Why Emphasize Growth Rates? The algorithm with the lower growth rate will be faster for all but a finite number of cases Performance is most important for larger problem size As memory prices continue to fall, bigger problem sizes become feasible Improving growth rate often requires new techniques

Formalizing Growth Rates Upper bounds: T(n) is O(f(n)) if there exist constants c > 0 and n 0  0 such that for all n  n 0 we have T(n)  c · f(n) Lower bounds: T(n) is  (f(n)) if there exist constants c > 0 and n 0  0 such that for all n  n 0 we have T(n)  c · f(n) Tight bounds: T(n) is  (f(n)) if T(n) is both O(f(n)) and  (f(n)) Ex: T(n) = 32n n + 32 –T(n) is O(n 2 ), O(n 3 ),  (n 2 ),  (n), and  (n 2 ) –T(n) is not O(n),  (n 3 ),  (n), or  (n 3 )