I Advanced Algorithms Analysis

What is Algorithm?  A computer algorithm is a detailed step-by-step method for solving a problem by using a computer.  An algorithm is a sequence of unambiguous instructions for solving a problem in a finite amount of time.  An Algorithm is well defined computational procedure that takes some value, or set of values, as input and produces some value, or set of values as output.  More generally, an Algorithm is any well defined computational procedure that takes collection of elements as input and produces a collection of elements as output.

Popular Algorithms, Factors of Dependence  Most basic and popular algorithms are  Sorting algorithms  Searching algorithms Which algorithm is best?  Mainly, it depends upon various factors, for example in case of Sorting :  The number of items to be sorted  The extent to which the items are already sorted  Possible restrictions on the item values  The kind of storage device to be used etc.

One Problem, Many Algorithms Problem  The statement of the problem specifies, in general terms, the desired input/output relationship. Algorithm  The algorithm describes a specific computational procedure for achieving input/output relationship. Example  One might need to sort a sequence of numbers into non- decreasing order. Algorithms  Various algorithms e.g. Merge sort, Quick sort, Heap sort etc.

Roadmap  Different problems  Sorting  Searching  String processing  Graph problems  Geometric problems  Numerical problems Different design paradigms – Divide-and-conquer – Incremental – Dynamic programming – Greedy algorithms – Randomized/probabilistic – NP Completeness

Important Designing Techniques  Brute Force  Straightforward, naive approach  Mostly expensive  Divide-and-Conquer  Divide into smaller sub-problems  Iterative Improvement  Improve one change at a time  Decrease-and-Conquer  Decrease instance size  Transform-and-Conquer  Modify problem first and then solve it  Space and Time Tradeoffs  Use more space now to save time later

Important Designing Techniques  Greedy Approach  Locally optimal decisions, can not change once made.  Efficient  Easy to implement  The solution is expected to be optimal  Every problem may not have greedy solution  Dynamic programming  Decompose into sub-problems like divide and conquer  Sub-problems are dependant  Record results of smaller sub-problems  Re-use it for further occurrence  Mostly reduces complexity exponential to polynomial

Problem Solving Phases  Analysis  How does system work?  Breaking a system down to known components  How components (processes) relate to each other  Breaking a process down to known functions  Synthesis  Building tools  Building functions with supporting tools  Composing functions to form a process  How components should be put together?  Final solution

Problem Solving Process  Problem  Strategy  Algorithm  Input  Output  Steps  Analysis  Correctness  Time & Space  Optimality  Implementation  Verification

Analyzing Algorithms  Predict the amount of resources required: memory: how much space is needed? computational time: how fast the algorithm runs?  FACT: running time grows with the size of the input  Input size (number of elements in the input)  Size of an array, polynomial degree, # of elements in a matrix, # of bits in the binary representation of the input, vertices and edges in a graph Def: Running time = the number of primitive operations (steps) executed before termination  Arithmetic operations (+, -, *), data movement, control, decision making (if, while), comparison

Algorithm Efficiency vs. Speed E.g.: sorting n numbers Friend’s computer-A = 10 9 instructions/second Friend’s algorithm = 2 n 2 instructions Your computer-B = 10 7 instructions/second Your algorithm = 50 nlgn instructions Your friend = You = Sort 10 6 numbers! 20 times better!!

Model of Computation (Assumptions)  Design assumption  Level of abstraction which meets our requirements  Neither more nor less e.g. [0, 1] infinite continuous interval  Analysis independent of the variations in  Machine  Operating system  Programming languages  Compiler etc.  Low-level details will not be considered  Our model will be an abstraction of a standard generic single- processor machine, called a random access machine or RAM.

Model of Computation (Assumptions)  A RAM is assumed to be an idealized machine  Infinitely large random-access memory  Instructions execute sequentially  Every instruction is in fact a basic operation on two values in the machines memory which takes unit time.  These might be characters or integers.  Example of basic operations include  Assigning a value to a variable  Arithmetic operation (+, -, ×, /) on integers  Performing any comparison e.g. a < b  Boolean operations  Accessing an element of an array.

Model of Computation (Assumptions)  In theoretical analysis, computational complexity  Estimated in asymptotic sense, i.e.  Estimating for large inputs  Big O, Omega, Theta etc. notations are used to compute the complexity  Asymptotic notations are used because different implementations of algorithm may differ in efficiency  Efficiencies of two given algorithm are related  By a constant multiplicative factor  Called hidden constant.

Algorithm Analysis: Example  Alg.: MIN ( a[1], …, a[n] ) m ← a[1]; for i ← 2 to n if a[i] < m then m ← a[i];  Running time:  the number of primitive operations (steps) executed before termination T(n) =1 [first step] + (n) [for loop] + (n-1) [if condition] + (n-1) [the assignment in then] = 3n - 1  Order (rate) of growth:  The leading term of the formula  Expresses the asymptotic behavior of the algorithm

Typical Running Time Functions  1 (constant running time):  Instructions are executed once or a few times  logN (logarithmic)  A big problem is solved by cutting the original problem in smaller sizes, by a constant fraction at each step  N (linear)  A small amount of processing is done on each input element  N logN  A problem is solved by dividing it into smaller problems, solving them independently and combining the solution

Typical Running Time Functions  N 2 (quadratic)  Typical for algorithms that process all pairs of data items (double nested loops)  N 3 (cubic)  Processing of triples of data (triple nested loops)  N K (polynomial)  2 N (exponential)  Few exponential algorithms are appropriate for practical use

Some rules of thumb  Multiplicative constants can be omitted  14n 2 becomes n 2  7 log n become log n  Lower order functions can be omitted  n + 5 becomes n  n 2 + n becomes n 2  n a dominates n b if a > b  n 2 dominates n, so n 2 +n becomes n 2  n 1.5 dominates n 1.4

Some rules of thumb  a n dominates b n if a > b  3 n dominates 2 n  Any exponential dominates any polynomial  3 n dominates n 5  2 n dominates n c  Any polynomial dominates any logorithm  n dominates log n or log log n  n 2 dominates n log n  n 1/2 dominates log n  Do not omit lower order terms of different variables (n 2 + m) does not become n 2

Why Faster Algorithms? InputlogNNlogNN^2N^32^N

Drawbacks in Model of Computation First poor assumption  We assumed that each basic operation takes constant time, i.e. model allows  Adding  Multiplying  Comparing etc. two numbers of any length in constant time  Addition of two numbers takes a unit time!  Not good because numbers may be arbitrarily  Addition and multiplication both take unit time!  Again very bad assumption

Model of Computation  But with all these weaknesses, RAM model is not so bad because we have to give the  Comparison not the absolute analysis of any algorithm.  We have to deal with large inputs not with the small size  Model seems to work well, describing computational power of modern nonparallel machines Can we do Exact Measure of Efficiency ?  Exact measure of efficiency can be sometimes computed but it usually requires certain assumptions concerning implementation

Model of Computation  Analysis will be performed with respect to this computational model for comparison of algorithms  We will give asymptotic analysis not detailed comparison i.e. for large inputs  We will use generic uniprocessor random-access machine (RAM) in analysis  All memory equally expensive to access  No concurrent operations  All reasonable instructions take unit time, except, of course, function calls

Asymptotic Notation  Running time of an algorithm as a function of input size n for large n.  Define a set of functions; in practice used to compare two function sizes.  The notations describe different rate-of-growth relations between the defining function and the defined set of functions. 24

O - Notation  Asymptotic Less than and Equal  For function g(n), we define O(g(n)), big-O of n, as:  Set of all functions whose rate of growth is the same as or lower than that of g(n).  g(n) is an asymptotically upper bound for f(n).  f(n) ≤ g(n) 25

O - Notation 26 20n nlgn + 5= O(n 3 ) 3lgn + lg(lgn)= O(lgn) = O(1)

Rules to Manipulate Big-O If T1(n) = O(f(n)) and T2(n) = O(g(n)) then T1(n) + T2(n) = max(O(f(n)), O(g(n))) and T1(n) * T2(n) = O(f(n) * g(n)) 27

 - Notation  Asymptotic Equality  For function g(n), we define  (g(n)), big-Theta of n, as:  Set of all functions that have the same rate of growth as g(n).  g(n) is an asymptotically tight bound for f(n).  f(n) = g(n)old concept  f(n)  g(n)new concept 28

 - Notation 29 Θ(n 3 ):n 3 5n 3 + 4n 105n 3 + 4n 2 + 6n Θ(n 2 ):n 2 5n 2 + 4n + 6 n Θ(log n):log n log n 2 log (n + n 3 )

 - Notation  Asymptotic Lower bound  For function g(n), we define  (g(n)), big- Omega of n, as:  Set of all functions whose rate of growth is the same as or higher than that of g(n).  f(n) = g(n)old concept  f(n)  g(n)new concept 30

Examples  5n 2 =  (n)  n =  (2n), n 3 =  (n 2 ), n =  (logn)  c, n 0 such that: 0  cn  5n 2  cn  5n 2  c = 1 and n 0 = 1

Relations Between , O, 

Development of Notation  Drop insignificant terms and constants  Say function is of O(n 2 ) called Big-O of n 2  Common Big-O functions in algorithm analysis  g(n) = 1 (growth is constant)  g(n) = log 2 n (growth is logarithmic)  g(n) = n (growth is linear)  g(n) = n log 2 n (growth is faster than linear)  g(n) = n 2 (growth is quadratic)  g(n) = 2 n (growth is exponential)

Conclusion  What, Why and Where Algorithms?  Designing Techniques  Problem solving Phases and Procedure  Model of computations:  Major assumptions at design and analysis level  Merits and demerits, justification of assumptions taken  We proved that algorithm is a technology  Discussed importance of algorithms  In almost all areas of computer science and engineering  Algorithms make difference in users and developers  Asymptotic Notations