ECOE 456/556: Algorithms and Computational Complexity Lecture 1 Serdar Taşıran

ECOE 556, 2 This Week’s Outline Introduction and basics Fundamental concepts, formal definitions, pseudocode Motivation for studying algorithms Analyzing algorithms Growth of functions Asymptotic notation Recurrence equations Solving recurrences to compute asymptotic complexity

ECOE 556, 3 Algorithms Al-gorithm: Named after the 9 th century Arab mathematician al Harezmi (not Al Gore) An algorithm: A tool for solving a well-specified computational problem. Problem statement: Inputs, outputs The desired input/output relationship Algorithm describes a specific computational procedure for producing the required output Example: Sorting Input: A sequence of n numbers Output: A reordering of the sequence such that a’ 1  a’ 2  …  a’ n Given the input, the algorithm should produce Called an instance of the problem

ECOE 556, 4 When do we need algorithmic solutions? Almost every engineering application The Human Genome Project Bioinformatics in general The Internet Routing algorithms Searching, indexing E-commerce Cryptography Authentication Scheduling, optimization of industrial processes Numerical algorithms e.g. matrix multiplication, finite-element simulation

ECOE 556, 5 When do we need non-trivial data structures? Almost any industrial strength software tool needs them Need to represent Sets Relations Discrete functions Sequences Queues … Have varied requirements for what kind of operations to perform on them. Insert, delete, lookup Next, previous Minimum, maximum …

ECOE 556, 6 Why worry about efficiency? Hardware and memory are fast and cheap. Computing getting cheaper and cheaper. Intelligent manpower is expensive Why worry about efficient algorithms and data structures?

ECOE 556, 7 Why worry about algorithms and data structures? Algorithms are as important a technology as other advanced technologies, such as Hardware architecture Graphical user interfaces Programming technologies Networks A stupid approach uses up computing power faster than you might think. Examples: Sorting a million numbers Interactive graphics: Algorithms must terminate in 1/30 of a sec. O(n 2 ) algorithm 2n 2 instructions 10 9 inst/second 2000 sec. O(n lg n) algorithm 50 n lg n instructions 10 7 inst/second 100 sec.

ECOE 556, 8 Movie: Sorting Algorithms

ECOE 556, 9 Computationally hard problems Polynomial complexity: Requires resources O(P(n)) for some polynomial Exponential complexity Why divide this way? Most polynomial problems have low-degree polynomial complexity Any exponential is asymptotically bigger than any polynomial Grey area in between No known polynomial algorithm No proof that one doesn’t exist Interesting class of problems: NP-complete problems Come up very often in practical applications All computationally equivalent If an efficient algorithm exists for one, all NP-complete problems are polynomially solvable

ECOE 556, 10 Analyzing algorithms Is the algorithm correct? Does it terminate on all inputs? Does it produce the required output? What amount of resources does the algorithm use up? Memory Communication bandwidth Logic gates (if implemented in hardware), speed of logic circuit Running time Machine model: Single processor, random-access machine One instruction at a time, no concurrent processing Memory access also counts as one instruction Idealized, but adequate for characterizing general behavior of algorithms

ECOE 556, 11 Example: Insertion Sort Takes array A[1..n] containing a sequence of length n to be sorted Sorts the array in place Numbers rearranged inside A with at most a constant number of them stored outside.

ECOE 556, 12

ECOE 556, 13

ECOE 556, 14

ECOE 556, 15 Correctness of INSERTION-SORT

ECOE 556, 16 Correctness of INSERTION-SORT Loop invariant: At the start of each iteration, the subarray A[1..j-1] contains the elements originally in A[1..j-1] but in sorted (increasing) order Prove initialization, maintenance, termination Invariant must imply interesting property about algorithm

ECOE 556, 17 Analyzing running time Depends on input size: How to quantify? Number of items in input Number of bits required to encode input Running time = Number of primitive steps executed in computation Each line of pseudocode takes constant time

ECOE 556, 18 Running time of INSERTION-SORT

ECOE 556, 19 Running-time of Insertion-Sort

ECOE 556, 20 Running time of INSERTION-SORT Depends on the input instance Best case: Worst case: We usually care about the worst and average case Why worst case? Upper bound May occur often. e.g. search, database look-up Average case often as bad as worst case Many exceptions Don’t need to assume particular input probability distribution Simplifying assumption: We care about rate of growth of running time only. Called “asymptotic complexity”.

ECOE 556, 21 Growth of functions: Asymptotic notation  Examples: 1/6 n 2 – 7n =  (n 2 ) n 3 =  (n 2 ) ?

ECOE 556, 22 How to design algorithms? Many styles We’ll see examples throughout the semester Insertion sort was an example of an “incremental algorithm” Another common paradigm: Divide and conquer Steps Divide into simpler/smaller subproblems Solve subproblems recursively Combine results of subproblems Example: Merge-sort

ECOE 556, 23 Merge Sort Divide into two subsequences of half size Call yourself recursively on the halves Combine results How do you prove MergeSort correct?

ECOE 556, 24 Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.

ECOE 556, 25 Recurrence equation for MERGE-SORT Express T(n) in terms of subproblems and cost of division into subproblems. T(n/2) O(n)

ECOE 556, 26 Recurrence equation for MERGE-SORT Express T(n) in terms of subproblems and cost of division into subproblems. T(n/2) O(n) T(n) =  (n lg n) Insertion sort was  (n 2 )

ECOE 556, 27 Solving recurrences The substitution method The recursion tree method A graphical method for coming up with a good guess Guess needs to be verified using substitution method The master method Technicalities Integer arguments. Floors, ceilings Boundary conditions: T(n) constant for some small enough n. Powers of k

ECOE 556, 28 Substitution method Guess the form of the solution Use mathematical induction to verify correctness Example: T(n) = 2T(n/2) + n Guess T(n) = O(n lg n) Prove that T(n)  cn lg n for appropriate choice of c using strong induction How about T(n) = O(n)?

ECOE 556, 29 The recursion tree method: MERGE-SORT

ECOE 556, 30 The recursion tree method

ECOE 556, 31 T(n) = 3 T(n/4) + cn 2

ECOE 556, 32 T(n) = 3 T(n/4) + cn 2

ECOE 556, 33