CSC2105: Algorithms Mashiour Rahman American International University Bangladesh.

CSC2105: Algorithms Mashiour Rahman mashiour@aiub.edu American International University Bangladesh

CSC2105Mashiour Rahman2 Literature Introduction to Algorithms, Second Edition, Thomas H. Cormen, Charle E. Leiserson, Ronald L. Rivest, Clifford Stein (CLRS). Introduction to Algorithms, Second Edition, Thomas H. Cormen, Charle E. Leiserson, Ronald L. Rivest, Clifford Stein (CLRS). Fundamental of Computer Algorithms, Ellis Horowitz, Sartaj Sahni, Sanguthevar Rajasekaran (HSR). Fundamental of Computer Algorithms, Ellis Horowitz, Sartaj Sahni, Sanguthevar Rajasekaran (HSR). Helpful link for Problem Solving : http://acm.uva.es/problemset/ Helpful link for Problem Solving : http://acm.uva.es/problemset/

CSC2105Mashiour Rahman3 The Goals of this Course The main things we will learn in this course: The main things we will learn in this course: To think algorithmically and get the spirit of how algorithms are designed. To think algorithmically and get the spirit of how algorithms are designed. To get to know a toolbox of classical algorithms. To get to know a toolbox of classical algorithms. To learn a number of algorithm design techniques (such as divide-and-conquer). To learn a number of algorithm design techniques (such as divide-and-conquer). To reason (in a precise and formal way) about the efficiency and the correctness of algorithms. To reason (in a precise and formal way) about the efficiency and the correctness of algorithms.

CSC2105Mashiour Rahman4 General Algorithms are first solved on paper and later keyed in on the computer. Algorithms are first solved on paper and later keyed in on the computer. The most important thing is to be simple and precise. The most important thing is to be simple and precise. During lectures: During lectures: Interaction is welcome; ask questions. Interaction is welcome; ask questions. Additional explanations and examples if desired. Additional explanations and examples if desired. Speed up/slow down the progress. Speed up/slow down the progress.

CSC2105Mashiour Rahman5 Prerequisites Introduction to programming Introduction to programming Data types, operations Data types, operations Conditional statements Conditional statements Loops Loops Procedures and functions Procedures and functions C/C++/Java C/C++/Java Computer lab (edit, compile, execute) Computer lab (edit, compile, execute)

CSC2105Mashiour Rahman6 History Name: Persian mathematician Mohammed al- Khowarizmi, in Latin became Algorismus. Name: Persian mathematician Mohammed al- Khowarizmi, in Latin became Algorismus. First algorithm: Euclidean Algorithm, greatest common divisor, 400-300 B.C. First algorithm: Euclidean Algorithm, greatest common divisor, 400-300 B.C. 19 th century – Charles Babbage, Ada Lovelace. 19 th century – Charles Babbage, Ada Lovelace. 20 th century – Alan Turing, Alonzo Church, John von Neumann. 20 th century – Alan Turing, Alonzo Church, John von Neumann.

CSC2105Mashiour Rahman7 Data Structures and Algorithms Data structure Data structure Organization of data to solve the problem at hand. Organization of data to solve the problem at hand. Algorithm Algorithm Outline, the essence of a computational procedure, step-by-step instructions. Outline, the essence of a computational procedure, step-by-step instructions. Program Program Implementation of an algorithm in some programming language. Implementation of an algorithm in some programming language.

CSC2105Mashiour Rahman8 Overall Picture Using a computer to help solve problems. Using a computer to help solve problems. Precisely specify the problem. Precisely specify the problem. Designing programs Designing programs architecture architecture algorithms algorithms Writing programs Writing programs Verifying (testing) programs Verifying (testing) programs Data Structure and Algorithm Design Goals Implementation Goals Correctness Efficiency Robustness Adaptability Reusability

CSC2105Mashiour Rahman9 Overall Picture/2 This course is not about: This course is not about: Programming languages Programming languages Computer architecture Computer architecture Software architecture Software architecture Software design and implementation principles Software design and implementation principles Issues concerning small and large scale programming. Issues concerning small and large scale programming. We will only touch upon the theory of complexity and computability. We will only touch upon the theory of complexity and computability.

CSC2105Mashiour Rahman10 Algorithmic problem Infinite number of input instances satisfying the specification. For example: Infinite number of input instances satisfying the specification. For example: A sorted, non-decreasing sequence of natural numbers. The sequence is of non-zero, finite length: A sorted, non-decreasing sequence of natural numbers. The sequence is of non-zero, finite length: 1, 20, 908, 909, 100000, 1000000000. 1, 20, 908, 909, 100000, 1000000000. 3. 3. Specification of input ? Specification of output as a function of input

CSC2105Mashiour Rahman11 Algorithmic Solution Algorithm describes actions on the input instance. Algorithm describes actions on the input instance. There may be many correct algorithms for the same algorithmic problem. There may be many correct algorithms for the same algorithmic problem. Input instance, adhering to the specification Algorithm Output related to the input as required

CSC2105Mashiour Rahman12 Definition of an Algorithm An algorithm is a sequence of unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time. An algorithm is a sequence of unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time. Properties: Properties: Precision Precision Determinism Determinism Finiteness Finiteness Efficiency Correctness Generality

CSC2105Mashiour Rahman13 How to Develop an Algorithm Precisely define the problem. Precisely specify the input and output. Consider all cases. Precisely define the problem. Precisely specify the input and output. Consider all cases. Come up with a simple plan to solve the problem at hand. Come up with a simple plan to solve the problem at hand. The plan is language independent. The plan is language independent. The precise problem specification influences the plan. The precise problem specification influences the plan. Turn the plan into an implementation Turn the plan into an implementation The problem representation (data structure) influences the implementation. The problem representation (data structure) influences the implementation.

CSC2105Mashiour Rahman14 Preconditions, Postconditions It is important to specify the preconditions and the postconditions of algorithms: It is important to specify the preconditions and the postconditions of algorithms: INPUT: precise specifications of what the algorithm gets as an input. INPUT: precise specifications of what the algorithm gets as an input. OUTPUT: precise specifications of what the algorithm produces as an output, and how this relates to the input. The handling of special cases of the input should be described. OUTPUT: precise specifications of what the algorithm produces as an output, and how this relates to the input. The handling of special cases of the input should be described.

CSC2105Mashiour Rahman15 Analysis of Algorithms Efficiency: Efficiency: Running time Running time Space used Space used Efficiency as a function of the input size: Efficiency as a function of the input size: Number of data elements (numbers, points). Number of data elements (numbers, points). The number of bits of an input number. The number of bits of an input number.

CSC2105Mashiour Rahman16 The RAM model It is important to choose the level of detail. It is important to choose the level of detail. The RAM model: The RAM model: Instructions (each taking constant time), we usually choose one type of instruction as a characteristic operation that is counted: Instructions (each taking constant time), we usually choose one type of instruction as a characteristic operation that is counted: Arithmetic (add, subtract, multiply, etc.) Arithmetic (add, subtract, multiply, etc.) Data movement (assign) Data movement (assign) Control flow (branch, subroutine call, return) Control flow (branch, subroutine call, return) Comparison Comparison Data types – integers, characters, and floats Data types – integers, characters, and floats

CSC2105Mashiour Rahman17 Analysis of Insertion Sort for j  2 to n do key  A[j] // Insert A[j] into A[1..j-1] i  j-1 while i>0 and A[i]>key do A[i+1]  A[i] i-- A[i+1]:=key cost c 1 c 2 0 c 3 c 4 c 5 c 6 c 7 times n n-1 n-1 n-1 n-1 Time to compute the running time as a function of the input size (exact analysis). Time to compute the running time as a function of the input size (exact analysis).

CSC2105Mashiour Rahman18 Analysis of Insertion Sort/2 The running time of an algorithm is the sum of the running times of each state-ment. The running time of an algorithm is the sum of the running times of each state-ment. A statement with cost c that is executed n times contributes c*n to the running time. A statement with cost c that is executed n times contributes c*n to the running time. The total running time T(n) of insertion sort is The total running time T(n) of insertion sort is T(n) = c1*n + c2(n-1) + c3(n-1) + c4 + T(n) = c1*n + c2(n-1) + c3(n-1) + c4 + c5 + c6 + c7(n-1) c5 + c6 + c7(n-1)

CSC2105Mashiour Rahman19 Analysis of Insertion Sort/3 Often the performance depends on the details of the input (not only the length n). Often the performance depends on the details of the input (not only the length n). This is modeled by t j. This is modeled by t j. In the case of insertion sort the time t j depends on the original sorting of the input array. In the case of insertion sort the time t j depends on the original sorting of the input array.

CSC2105Mashiour Rahman20 Performance Analysis Often it is sufficient to count the number of iterations of the core (innermost) part. Often it is sufficient to count the number of iterations of the core (innermost) part. No distinction between comparisons, assignments, etc (that means roughly the same cost for all of them). No distinction between comparisons, assignments, etc (that means roughly the same cost for all of them). Gives precise enough results. Gives precise enough results. In some cases the cost of selected operations dominates all other costs. In some cases the cost of selected operations dominates all other costs. Disk I/O versus RAM operations. Disk I/O versus RAM operations. Database systems. Database systems.

CSC2105Mashiour Rahman21 Best/Worst/Average Case Analyzing insertion sort’s Analyzing insertion sort’s Best case: elements already sorted, t j =1, running time = n-1, i.e., linear time. Best case: elements already sorted, t j =1, running time = n-1, i.e., linear time. Worst case: elements are sorted in inverse order, t j =j-1, running time = (n 2 -n)/2, i.e., quadratic time. Worst case: elements are sorted in inverse order, t j =j-1, running time = (n 2 -n)/2, i.e., quadratic time. Average case: t j =j/2, running time = (n 2 +n-2)/4, i.e., quadratic time. Average case: t j =j/2, running time = (n 2 +n-2)/4, i.e., quadratic time.

CSC2105Mashiour Rahman22 Best/Worst/Average Case/3 For inputs of all sizes: For inputs of all sizes: 1n 2n 3n 4n 5n 6n Input instance size Running time 1 2 3 4 5 6 7 8 9 10 11 12 ….. best-case average-case worst-case

CSC2105Mashiour Rahman23 Best/Worst/Average Case/4 Worst case is usually used: Worst case is usually used: It is an upper-bound. It is an upper-bound. In certain application domains (e.g., air traffic control, surgery) knowing the worst-case time complexity is of crucial importance. In certain application domains (e.g., air traffic control, surgery) knowing the worst-case time complexity is of crucial importance. For some algorithms worst case occurs fairly often- For some algorithms worst case occurs fairly often- The average case is often as bad as the worst case. The average case is often as bad as the worst case. Finding the average case can be very difficult. Finding the average case can be very difficult.

CSC2105Mashiour Rahman24 Analysis of Linear Search INPUT: A[1..n] – a sorted array of integers, q – an integer. OUTPUT: an index j such that A[j] = q. NIL if  j (1  j  n): A[j]  q j := 1 while j  n and A[j]  q do j++ if j  n then return j else return NIL INPUT: A[1..n] – a sorted array of integers, q – an integer. OUTPUT: an index j such that A[j] = q. NIL if  j (1  j  n): A[j]  q j := 1 while j  n and A[j]  q do j++ if j  n then return j else return NIL Worst case running time: n Average case running time: n/2 Best case running time: 0

CSC2105Mashiour Rahman25 Binary Search INPUT: A[1..n] – a sorted array of (increasing) integers, q – an integer. OUTPUT: an index j such that A[j] = q. NIL, if  j (1  j  n): A[j]  q l := 1; r := n do m :=  (l+r)/2  if A[m] = q then return m else if A[m] > q then r := m-1 else l := m+1 while l <= r return NIL INPUT: A[1..n] – a sorted array of (increasing) integers, q – an integer. OUTPUT: an index j such that A[j] = q. NIL, if  j (1  j  n): A[j]  q l := 1; r := n do m :=  (l+r)/2  if A[m] = q then return m else if A[m] > q then r := m-1 else l := m+1 while l <= r return NIL Idea: Have a left and right bound. Elements to the right of r are bigger than the search element. Equivalent for l. In each step reduce the range of the search space by half.

CSC2105Mashiour Rahman26 Analysis of Binary Search How many times the loop is executed? How many times the loop is executed? With each execution the difference between l and r is cut in half. With each execution the difference between l and r is cut in half. Initially the difference is n. Initially the difference is n. The loop stops when the difference becomes 0 (less than 1). The loop stops when the difference becomes 0 (less than 1). How many times do you have to cut n in half to get 0? How many times do you have to cut n in half to get 0? log n – better than the brute-force approach of linear search (n). log n – better than the brute-force approach of linear search (n).

CSC2105Mashiour Rahman27 Linear Search vs Binary Search Costs of linear search: n Costs of linear search: n Costs of binary search: log(n) Costs of binary search: log(n) Should we care? Should we care? Phone book with 200’000 entries: Phone book with 200’000 entries: n = 200’000 n = 200’000 log n = log 200’000 = 17.6 log n = log 200’000 = 17.6

CSC2105Mashiour Rahman28 Asymptotic Analysis Goal: to simplify the analysis of the running time by getting rid of details, which are affected by specific implementation and hardware Goal: to simplify the analysis of the running time by getting rid of details, which are affected by specific implementation and hardware “rounding” of numbers: 1,000,001  1,000,000 “rounding” of numbers: 1,000,001  1,000,000 “rounding” of functions: 3n 2  n 2 “rounding” of functions: 3n 2  n 2 Capturing the essence: how the running time of an algorithm increases with the size of the input in the limit. Capturing the essence: how the running time of an algorithm increases with the size of the input in the limit. Asymptotically more efficient algorithms are best for all but small inputs Asymptotically more efficient algorithms are best for all but small inputs

CSC2105Mashiour Rahman29 Asymptotic Notation The “big-Oh” O-Notation The “big-Oh” O-Notation asymptotic upper bound asymptotic upper bound f(n) = O(g(n)), if there exists constants c>0 and n 0 >0, s.t. f(n)  c g(n) for n  n 0 f(n) = O(g(n)), if there exists constants c>0 and n 0 >0, s.t. f(n)  c g(n) for n  n 0 f(n) and g(n) are functions over non-negative integers f(n) and g(n) are functions over non-negative integers Used for worst-case analysis Used for worst-case analysis Input Size Running Time

CSC2105Mashiour Rahman30 The “big-Omega”  Notation The “big-Omega”  Notation asymptotic lower bound asymptotic lower bound f(n) =  (g(n)) if there exists constants c>0 and n 0 >0, s.t. c g(n)  f(n) for n  n 0 f(n) =  (g(n)) if there exists constants c>0 and n 0 >0, s.t. c g(n)  f(n) for n  n 0 Used to describe best-case running times or lower bounds of algorithmic problems. Used to describe best-case running times or lower bounds of algorithmic problems. E.g., lower-bound of searching in an unsorted array is  (n). E.g., lower-bound of searching in an unsorted array is  (n). Input Size Running Time Asymptotic Notation/2

CSC2105Mashiour Rahman31 Asymptotic Notation/3 Simple Rule: Drop lower order terms and constant factors. Simple Rule: Drop lower order terms and constant factors. 50 n log n is O(n log n) 50 n log n is O(n log n) 7n - 3 is O(n) 7n - 3 is O(n) 8n 2 log n + 5n 2 + n is O(n 2 log n) 8n 2 log n + 5n 2 + n is O(n 2 log n) Note: Although (50 n log n) is O(n 5 ), it is expected that an approximation is of the smallest possible order. Note: Although (50 n log n) is O(n 5 ), it is expected that an approximation is of the smallest possible order.

CSC2105Mashiour Rahman32 The “big-Theta”  Notation The “big-Theta”  Notation asymptoticly tight bound asymptoticly tight bound f(n) =  (g(n)) if there exists constants c 1 >0, c 2 >0, and n 0 >0, s.t. for n  n 0 c 1 g(n)  f(n)  c 2 g(n) f(n) =  (g(n)) if there exists constants c 1 >0, c 2 >0, and n 0 >0, s.t. for n  n 0 c 1 g(n)  f(n)  c 2 g(n) f(n) =  (g(n)) if and only if f(n) =  (g(n)) and f(n) =  (g(n)) f(n) =  (g(n)) if and only if f(n) =  (g(n)) and f(n) =  (g(n)) O(f(n)) is often abused instead of  (f(n)) O(f(n)) is often abused instead of  (f(n)) Input Size Running Time Asymptotic Notation/4

CSC2105Mashiour Rahman33 A Quick Math Refresher Arithmetic progression Arithmetic progression Geometric progression Geometric progression given an integer n 0 and a real number 0<a  1 given an integer n 0 and a real number 0<a  1 geometric progressions exhibit exponential growth geometric progressions exhibit exponential growth

CSC2105Mashiour Rahman34 Miscellaneous Manipulating logarithms: Manipulating logarithms: a log b = log b / log a a log b = log b / log a log a b = b log a log a b = b log a Manipulating summations: Manipulating summations:

CSC2105Mashiour Rahman35 Summations The running time of insertion sort is determined by a nested loop. The running time of insertion sort is determined by a nested loop. Nested loops correspond to summations: Nested loops correspond to summations: for j := 2 to n key := A[j] i := j-1 while i>0 and A[i]>key A[i+1] := A[i] i := i-1 A[i+1] := key for j := 2 to n key := A[j] i := j-1 while i>0 and A[i]>key A[i+1] := A[i] i := i-1 A[i+1] := key

CSC2105Mashiour Rahman36 Proof by Induction We want to show that property P is true for all integers n  n 0. We want to show that property P is true for all integers n  n 0. Basis: prove that P is true for n 0. Basis: prove that P is true for n 0. Inductive step: prove that if P is true for all k such that n 0  k  n – 1 then P is also true for n. Inductive step: prove that if P is true for all k such that n 0  k  n – 1 then P is also true for n. Example Example Basis Basis

CSC2105Mashiour Rahman37 Proof by Induction/2 Inductive Step Inductive Step

CSC2105Mashiour Rahman38 Correctness of Algorithms An algorithm is correct if for any legal input it terminates and produces the desired output. An algorithm is correct if for any legal input it terminates and produces the desired output. Automatic proof of correctness is not possible. Automatic proof of correctness is not possible. There are practical techniques and rigorous formalisms that help to reason about the correctness of (parts of) algorithms. There are practical techniques and rigorous formalisms that help to reason about the correctness of (parts of) algorithms.

CSC2105Mashiour Rahman39 Partial and Total Correctness Partial correctness Partial correctness Any legal input Algorithm Output IF this point is reached, THEN this is the desired output Total correctness Any legal input Algorithm Output INDEED this point is reached, AND this is the desired output

CSC2105Mashiour Rahman40 Assertions To prove partial correctness we associate a number of assertions (statements about the state of the execution) with specific checkpoints in the algorithm. To prove partial correctness we associate a number of assertions (statements about the state of the execution) with specific checkpoints in the algorithm. E.g., A[1], …, A[j] form an increasing sequence E.g., A[1], …, A[j] form an increasing sequence Preconditions – assertions that must be valid before the execution of an algorithm or a subroutine (INPUT). Preconditions – assertions that must be valid before the execution of an algorithm or a subroutine (INPUT). Postconditions – assertions that must be valid after the execution of an algorithm or a subroutine (OUTPUT). Postconditions – assertions that must be valid after the execution of an algorithm or a subroutine (OUTPUT).

CSC2105Mashiour Rahman41 Pre/post-conditions Example: Example: Write a pseudocode algorithm to find the two smallest numbers in a sequence of numbers (given as an array). Write a pseudocode algorithm to find the two smallest numbers in a sequence of numbers (given as an array). INPUT: an array of integers A[1..n], n > 0 INPUT: an array of integers A[1..n], n > 0 OUTPUT: (m1, m2) such that OUTPUT: (m1, m2) such that m1<m2 and for each i  [1..n]: m1  A[i] and, if A[i]  m1, then m2  A[i]. m1<m2 and for each i  [1..n]: m1  A[i] and, if A[i]  m1, then m2  A[i]. m2 = m1 = A[1] if  j,i  [1..n]: A[i]=A[j] m2 = m1 = A[1] if  j,i  [1..n]: A[i]=A[j]

CSC2105Mashiour Rahman42 Loop Invariants Invariants: assertions that are valid any time they are reached (many times during the execution of an algorithm, e.g., in loops) Invariants: assertions that are valid any time they are reached (many times during the execution of an algorithm, e.g., in loops) We must show three things about loop invariants: We must show three things about loop invariants: Initialization: it is true prior to the first iteration. Initialization: it is true prior to the first iteration. Maintenance: if it is true before an iteration, then it is true after the iteration. Maintenance: if it is true before an iteration, then it is true after the iteration. Termination: when a loop terminates the invariant gives a useful property to show the correctness of the algorithm Termination: when a loop terminates the invariant gives a useful property to show the correctness of the algorithm

CSC2105Mashiour Rahman43 Example: Binary Search/1 Initialization: l = 1, r = n the invariant holds because there are no elements to the left of l or to the right of r. Initialization: l = 1, r = n the invariant holds because there are no elements to the left of l or to the right of r. l=1 yields  j,i  [1..0]: A[i]<q this holds because [1..0] is empty l=1 yields  j,i  [1..0]: A[i]<q this holds because [1..0] is empty r=n yields  j,i  [n+1..n]: A[i]>q this holds because [n+1..n] is empty r=n yields  j,i  [n+1..n]: A[i]>q this holds because [n+1..n] is empty l := 1 r := n do m :=  (l+r)/2  if A[m] = q then return m else if A[m] > q then r := m-1 else l := m+1 while l <= r return NIL l := 1 r := n do m :=  (l+r)/2  if A[m] = q then return m else if A[m] > q then r := m-1 else l := m+1 while l <= r return NIL We want to show that q is not in A if NIL is returned. Invariant:  i  [1..l-1]: A[i] q (Ib)

CSC2105Mashiour Rahman44 Example: Binary Search/2 Maintenance: l, r, m =  (l+r)/2  Maintenance: l, r, m =  (l+r)/2  A[m]!=q & A[m]>q, r=m-1, A sorted implies  k  [r+1..n]: A[k]>q (Ib) A[m]!=q & A[m]>q, r=m-1, A sorted implies  k  [r+1..n]: A[k]>q (Ib) A[m]!=q & A[m]<q, l=m+1, A sorted implies  k  [1..l-1]: A[k]<q (Ib) A[m]!=q & A[m]<q, l=m+1, A sorted implies  k  [1..l-1]: A[k]<q (Ib) Invariant:  i  [1..l-1]: A[i] q (Ib) l := 1 r := n do m :=  (l+r)/2  if A[m] = q then return m else if A[m] > q then r := m-1 else l := m+1 while l <= r return NIL l := 1 r := n do m :=  (l+r)/2  if A[m] = q then return m else if A[m] > q then r := m-1 else l := m+1 while l <= r return NIL

CSC2105Mashiour Rahman45 Example: Binary Search/3 Termination: l, r, l<=r Termination: l, r, l<=r Two cases: Two cases: l:=m+1 we get  (l+r)/2  +1 > l l:=m+1 we get  (l+r)/2  +1 > l r:=m-1 we get  (l+r)/2  -1 < r r:=m-1 we get  (l+r)/2  -1 < r The range gets smaller during each iteration and the loop will terminate when l<=r no longer holds. The range gets smaller during each iteration and the loop will terminate when l<=r no longer holds. Invariant:  i  [1..l-1]: A[i] q (Ib) l := 1 r := n do m :=  (l+r)/2  if A[m] = q then return m else if A[m] > q then r := m-1 else l := m+1 while l <= r return NIL l := 1 r := n do m :=  (l+r)/2  if A[m] = q then return m else if A[m] > q then r := m-1 else l := m+1 while l <= r return NIL

CSC2105Mashiour Rahman46 Example: Insertion Sort/1 Invariant: Invariant: outside while loop outside while loop A[1...j-1] is sorted A[1...j-1] is sorted A[1...j-1]  A orig A[1...j-1]  A orig inside while loop: inside while loop: A[1...i], key, A[i+1…j-1] A[1...i], key, A[i+1…j-1] A[1...i] is sorted A[1...i] is sorted A[i+1…j-1] is sorted A[i+1…j-1] is sorted A[k] > key, i+1 key, i+1<=k<=j-1 for j := 2 to n do key := A[j] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-- A[i+1] := key for j := 2 to n do key := A[j] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-- A[i+1] := key

CSC2105Mashiour Rahman47 Example: Insertion Sort/2 outside while loop outside while loop A[1...j-1] is sorted A[1...j-1] is sorted A[1...j-1]  A orig A[1...j-1]  A orig inside while loop: inside while loop: A[1...i], key, A[i+1…j-1] A[1...i], key, A[i+1…j-1] A[1...i] is sorted A[1...i] is sorted A[i+1…j-1] is sorted A[i+1…j-1] is sorted A[k] > key, i+1 key, i+1<=k<=j-1 Initialization: j=2: the invariant holds, A[1…1] is trivially sorted. i=j-1: A[1...j-1], key, A[j…j-1] where key=A[j] A[j…j-1] is empty (and thus trivially sorted) A[1…j-1] is sorted (invariant of outer loop) for j := 2 to length(A)do key := A[j] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-- A[i+1] := key for j := 2 to length(A)do key := A[j] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-- A[i+1] := key

CSC2105Mashiour Rahman48 Example: Insertion Sort/2 outside while loop outside while loop A[1…j-1] is sorted A[1…j-1] is sorted A[1...j-1]  A orig A[1...j-1]  A orig inside while loop: inside while loop: A[1...i], key, A[i+1…j-1] A[1...i], key, A[i+1…j-1] A[1...i] is sorted A[1...i] is sorted A[i+1…j-1] is sorted A[i+1…j-1] is sorted A[k] > key, i+1 key, i+1<=k<=j-1 Maintenance: (A[1…j-1] sorted) + (insert A[j])  A[1…j] sorted). A[1...i-1], key, A[i,i+1…j-1] satisfies conditions because of condition A[i]>key and A[1...j-1] being sorted. for j := 2 to length(A)do key := A[j] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-- A[i+1] := key for j := 2 to length(A)do key := A[j] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-- A[i+1] := key

CSC2105Mashiour Rahman49 Example: Insertion Sort/2 outside while loop outside while loop A[1…j-1] is sorted A[1…j-1] is sorted A[1...j-1]  A orig A[1...j-1]  A orig inside while loop: inside while loop: A[1...i], key, A[i+1…j-1] A[1...i], key, A[i+1…j-1] A[1...i] is sorted A[1...i] is sorted A[i+1…j-1] is sorted A[i+1…j-1] is sorted A[k] > key, i+1 key, i+1<=k<=j-1 Termination: main loop, j=n+1: A[1…n] sorted. A[i]  key: (A[1...i], key, A[i+1…j-1]) = A[1…j-1] is sorted i=0: (key, A[1…j-1]) = A[1…j-1] is sorted. for j := 2 to length(A)do key := A[j] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-- A[i+1] := key for j := 2 to length(A)do key := A[j] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-- A[i+1] := key

CSC2105Mashiour Rahman50 Recursion An object is recursive if An object is recursive if it contains itself as part of it, or it contains itself as part of it, or it is defined in terms of itself. it is defined in terms of itself. Factorial: n! Factorial: n! How do you compute 10!? How do you compute 10!? n! = 1 * 2 * 3 *...* n n! = 1 * 2 * 3 *...* n n! = n * (n-1)! n! = n * (n-1)!

CSC2105Mashiour Rahman51 Factorial Function Pseudocode of factorial: Pseudocode of factorial: fac1 INPUT: n – a natural number. OUTPUT: n! (factorial of n) fac1(n) if n < 2 then return 1 if n < 2 then return 1 else return n * fac1(n-1) else return n * fac1(n-1)fac1 INPUT: n – a natural number. OUTPUT: n! (factorial of n) fac1(n) if n < 2 then return 1 if n < 2 then return 1 else return n * fac1(n-1) else return n * fac1(n-1) A recursive procedure includes a A recursive procedure includes a Termination condition (determines when and how to stop the recursion). Termination condition (determines when and how to stop the recursion). One (or more) recursive calls. One (or more) recursive calls.

CSC2105Mashiour Rahman52 Tracing the Execution fac(3) fac(1) 1 fac(2) 2 * fac(1) 1 2 6 3 * fac(2)

CSC2105Mashiour Rahman53 Bookkeeping fac(5) 5*fac(4) fac(4) 4*fac(3) fac(3) 3*fac(2) fac(2) 2*fac(1) fac(1) 1 1 2 6 24 120 The computer maintains an activation stack for active procedure calls (-> compiler construction). Example for fac(5). The computer maintains an activation stack for active procedure calls (-> compiler construction). Example for fac(5).

CSC2105Mashiour Rahman54 Variants of Factorial fac2 INPUT: n – a natural number. OUTPUT: n! (factorial of n) fac2(n) if n  2 then return 1 if n  2 then return 1 return n * fac2(n-1) return n * fac2(n-1)fac2 INPUT: n – a natural number. OUTPUT: n! (factorial of n) fac2(n) if n  2 then return 1 if n  2 then return 1 return n * fac2(n-1) return n * fac2(n-1) fac3 INPUT: n – a natural number. OUTPUT: n! (factorial of n) fac3(n) return n * fac3(n-1) return n * fac3(n-1) if n  2 then return 1 if n  2 then return 1fac3 INPUT: n – a natural number. OUTPUT: n! (factorial of n) fac3(n) return n * fac3(n-1) return n * fac3(n-1) if n  2 then return 1 if n  2 then return 1

CSC2105Mashiour Rahman55 Analysis of the solutions fac2 is correct fac2 is correct The return statement in the if clause terminates the function and, thus, the entire recursion. The return statement in the if clause terminates the function and, thus, the entire recursion. fac3 is incorrect fac3 is incorrect Infinite recursion. The termination condition is never reached. Infinite recursion. The termination condition is never reached.

CSC2105Mashiour Rahman56 Fibonacci Numbers Definition Definition fib(1) = 1 fib(1) = 1 fib(2) = 1 fib(2) = 1 fib(n) = fib(n-1) + fib(n-2), n>2 fib(n) = fib(n-1) + fib(n-2), n>2 Numbers in the series: Numbers in the series: 1, 1, 2, 3, 5, 8, 13, 21, 34,... 1, 1, 2, 3, 5, 8, 13, 21, 34,...

CSC2105Mashiour Rahman57 Fibonacci Implementation fib INPUT: n – a natural number larger than 0. OUTPUT: fib(n), the nth Fibonacci number. fib(n) if n  2 then return 1 if n  2 then return 1 else return fib(n-1) + fib(n-2) else return fib(n-1) + fib(n-2)fib INPUT: n – a natural number larger than 0. OUTPUT: fib(n), the nth Fibonacci number. fib(n) if n  2 then return 1 if n  2 then return 1 else return fib(n-1) + fib(n-2) else return fib(n-1) + fib(n-2) Multiple recursive calls are possible. Multiple recursive calls are possible.

CSC2105Mashiour Rahman58 Fibonacci Implementation/2 int fib(int i) { if (i <= 2) { return 1;} if (i <= 2) { return 1;} else { return fib(i-1) + fib(i-2); } else { return fib(i-1) + fib(i-2); }} int main() { printf(“Fibonacci of 5 is %d\n", fac(5)); printf(“Fibonacci of 5 is %d\n", fac(5));} int fib(int i) { if (i <= 2) { return 1;} if (i <= 2) { return 1;} else { return fib(i-1) + fib(i-2); } else { return fib(i-1) + fib(i-2); }} int main() { printf(“Fibonacci of 5 is %d\n", fac(5)); printf(“Fibonacci of 5 is %d\n", fac(5));}

CSC2105Mashiour Rahman59 Tracing fib(4) fib(4) fib(3) + fib(2) fib(2) 1 fib(3) fib(2) + fib(1) 1 1 3 fib(2) 1 fib(1) 1 2 1

CSC2105Mashiour Rahman60 Bookkeeping fib(4) fib(3) + fib(2) fib(2) 1 fib(1) 1 2 3 Activation stack for fib(4). Activation stack for fib(4). fib(3) fib(2) + fib(1) 1 1 1 fib(2) 1

CSC2105Mashiour Rahman61 Mutual Recursion Recursion does not always occur because a procedure calls itself. Recursion does not always occur because a procedure calls itself. Mutual recursion occurs if two procedures call each other. Mutual recursion occurs if two procedures call each other. B A

CSC2105Mashiour Rahman62 Mutual Recursion Example Problem: Determine whether a natural number is even. Problem: Determine whether a natural number is even. Definition of even: Definition of even: 0 is even 0 is even N is odd if N-1 is even N is odd if N-1 is even N is even if N-1 is odd N is even if N-1 is odd

CSC2105Mashiour Rahman63 Implementation of even Can it be used to determine whether a number is odd? Can it be used to determine whether a number is odd? even INPUT: n – a natural number. OUTPUT: true if n is even; false otherwise odd(n) if n = 0 then return TRUE if n = 0 then return TRUE return !even(n-1) return !even(n-1)even(n) if n = 0 then return TRUE if n = 0 then return TRUE else return !odd(n-1) else return !odd(n-1)even INPUT: n – a natural number. OUTPUT: true if n is even; false otherwise odd(n) if n = 0 then return TRUE if n = 0 then return TRUE return !even(n-1) return !even(n-1)even(n) if n = 0 then return TRUE if n = 0 then return TRUE else return !odd(n-1) else return !odd(n-1)

CSC2105Mashiour Rahman64 Is Recursion Necessary? Theory: You can always resort to iteration and explicitly maintain a recursion stack. Theory: You can always resort to iteration and explicitly maintain a recursion stack. Practice: Recursion is elegant and in some cases the best solution by far. Practice: Recursion is elegant and in some cases the best solution by far. In the previous examples recursion was never appropriate since there exist simple iterative solutions. In the previous examples recursion was never appropriate since there exist simple iterative solutions. Recursion is more expensive than corresponding iterative solutions since bookkeeping is necessary. Recursion is more expensive than corresponding iterative solutions since bookkeeping is necessary.

CSC2105Mashiour Rahman65 Sorting Sorting is a classical and important algorithmic problem. Sorting is a classical and important algorithmic problem. We look at sorting arrays (in contrast to files, which restrict random access). We look at sorting arrays (in contrast to files, which restrict random access). A key constraint is the efficient management of the space A key constraint is the efficient management of the space In-place sorting algorithms In-place sorting algorithms The efficiency comparison is based on the number of comparisons (C) and the number of movements (M). The efficiency comparison is based on the number of comparisons (C) and the number of movements (M).

CSC2105Mashiour Rahman66 Sorting Simple sorting methods use roughly n * n comparisons Simple sorting methods use roughly n * n comparisons Insertion sort Insertion sort Selection sort Selection sort Bubble sort Bubble sort Fast sorting methods use roughly n * log n comparisons. Fast sorting methods use roughly n * log n comparisons. Merge sort Merge sort Heap sort Heap sort Quicksort Quicksort

CSC2105Mashiour Rahman67 Sort Example 2: Sorting INPUT sequence of n numbers a 1, a 2, a 3,….,a n b 1,b 2,b 3,….,b n OUTPUT a permutation of the input sequence of numbers 2 5 4 10 7 2 4 5 7 10 Correctness (requirements for the output) For any given input the algorithm halts with the output: b 1 < b 2 < b 3 < …. < b n b 1, b 2, b 3, …., b n is a permutation of a 1, a 2, a 3,….,a n Correctness (requirements for the output) For any given input the algorithm halts with the output: b 1 < b 2 < b 3 < …. < b n b 1, b 2, b 3, …., b n is a permutation of a 1, a 2, a 3,….,a n

CSC2105Mashiour Rahman68 Insertion Sort A 1nj 368497251 i Strategy Start with one sorted card. Insert an unsorted card at the correct position in the sorted part. Continue until all unsorted cards are inserted/sorted. Strategy Start with one sorted card. Insert an unsorted card at the correct position in the sorted part. Continue until all unsorted cards are inserted/sorted. A44 55 12 42 94 18 06 67 44 55 12 42 94 18 06 67 12 44 55 42 94 18 06 67 12 42 44 55 94 18 06 67 12 18 42 44 55 94 06 67 06 12 18 42 44 55 94 67 06 12 18 42 44 55 67 94

CSC2105Mashiour Rahman69 Insertion Sort/2 The number of comparisons during the jth iteration is The number of comparisons during the jth iteration is at least 1: Cmin = = n-1 at most j-1: Cmax = = (n*n-n-2)/2 INPUT: A[1..n] – an array of integers OUTPUT: a permutation of A such that A[1]  A[2]  …  A[n] for j := 2 to n do key  = A[j] i := j-1 while i > 0 and A[i] > key do A[i+1] := A[i]; i-- A[j+1] := key INPUT: A[1..n] – an array of integers OUTPUT: a permutation of A such that A[1]  A[2]  …  A[n] for j := 2 to n do key  = A[j] i := j-1 while i > 0 and A[i] > key do A[i+1] := A[i]; i-- A[j+1] := key

CSC2105Mashiour Rahman70 Insertion Sort/3 The number of comparisons during the jth iteration is: The number of comparisons during the jth iteration is: j/2 in average: Cavg = = (n*n+n–2)/4 j/2 in average: Cavg = = (n*n+n–2)/4 The number of movements is Ci+1: The number of movements is Ci+1: Mmin = = 2*(n-1), Mmin = = 2*(n-1), Mavg = = (n*n+5n-6)/4 Mavg = = (n*n+5n-6)/4 Mmax = = (n*n+n-2)/2 Mmax = = (n*n+n-2)/2

CSC2105Mashiour Rahman71 Selection Sort A 1nj 134257896 i Strategy Start empty handed. Enlarge the sorted part by switching the first element of the unsorted part with the smallest element of the unsorted part. Continue until the unsorted part consists of one element only. Strategy Start empty handed. Enlarge the sorted part by switching the first element of the unsorted part with the smallest element of the unsorted part. Continue until the unsorted part consists of one element only. A 44 55 12 42 94 18 06 67 06 55 12 42 94 18 44 67 06 12 55 42 94 18 44 67 06 12 18 42 94 55 44 67 06 12 18 42 44 55 94 67 06 12 18 42 44 55 67 94

CSC2105Mashiour Rahman72 Selection Sort/2 The number of comparisons is indepen-dent of the original ordering (this is less natural behavior than insertion sort): The number of comparisons is indepen-dent of the original ordering (this is less natural behavior than insertion sort): C = = (n*n-n)/2 C = = (n*n-n)/2 INPUT: A[1..n] – an array of integers OUTPUT: a permutation of A such that A[1]  A[2]  …  A[n] for j := 1 to n-1 do key := A[j]; ptr := j for i := j+1 to n do if A[i] < key then ptr := i; key := A[i]; A[ptr] := A[j]; A[j] := key INPUT: A[1..n] – an array of integers OUTPUT: a permutation of A such that A[1]  A[2]  …  A[n] for j := 1 to n-1 do key := A[j]; ptr := j for i := j+1 to n do if A[i] < key then ptr := i; key := A[i]; A[ptr] := A[j]; A[j] := key

CSC2105Mashiour Rahman73 Selection Sort/3 The number of movements is: The number of movements is: Mmin = = 3*(n-1) Mmin = = 3*(n-1) Mmax = = (n*n–n)/4 + 3*(n-1) Mmax = = (n*n–n)/4 + 3*(n-1)

CSC2105Mashiour Rahman74 Bubble Sort A 1nj 134257986 i Strategy Start from the back and compare pairs of adjacent elements. Switch the elements if the larger comes before the smaller. In each step the smallest element of the unsorted part is moved to the beginning of the unsorted part and the sorted part grows by one. Strategy Start from the back and compare pairs of adjacent elements. Switch the elements if the larger comes before the smaller. In each step the smallest element of the unsorted part is moved to the beginning of the unsorted part and the sorted part grows by one. A 44 55 12 42 94 18 06 67 06 44 55 12 42 94 18 67 06 12 44 55 18 42 94 67 06 12 18 44 55 42 67 94 06 12 18 42 44 55 67 94

CSC2105Mashiour Rahman75 Bubble Sort/2 The number of comparisons is indepen-dent of the original ordering: The number of comparisons is indepen-dent of the original ordering: C = = (n*n-n)/2 C = = (n*n-n)/2 INPUT: A[1..n] – an array of integers OUTPUT: a permutation of A such that A[1]  A[2]  …  A[n] for j := 2 to n do for i := n to j do if A[i-1] < A[i] then key := A[i-1]; A[i-1] := A[i]; A[i]:=key INPUT: A[1..n] – an array of integers OUTPUT: a permutation of A such that A[1]  A[2]  …  A[n] for j := 2 to n do for i := n to j do if A[i-1] < A[i] then key := A[i-1]; A[i-1] := A[i]; A[i]:=key

CSC2105Mashiour Rahman76 Bubble Sort/3 The number of movements is: The number of movements is: Mmin = 0 Mmin = 0 Mmax = = 3*n*(n-1)/2 Mmax = = 3*n*(n-1)/2 Mavg = = 3*n*(n-1)/4 Mavg = = 3*n*(n-1)/4

CSC2105Mashiour Rahman77

CSC2105Mashiour Rahman81 The divide- and- conquer design paradigm 1. Divide the problem (instance) into subproblems. 2. Conquer the subproblems by solving them recursively. 3. Combine subproblem solutions.

CSC2105Mashiour Rahman82 The divide- and- conquer design paradigm Example: merge sort 1. Divide: Trivial. 2. Conquer: Recursively sort 2 subarrays. 3. Combine: Linear- time merge. Recurrence for merge sort T( n) = 2 T( n/ 2) T( n) = 2 T( n/ 2) # subproblemssubproblem size # subproblemssubproblem size + O( n) + O( n) work dividing and combining work dividing and combining

CSC2105Mashiour Rahman83 The divide- and- conquer design paradigm Binary search Find an element in a sorted array: 1. Divide: Check middle element. 2. Conquer: Recursively search 1 subarray. 3. Combine: Trivial. Recurrence for binary search T(n)= 1T(n/2)+ Θ(1) # subproblemssubproblem work # subproblemssubproblem work size dividing and size dividing and combining combining

CSC2105Mashiour Rahman84 The divide- and- conquer design paradigm

CSC2105Mashiour Rahman85 The divide- and- conquer design paradigm Conclusion Divide and conquer is just one of several powerful techniques for algorithm design. Divide and conquer is just one of several powerful techniques for algorithm design. Divide- and- conquer algorithms can be analyzed using recurrences and the master method (so practice this math). Divide- and- conquer algorithms can be analyzed using recurrences and the master method (so practice this math). Can lead to more efficient algorithms Can lead to more efficient algorithms

CSC2105: Algorithms Mashiour Rahman American International University Bangladesh.

Similar presentations

Presentation on theme: "CSC2105: Algorithms Mashiour Rahman American International University Bangladesh."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

CSC2105: Algorithms Mashiour Rahman American International University Bangladesh.

Similar presentations

Presentation on theme: "CSC2105: Algorithms Mashiour Rahman American International University Bangladesh."— Presentation transcript:

Similar presentations

About project

Feedback