MS 101: Algorithms Instructor Neelima Gupta

Table Of Contents Introduction to some tools to designing algorithms through Sorting Iterative Divide and Conquer

Iterative Algorithms: Insertion Sort – an example x 1,x 2, , x i-1,x i, …,x n For I = 2 to n Insert the ith element x i in the partially sorted list x 1,x 2, , x i-1. (at r th position)

An Example: Insertion Sort

An Example: Insertion Sort

An Example: Insertion Sort

An Example: Insertion Sort

An Example: Insertion Sort

An Example: Insertion Sort

An Example: Insertion Sort Done!

Analysis: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } How many times will this loop execute?

Insertion Sort Statement Effort InsertionSort(A, n) { for i = 2 to n { c 1 n – 1) key = A[i] c 2 (n-1) j = i - 1; c 3 (n-1) while (j > 0) and (A[j] > key) { c 4 T A[j+1] = A[j] c 5 (T-(n-1)) j = j - 1 c 6 (T-(n-1)) } 0 A[j+1] = key c 7 (n-1) } 0 } T = t 2 + t 3 + … + t n where t i is number of while expression evaluations for the i th for loop iteration

Analyzing Insertion Sort T(n) = c 1 n + c 2 (n-1) + c 3 (n-1) + c 4 T + c 5 (T - (n-1)) + c 6 (T - (n-1)) + c 7 (n-1) = c 8 T + c 9 n + c 10 What can T be? –Best case -- inner loop body never executed t i = 1  T(n) is a linear function –Worst case -- inner loop body executed for all previous elements t i = i  T(n) is a quadratic function –Average case ???

Analysis Simplifications –Ignore actual and abstract statement costs –Order of growth is the interesting measure: Highest-order term is what counts –Remember, we are doing asymptotic analysis –As the input size grows larger it is the high order term that dominates

Insertion Sort Is O(n 2 ) Proof –Suppose runtime is an 2 + bn + c If any of a, b, and c are less than 0 replace the constant with its absolute value –an 2 + bn + c  (a + b + c)n 2 + (a + b + c)n + (a + b + c) –  3(a + b + c)n 2 for n  1 –Let c’ = 3(a + b + c) and let n 0 = 1 Question –Is InsertionSort O(n 3 )? –Is InsertionSort O(n)?

Big O Fact A polynomial of degree k is O(n k ) Proof: –Suppose f(n) = b k n k + b k-1 n k-1 + … + b 1 n + b 0 Let a i = | b i | –f(n)  a k n k + a k-1 n k-1 + … + a 1 n + a 0

Lower Bound Notation We say InsertionSort’s run time is  (n) Proof: –Suppose run time is an + b Assume a and b are positive (what if b is negative?) –an  an + b

Analysis of Algorithms Before we move ahead, let us define the notion of analysis of algorithms more formally

Input Size Time and space complexity –This is generally a function of the input size –How we characterize input size depends: Sorting: number of input items Multiplication: total number of bits Graph algorithms: number of nodes & edges Etc

Cases to Analyze algorithms Worst case –Provides an upper bound on running time : The big O bound –An absolute guarantee Average case –Provides the expected running time –Very useful, but treat with care: what is “average”? Random (equally likely) inputs Real-life inputs Best Case –Minimum time the algorithm will take on any input: Omega bound. Will come back to this again later.

Avg. case Analysis of Insertion Sort Let x i be the random variable which represents the number of comparisons required to insert i th element of the input array in the sorted sub array of first i-1 elements. X : x 1,x 2, …,x n E(X) = Σx i p(x i ) where E(X) is expected value X And, p(x i ) is probability of inserting x i in the r th position 1≤r≤i

Avg. case Analysis of Insertion Sort Let x i be the random variable which represents the i th element of the input array that has to be inserted in the sorted sub array of sorted first i-1 elements. X : x 1,x 2, …,x n E(X) = Σx r p(x r ) where E(X) is expected value X And, p(x r ) is probability of inserting x i in the r th position 1≤r≤i

x 1,x 2, , x i-1,x i, …,x n How many comparisons it makes to insert i th element in r th position? (at r th position)

Position# of Comparisions i1 i-12 i i-1 1i-1 Note: Here, both position 2 and 1 have # of Comparisions equal to i-1. Why? Because to insert element at position 2 we have to compare with previously first element. and after that comparison we know which of them come first and which at second.

For n number of elements, expected time taken is, T = n Σ i=2 (1/i) { i-1 Σ k=1 k + (i-1) } where 1/i is the probability to insert at r th position in the i possible positions. E(X 1 + X X n ) = n Σ i=1 E(X i ) Where,Xi is expected value of inserting X i element. T = (n-1)(n-4) / 4 Therefore average case of insertion sort takes Θ(n 2 )

Divide and Conquer: Merge Sort – an example T(n) = 2T(n/2) +  (n) when n > 1 T(1) = 1 What is T(n)? MergeSort(A, left, right) {T(n) if (left < right) {  (1) mid = floor((left + right) / 2);  (1) MergeSort(A, left, mid); T(n/2) MergeSort(A, mid+1, right); T(n/2) Merge(A, left, mid, right);  (n) }

What is the –Worst case –Average Case –Best Case for Merge Sort?

Merge Sort Vs Insertion Sort What is the advantage of merge sort? What is the advantage of insertion sort?

Merge Sort Vs Insertion Sort contd.. Merge Sort is faster but does not take advantage if the list is already partially sorted. Insertion Sort takes advantage if the list is already partially sorted.

Quick Sort Algorithm and Analysis to be skipped for MCS 101 Skip slides upto 21.

A Recursive Sorting Algorithm void quickSort(int a[], int first, int last) { if(first >= last) return; // Done: we have an empty array // The difficult algorithm is in partition int position = partition ( a, first, last ); // Recursively Quicksort left, then right quickSort(a, first, position-1); // sort left quickSort(a, position+1, last); // sort right }

Partitioning Pick one item from the array--call it the pivot Partition the items in the array around the pivot so all elements to the left are  to the pivot and all elements to the right are greater than the pivot Use recursion to sort the two partitions pivot partition: items > pivot partition 1: items  pivot

Before and After –Let's sort integers Pick the leftmost one (27) as the pivot value The array before call to partition(a, 0, n-1) Rearranged array after partition is done pivotitems < pivotitems > pivot

Analyzing Quicksort Worst Case? –Partition is always unbalanced Worst Case input? –Already-sorted input Best case? –Partition is perfectly balanced

Worst Case of Quicksort In the worst case: T(1) =  (1) T(n) = T(n - 1) +  (n) Does the recursion look familiar? T(n) =  (n 2 )

Best Case of Quicksort In the best case: T(n) = 2T(n/2) +  (n) Does the recursion familiar? T(n) =  (n lg n)

Analyzing Quicksort: Average Case an intuitive explanation/example: –Suppose that partition() always produces a 9- to-1 split. This looks quite unbalanced! –The recurrence is: T(n) = T(9n/10) + T(n/10) + n –T(n) = θ (n log n) –Such an imbalanced partition and θ(n log n) time?

Analyzing Quicksort: Average Case Intuitively, a real-life run of quicksort will produce a mix of “bad” and “good” splits –Pretend for intuition that they alternate between best-case (n/2 : n/2) and worst-case (n-1 : 1) –What happens if we bad-split root node, then good-split the resulting size (n-1) node?

Analyzing Quicksort: Average Case Intuitively, a real-life run of quicksort will produce a mix of “bad” and “good” splits –Pretend for intuition that they alternate between best- case (n/2 : n/2) and worst-case (n-1 : 1) –What happens if we bad-split root node, then good- split the resulting size (n-1) node? We end up with three subarrays, size 1, (n-1)/2, (n-1)/2 Combined cost of splits = n + n -1 = 2n -1 = O(n) No worse than if we had good-split the root node!

Analyzing Quicksort: Average Case Intuitively, the O(n) cost of a bad split (or 2 or 3 bad splits) can be absorbed into the O(n) cost of each good split Thus running time of alternating bad and good splits is still O(n lg n), with slightly higher constants How can we be more rigorous?

Analyzing Quicksort: Average Case For simplicity, assume: –All inputs distinct (no repeats) –Slightly different partition() procedure partition around a random element, which is not included in subarrays all splits (0:n-1, 1:n-2, 2:n-3, …, n-1:0) equally likely What is the probability of a particular split happening? Answer: 1/n

Analyzing Quicksort: Average Case So partition generates splits (0:n-1, 1:n-2, 2:n-3, …, n-2:1, n-1:0) each with probability 1/n If T(n) is the expected running time, What is each term under the summation for? What is the  (n) term for?

Quicksort Vs Merge Sort Merge Sort takes O(n lg n) in the worst case Quick Sort takes O(n 2 ) in the worst case So why would anybody use Qsort instead of merge sort? –Because in practice, Qsort is quick as the worst case doesn’t happen often.

Heap What is a heap? Heap is a complete binary tree in which all the levels are completely filled except possibly the last.

Heap property In addition to above, a heap satisfies the following property: A[parent(i)] >= A[i] i.e the value at a child is no more than that of its parent.

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.

Maintaining the heap property: Heapify If the heap property is violated at node I, it is restored as follows: Swap it with the largest of its children say C and, If the heap property at C is satisfied we are done else recursively maintain the property at C.

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.

Time to heapify The time to heapify satisfies the following recurrence relation T(n) <= T(2n/3) + theta(1) since the children’s subtree has at most 2n/3 nodes, worst case occurring when the last level is exactly half full. Hence T(n) = O(height) of the tree = O(lg n)

Constructing a heap Insert all the elements in the array. For i = floor(n/2) down to 1 heapify at i

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.

Time to construct heap First look : n/2 heapify => O(n lg n) More careful analysis: At most ceil(n/2 h+1 ) nodes at height h, therefore time is ∑ {h = 0 to lg n} ceil(n/2 h+1 ) O(h) = O(n)

Heapsort Construct the heap For I = n down to 1 Swap the root with i th element and heapify at the root leaving the i th element. Time: O(n) + O(n lg n) = O(lg n)

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.

Heap as priorityQ Heapsort is a very good algorithm but a good implementation of Qsort beats it in practice. PriorityQ is a data structure that supports the following operations: –Insert(S,x) –Max(S) –ExtractMax(S)

Insert Insert at the end and move up the heap until the heap property is satisfied. Time = O(height) = O(lg n)

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.

ExtractMax Max(S) is straight forward and is O(1) ExtractMax: Return the value at the root Swap it with the last element and heapify at the root leaving the last element. Time: O(height) =O(lg n)

In-place sorting A sorting algorithm is said to be in-place if it takes at most constant amount of extra space. Which of the algorithms you have studied are in-place? –Insertion Sort

In-place sorting A sorting algorithm is said to be in-place if it takes at most constant amount of extra space. Which of the algorithms you have studied are in-place? –Insertion Sort √

In-place sorting A sorting algorithm is said to be in-place if it takes at most constant amount of extra space. Which of the algorithms you have studied are in-place? –Insertion Sort √ –Merge Sort

In-place sorting A sorting algorithm is said to be in-place if it takes at most constant amount of extra space. Which of the algorithms you have studied are in-place? –Insertion Sort √ –Merge Sort X

In-place sorting A sorting algorithm is said to be in-place if it takes at most constant amount of extra space. Which of the algorithms you have studied are in-place? –Insertion Sort √ –Merge Sort X –Heap Sort ? Find out yourself!

In-place sorting A sorting algorithm is said to be in-place if it takes at most constant amount of extra space. Which of the algorithms you have studied are in-place? –Insertion Sort √ –Merge Sort X –Heap Sort ? –Quick Sort

In-place sorting A sorting algorithm is said to be in-place if it takes at most constant amount of extra space. Which of the algorithms you have studied are in-place? –Insertion Sort √ –Merge Sort X –Heap Sort ? –Quick Sort - appears to be in-place but

In-place sorting A sorting algorithm is said to be in-place if it takes at most constant amount of extra space. Which of the algorithms you have studied are in-place? –Insertion Sort √ –Merge Sort X –Heap Sort ? –Quick Sort it is not

Reminder for Assignment 5 Show that Insertion sort works correctly by using Induction.

Assignment 6 Problem 2.38 of Sara Baase. Give linear time algorithm for the problem. Problem For part (b) number of comparisons should be exactly 7.

Instructions for Minor If you’ll use more than one method to solve a problem, you will be evaluated out of half of the marks. For example, if you solve a Q of 4 marks in 2 ways, one of them I’ll strike off randomly and will evaluate out of 2 marks.

Stable sorting

Tools for designing algorithms Build up the solution Iteratively: –Example ? Divide and Conquer –Example ?

Tools for designing algorithms Build up the solution Iteratively: –Example : Insertion Sort Divide and Conquer –Example : Merge Sort and QSort Other tools to follow later

Up Next Lower Bounds Linear-Time Sorting Algorithms

The End