CS 3343: Analysis of Algorithms Order Statistics

Order statistics The ith order statistic in a set of n elements is the ith smallest element The minimum is thus the 1st order statistic The maximum is the nth order statistic The median is the n/2 order statistic If n is even, there are 2 medians How can we calculate order statistics? What is the running time?

Order statistics – selection problem Select the ith smallest of n elements Naive algorithm: Sort. Worst-case running time Q(n log n) using merge sort or heapsort (not quicksort). We will show: A practical randomized algorithm with Q(n) expected running time A cool algorithm of theoretical interest only with Q(n) worst-case running time

Recall: Quicksort £ x x ³ x r p q k The function Partition gives us the rank of the pivot If we are lucky, k = i. done! If not, at least get a smaller subarray to work with k > i: ith smallest is on the left subarray k < i : ith smallest is on the right subarray Divide and conquer If we are lucky, k close to n/2, or desired # is in smaller subarray If unlucky, desired # is in larger subarray (possible size n-1) £ x x ³ x r p q k

Randomized divide-and-conquer algorithm RAND-SELECT(A, p, q, i) ⊳ i th smallest of A[ p . . q] if p = q & i > 1 then error! r  RAND-PARTITION(A, p, q) k  r – p + 1 ⊳ k = rank(A[r]) if i = k then return A[ r] if i < k then return RAND-SELECT( A, p, r – 1, i ) else return RAND-SELECT( A, r + 1, q, i – k ) £ A[r] ³ A[r] r p q k

Randomized Partition Randomly choose an element as pivot Every time need to do a partition, throw a die to decide which element to use as the pivot Each element has 1/n probability to be selected Rand-Partition(A, p, q){ d = random(); // draw a random number between 0 and 1 index = p + floor((q-p+1) * d); // p<=index<=q swap(A[p], A[index]); Partition(A, p, q); // now use A[p] as pivot }

Select the 6 – 4 = 2nd smallest recursively. Example Select the i = 6th smallest: 7 10 5 8 11 3 2 13 i = 6 pivot 3 2 5 7 11 8 10 13 Partition: k = 4 Select the 6 – 4 = 2nd smallest recursively.

Complete example: select the 6th smallest element. i = 6 7 10 5 8 11 3 2 13 3 2 5 7 11 8 10 13 k = 4 i = 6 – 4 = 2 10 8 11 13 k = 3 i = 2 < k Note: here we always used first element as pivot to do the partition (instead of rand-partition). 8 10 k = 2 i = 2 = k 10

Intuition for analysis (All our analyses today assume that all elements are distinct.) Lucky: T(n) = T(9n/10) + Q(n) = Q(n) CASE 3 Unlucky: T(n) = T(n – 1) + Q(n) = Q(n2) arithmetic series Worse than sorting!

Running time of randomized selection T(max(0, n–1)) + n if 0 : n–1 split, T(max(1, n–2)) + n if 1 : n–2 split, M T(max(n–1, 0)) + n if n–1 : 0 split, T(n) ≤ For upper bound, assume ith element always falls in larger side of partition The expected running time is an average of all cases Expectation

Substitution method Want to show T(n) = O(n). So need to prove T(n) ≤ cn for n > n0 Assume: T(k) ≤ ck for all k < n if c ≥ 4 Therefore, T(n) = O(n)

Summary of randomized selection Works fast: linear expected time. Excellent algorithm in practice. But, the worst case is very bad: Q(n2). Q. Is there an algorithm that runs in linear time in the worst case? IDEA: Generate a good pivot recursively. A. Yes, due to Blum, Floyd, Pratt, Rivest, and Tarjan [1973].

Worst-case linear-time selection if i = k then return x elseif i < k then recursively SELECT the i th smallest element in the lower part else recursively SELECT the (i–k)th smallest element in the upper part SELECT(i, n) Divide the n elements into groups of 5. Find the median of each 5-element group by rote. Recursively SELECT the median x of the ën/5û group medians to be the pivot. Partition around the pivot x. Let k = rank(x). Same as RAND-SELECT

Choosing the pivot

Choosing the pivot Divide the n elements into groups of 5.

Choosing the pivot lesser greater Divide the n elements into groups of 5. Find the median of each 5-element group by rote.

Choosing the pivot x lesser greater Divide the n elements into groups of 5. Find the median of each 5-element group by rote. Recursively SELECT the median x of the ë n/5û group medians to be the pivot.

Analysis x lesser greater At least half the group medians are £ x, which is at least ë ë n/5û /2û = ë n/10û group medians.

Analysis x lesser greater At least half the group medians are £ x, which is at least ë ë n/5û /2û = ë n/10û group medians. Therefore, at least 3 ë n/10û elements are £ x. (Assume all elements are distinct.)

Analysis x lesser greater At least half the group medians are £ x, which is at least ë ë n/5û /2û = ë n/10û group medians. Therefore, at least 3 ë n/10û elements are £ x. Similarly, at least 3 ë n/10û elements are ³ x.

Need “at most” for worst-case runtime Analysis Need “at most” for worst-case runtime At least 3 ë n/10û elements are £ x  at most n-3 ë n/10û elements are  x At least 3 ë n/10û elements are  x  at most n-3 ë n/10û elements are  x The recursive call to SELECT in Step 4 is executed recursively on at most n-3 ë n/10û elements. 3 ë n/10û Possible position for pivot

Analysis Use fact that ë a/bû > a/b-1 n-3 ë n/10û < n-3(n/10-1)  7n/10 + 3  3n/4 if n ≥ 60 The recursive call to SELECT in Step 4 is executed recursively on at most 7n/10+3 elements.

Developing the recurrence T(n) if i = k then return x elseif i < k then recursively SELECT the i th smallest element in the lower part else recursively SELECT the (i–k)th smallest element in the upper part SELECT(i, n) Divide the n elements into groups of 5. Find the median of each 5-element group by rote. Recursively SELECT the median x of the ën/5û group medians to be the pivot. Partition around the pivot x. Let k = rank(x). Q(n) T(n/5) Q(n) T(7n/10+3)

Solving the recurrence Assumption: T(k) £ ck for all k < n if n ≥ 60 if c ≥ 20 and n ≥ 60

Conclusions Since the work at each level of recursion is basically a constant fraction (19/20) smaller, the work per level is a geometric series dominated by the linear work at the root. In practice, this algorithm runs slowly, because the constant in front of n is large. The randomized algorithm is far more practical. Exercise: Try to divide into groups of 3 or 7. Exercise: Think about an application in sorting.