Chapter 7: Quick Sort Recursive sorting using Partition Partition: algorithm that enable divide and conquer Worst-case runtime of Quicksort = O(n2) Best-case runtime of Quicksort = W(nlgn) Quicksort is asymptotically optimal Is worst case not a good characterization of Quicksort run time? Randomized algorithms

Pseudocode Quicksort(A,p,r) % to sort whole array p=1, r=length[A] if p < r then q  Partition(A,p,r) % q is index that defines subarrarys Quicksort(A,p,q-1) Quicksort(A,q+1,r)

Partition: Algorithm for Divide and Conquer

Continue until j = r-1, then exchange A[i+1] and A[r] Note: upper and lower sub-arrays are not the same size and not sorted Note: pivot is sorted relative elements in upper and lower sub-arrays Partition runtime: T(n) =Q(n) why?

Quicksort’s worst case Quicksort(A,p,r]) if p < r then q  Partition(A,p,r) % q is the index of pivot Quicksort(A,p,q-1) Quicksort(A,q+1,r) Worst-case partitioning: (always maximum imbalance) Partition(A,p,r) always returns q = p What type of input would make this happen? A worst-case Quicksort(A,p,r]) p  Worst Partition(A,p,r) Quicksort(A,p+1,r) What happened to the lower sub-array? What recurrence describes this worst case? What is its solution?

Worst case input is sorted in decreasing order Worst case input is sorted in decreasing order. In very call to partition, A(r) is the smallest element. Worst-case runtime of Quicksort described by T(n) = T(n-1) + Q(n) In HW12 showed by tree analysis and informal proof that T(n) = Q(n2)

More rigorous worst-case analysis We argued for T(n)=T(n-1)+Q(n) as worst case for Quicksort based on a particular input (sorted in decreasing order). A derive a rigorous recurrence for worst-case Quicksort: Assume elements of A are indexed 0 to n-1 lower array contains elements 0 to q-1 upper array contains elements q+1 to n-1 On the 1st call to Partition, 0 < q < n-1 Include the pivot in the lower sub-array Find the value of q that maximizes T(n) T(n) = (T(q) + T(n-q-1)) + Q(n) Show by the substitution method that solution is T(n)=O(n2).

T(n) = (T(q) + T(n-q-1)) + Q(n) Replace Q(n) by dn Assume: T(q)=O(q2), T(n-q-1)=O((n-q-1)2) Exist c>0 such that T(q)<cq2 and T(n-q-1)<c(n-q-1)2 T(n) < (cq2 + c(n-q-1)2) + dn < c ( (q2 + (n-q-1)2)) + dn f(q)=q2+((n-1)-q)2 df/dq=2q-2(n-q-1)=0 qm = (n-1)/2 d2f/dq2 = 4 therefore f(q) is minimum at qm f(0)=(n-1)2 and f(n-1)=(n-1)2 so f(q) is maximum at q= 0 and n-1 If qmax= 0 or qmax= n-1 then T(n) < c(n-1)2 + dn < cn2 which is the same inequality we encounter in proof that T(n)=T(n-1)+dn has asymptotic solution T(n)=O(n2)

Best case runtime of Quicksort described by Use T(n) = (T(q) + T(n-q-1)) + Q(n) to show by structured substitution that T(n) has asymptotic solution T(n)=W(nlgn). This proves that Quicksort is asymptotically optimal

Therefore T(n)=W(nlgn) by definition

Suppose partition always returned split 9n/10 and n/10 Will the performance of Quicksort be O(n2)? What recurrence describes this performance? What is its solution?

T(n)=T(9n/10)+T(n/10)+Q(n) Guess solution by imbalanced tree analysis.

CptS 350 Spring 2019 Homework Assignment 14: due 3/8/19 Prove by structured substitution that T(n)=T(9n/10)+T(n/10)+Q(n) has asymptotic solution T(n)=Q(nlgn)

Summary: Worst case, imbalance 0 and n-1, Quicksort: T(n)=O(n2) With any degree of imbalance other than 0 and n-1: T(n)=Q(nlgn) Questions: Is the worst case an outlier? What is the runtime of Quicksort with a “typical” input? Define a stochastic version of Quicksort: Randomized Quicksort Find the expected value of the runtime In general, “randomized” sorting means “Given array of n elements to be sorted, choose at random one of the n! permutations of the input” For analysis of Randomized Quicksort, a different method of randomizing is easier to analyze.

Randomized by a random choice of pivot. Randomized Quicksort Randomized by a random choice of pivot. Randomized-Partition(A, p, r) i  Random(p, r) exchange A[r]  A[i] return Partition Randomized-Quicksort(A, p, r) if p < r then q  Ramdomized-Partition(A, p, r) Randomized-Quicksort(A, p, q-1) Randomized-Quicksort(A, q+1, r) Randomized-Partition just calls Partition after randomly choosing a pivot. Runtime of Randomized Quicksort is a random variable. Random choice of pivot varies the degree of imbalance between upper and lower subarrarys

Expected run time of Randomized Quicksort What is the expected number of comparison from all calls to Randomized-Partition in a complete execution of Randomized-Quicksort(A,1,n)? Before we answer this question Review probability theory (Appendix C) Learn the technique of “indicator random variables”

Appendix C.2: Theory discrete probabilities Discrete probabilities are defined in terms of a sample space S. Usually S is a collection of elementary events that are outcomes of independent experiments, such as flipping coins, throwing dice, pulling cards, etc. |S| = size of S (also called “cardinality” of S) Example: S = set of outcomes from flipping 2 coins = {HH, HT, TH, TT} Events are subsets of S. (S itself is called the “certain” set.) The event of getting 1 head and 1 trail = {HT, TH} The empty subset, , called the “null event”

Brief review of set theory (Appendix B.1, text p1158) If x  A implies x  B, then A is a subset of B (written A  B) If, in addition, A  B then A is a proper subset of B (written A  B) A  B = intersection of sets A and B defined by {x: x  A and x  B} A  B = union of sets A and B defined by {x: x  A or x  B} A - B = difference of sets A and B defined by {x: x  A and x  B} For laws obeyed by sets see text pp1159-1162

Definition of probability on sample space S (1) Pr{A} > 0 for any event A (2) Pr{S} = 1 (3) for any 2 mutually exclusive events (i.e. A  B = ) Pr{A  B} = Pr{A} + Pr{B} By definition, elementary events are always mutually exclusive. From the definition of discrete probabilities, it follows that Pr{} = 0 If A  B then Pr{A} < Pr{B} Pr{S – A} = 1- Pr{A} S-A is the complement of A For any two events (not necessarily mutually exclusive) Pr{A  B} = Pr{A} + Pr{B} – Pr{A  B} < Pr{A} + Pr{B}

Discrete probability distributions Defined over a finite or countable infinite sample space Let s denote elementary events of S, then for any event A Pr{A} =  sA P{s} If S is finite and every sS has probability 1/|S|, then P(s) is uniformly distributed on S. Example: S = {HH, HT, TH, TT} = elemental events from flipping 2 coins, Pr{HH} = Pr{HT} = Pr{TH} = P{TT} = ¼ for “fair” coins The probability of at least one head is Pr{HH, HT, TH} = Pr{HH} + Pr{HT} + Pr{TH} = ¾ = 1 – Pr{TT} = 1 – ¼ = ¾

A “fair” coin flipped n times defines a uniform probability distribution on S Elementary events are strings HHTHT… For each of n positions in the string we have 2 choices of letter |S| = 2n For n=2, S = {HH, HT, TH, TT}, What is the probability of event A = {exactly k heads in n-k trails}? Pr{A} = (nk)/2n where (nk) = n!/(k!(n-k)!) = number ways to choose k items out of a total of n (nk) are binomial coefficients (see text pp 1185-1186) S is normalized

Conditional probability: Given some knowledge about outcomes, we want the probability of an outcome conditioned on our prior knowledge about it. Example: Suppose that someone flips 2 coins and tells us that at least one coin shows heads. What is the probability that both coins are showing heads? S = {HH, HT, TH, TT} Our prior knowledge eliminates elementary event TT. The remaining 3 possibilities are equally likely. Pr{HH} conditioned on at least 1 head is showing = 1/3 > Pr{HH} in the absence of any information about events.

Conditional probability: general definition Probability of A conditioned on B, written Pr{A|B}, is meaningful only if Pr{B}  0. Given that B occurs, the probability that A also occurs is related to the set of outcomes in which both A and B occur.  Pr{A|B} is proportional to Pr{AB} If we normalize Pr{A|B} by dividing by Pr{B} (which ≠ 0), then Pr{B|B} = Pr{BB}/Pr{B} = Pr{B}/Pr{B} = 1

Apply Pr{A|B} = Pr{AB}/Pr{B} to the problem Conditional probability: application of definition Apply Pr{A|B} = Pr{AB}/Pr{B} to the problem “probability of 2 heads showing given that at least one head is showing” AB is the event with 2 heads showing and at least one head showing. In S = {HH, HT, TH, TT}, HH is the only such event. Pr{AB} = ¼ B is the event with at least one head showing. 3 out of 4 events in S have this property; therefore Pr{B} = ¾ Using the definition, Pr{A|B} = Pr{AB}/Pr{B} Pr{A|B} = (1/4)/(3/4) = 1/3 > Pr{AB} Additional knowledge increased probability

Two events are independent if Pr{AB} = Pr{A}Pr{B} Conditional probability: application to independent events Two events are independent if Pr{AB} = Pr{A}Pr{B} If events A and B are independent and Pr{B}  0, then Pr{A|B} = Pr{A}Pr{B}/Pr{B} = Pr{A}

Discrete random variables: Mappings of finite or countable infinite sets of events onto the real numbers For random variable X and real number x, the event X = x is the collection of all elementary events {sS: X(s) = x}. Pr{X = x} = {sS: X(s) = x} Pr{s} f(x) = Pr{X = x} is the “probability density function” of random variable X By axioms of probability f(x) > 0 and x f(x) = 1

Example of discrete random variable: A pair of 6-sided dice is rolled. Random variable X is the max of the 2 numbers showing. What is Pr{X = 3}? Use exhaustive enumeration Elementary events with 3 as max of the 2 numbers showing (1,3), (2,3), (3,3), (3,2), and (3,1). The probability of each elementary event is (1/6)(1/6) = 1/36  Pr{X = 3} = 5/36

Expected value of a discrete random variable E[X] = x xPr{X=x} If g(X) defines a new random variable, then E[g(X)] = x g(x)Pr{X=x} Special case: Given random variables X and Y, the property of sums implies E[aX + bY] = aE[X] + bE[Y] Called “linearity” of expectation values

For any two random variables X and Y joint probability density function is Pr{X=x and Y=y} for any fixed value of y Pr{Y=y} = x Pr{X=x and Y=y} called “marginal” probability Pr{X=x | Y=y} = Pr{X=x and Y=y}/Pr{Y=y} called “conditional” probability X and Y are independent if for all x and y Pr{X = x and Y = y} = Pr{X = x}Pr{Y = y} If X and Y are independent, then E[XY] = E[X]E[Y] proof on p1198 of text

Indicator Random Variables For any event A, Indicator Random Variable I{A} = 1 if A occurs = 0 if A does not occur Zero and one are the only values an indicator random variables can have Lemma 5.1: Let A be an event in sample space S, then the expected value of I{A} = probability of event A. Proof: For any random variable X, E[X] = x x Pr{X = x} For indicator random variables x = 0 or 1, only The complement of event A in sample space S is S-A E[I{A}] = (1)Pr(A) + (0)Pr(S - A) = Pr(A)

Structured solution to “What is the expected number of heads for n coin flips?” solved by indicator random variables What is the random variable whose expectation value we want to calculate? 2. What event defines the indicator random variable? 3. What is the probability that this occurs? 4. How is random variable whose expectation we are calculating related to the indicator random variable? 5. What is the expectation value of this random variable?

Structured solution to “What is the expected number of heads for n coin flips?” solved by indicator random variables 1. What is the random variable whose expectation value we want to calculate? X = number of heads showing in n coins flips 2. What event defines the indicator random variable? Ai = ith coin flip shows heads. Let I{Ai} = Xi 3. What is the probability that Ai occurs? E(Xi) = ½ 4. How is random variable whose expectation we are calculating related to the indicator random variable? X = i=1 to n (Xi) 5. What is the expectation value of this random variable? E[X] = E[i=1 to n (Xi)] = i=1 to n E[Xi] = i=1 to n (1/2) = n/2

Expected run time of Randomized Quicksort using indicator random variables What is the random variable whose expectation value we wish to calculate? X=number of comparison from all calls to Randomized-Partition in a complete execution of Randomized-Quicksort(A,1,n)

X=number of comparison from all calls to Randomized-Partition in a complete execution of Randomized-Quicksort(A,1,n) What is the event that defines the indicator random variable? Let Aij be the event when element i compared to j Xij = I{Aij} How is X related to the indicator random variable? X = sum over all distinct ordered pairs This is correct iff any pair of elements can be compared at most once.

Last line of Partition stores the pivot between partitioned subarrays Once used, a pivot is never involved in subsequent recursive calls In sorting an array of length n, Partition called at most n times Elements are compared to pivots only Elements in separate subarrays are never compared X= is the correct relationship random variable and indicator random variable

Rename the elements <a1, a2, Rename the elements <a1, a2, ..., an> of array A by their order statistic Call then z1, z2, ..., zn, where zi is the ith smallest element z1 is the minimum element, zn is maximum element Define subsets of elements Zij = {zi, zi+1, ..., zj} to be the elements (by size) between zi and zj inclusively Since elements are compared to pivots only and elements in separate subarrays are never compared, zi and zj are compared, at most, once when one or the other is a pivot. X = is the correct relationship between X and I{Aij} E(X) = =

zi and zj are compared iff the 1st pivot chosen from the set Zij={zi, zi+1,...zj} is either zi and zj. If zi is the 1st pivot chosen from the set Zij then zj cannot be the 1st pivot chosen from that set two possibilities leading to comparison of zi and zj are mutually exclusive.  Pr{ zi is compared to zj} = Pr{ zi is the 1st pivot chosen from Zij} + Pr{ zj is the 1st pivot chosen from Zij} = 2/(j – i + 1) because all elements in Zij equally likely to be first chosen

E(X) = = Evaluate this sum

E[X] = E[X] = (by change of variable k = j-i) E[X] < E[x] < O(lg(n)) E[x] = O(nlgn) k+1 -> k increases every term extending inner sum to n gives harmonic sum value, ln(n) + O(1), differs from lg(n) by constant

Cpt S 350 Spring 2019   Homework Assignment 15: due 3/20/19 Use indicator random variables to compute the expected value of the sum of faces showing when n fair dice are tossed. Give a structured solution.

Chapter 9: Selection of Order Statistics Since this is another application of partition, delay Quiz #4 until we cover this material