Probability Theory Overview and Analysis of Randomized Algorithms Analysis of Algorithms Prepared by John Reif, Ph.D.

Probability Theory Topics Random Variables: Binomial and Geometric Useful Probabilistic Bounds and Inequalities

Readings Main Reading Selections: CLR, Chapter 5 and Appendix C

Probability Measures A probability measure (Prob) is a mapping from a set of events to the reals such that: For any event A 0 < Prob(A) < 1 Prob (all possible events) = 1 If A, B are mutually exclusive events, then Prob(A  B) = Prob (A) + Prob (B)

Conditional Probability Define

Bayes’ Theorem If A1, …, An are mutually exclusive and contain all events then

Random Variable A (Over Real Numbers) Density Function

Random Variable A (cont’d) Prob Distribution Function

Random Variable A (cont’d) If for Random Variables A,B Then “A upper bounds B” and “B lower bounds A”

Expectation of Random Variable A Ā is also called “average of A” and “mean of A” = μA

Variance of Random Variable A

Variance of Random Variable A (cont’d)

Discrete Random Variable A

Discrete Random Variable A (cont’d)

Discrete Random Variable A Over Nonnegative Numbers Expectation

Pair-Wise Independent Random Variables A,B independent if Prob(A ∧ B) = Prob(A) * Prob(B) Equivalent definition of independence

Bounding Numbers of Permutations n! = n * (n-1) * 2 * 1 = number of permutations of n objects Stirling’s formula n! = f(n) (1+o(1)), where

Bounding Numbers of Permutations (cont’d) Note Tighter bound = number of permutations of n objects taken k at a time

Bounding Numbers of Combinations = number of (unordered) combinations of n objects taken k at a time Bounds (due to Erdos & Spencer, p. 18)

Bernoulli Variable Ai is 1 with prob P and 0 with prob 1-P Binomial Variable B is sum of n independent Bernoulli variables Ai each with some probability p

B is Binomial Variable with Parameters n,p

B is Binomial Variable with Parameters n,p (cont’d)

Poisson Trial Ai is 1 with prob Pi and 0 with prob 1-Pi Suppose B’ is the sum of n independent Poisson trials Ai with probability Pi for i > 1, …, n

Hoeffding’s Theorem B’ is upper bounded by a Binomial Variable B Parameters n,p where

Geometric Variable V parameter p GEOMETRIC parameter p V 0 loop with probability 1-p exit procedure begin goto ¬

Probabilistic Inequalities For Random Variable A

Markov and Chebychev Probabilistic Inequalities Markov Inequality (uses only mean) Chebychev Inequality (uses mean and variance)

Example of use of Markov and Chebychev Probabilistic Inequalities If B is a Binomial with parameters n,p

Gaussian Density Function

Normal Distribution Bounds x > 0 (Feller, p. 175)

Sums of Independently Distributed Variables Let Sn be the sum of n independently distributed variables A1, …, An Each with mean and variance So Sn has mean μ and variance σ2

Strong Law of Large Numbers: Limiting to Normal Distribution The probability density function of to normal distribution Φ(x) Hence Prob

Strong Law of Large Numbers (cont’d) So Prob (since 1- Φ(x) < Ψ(x)/x)

Moment Generating Functions and Chernoff Bounds Advanced Material Moment Generating Functions and Chernoff Bounds

Moments of Random Variable A (cont’d) n’th Moments of Random Variable A Moment generating function

Moments of Random Variable A (cont’d) Note S is a formal parameter

Moments of Discrete Random Variable A n’th moment

Probability Generating Function of Discrete Random Variable A

Moments of AND of Independent Random Variables If A1, …, An independent with same distribution

Generating Function of Binomial Variable B with Parameters n,p Interesting fact

Generating Function of Geometric Variable with parameter p

Chernoff Bound of Random Variable A Uses all moments Uses moment generating function By setting x = ɣ’ (s) 1st derivative minimizes bounds

Chernoff Bound of Discrete Random Variable A Choose z = z0 to minimize above bound Need Probability Generating function

Chernoff Bounds for Binomial B with parameters n,p Above mean x > μ

Chernoff Bounds for Binomial B with parameters n,p (cont’d) Below mean x < μ

Anguin-Valiant’s Bounds for Binomial B with Parameters n,p Just above mean Just below mean

Anguin-Valiant’s Bounds for Binomial B with Parameters n,p (cont’d) Tails are bounded by Normal distributions

Binomial Variable B with Parameters p,N and Expectation μ= np By Chernoff Bound for p < ½ Raghavan-Spencer bound for any ∂ > 0

Prepared by John Reif, Ph.D. Probability Theory Analysis of Algorithms Prepared by John Reif, Ph.D.