1. Probability Theory Overview and Analysis of Randomized Algorithms
Analysis of Algorithms
Prepared by
John Reif, Ph.D.

2. Probability Theory Topics
Random Variables: Binomial and Geometric
Useful Probabilistic Bounds and Inequalities

3. Readings
Main Reading Selections:
CLR, Chapter 5 and Appendix C

4. Probability Measures
A probability measure (Prob) is a mapping from a set of events to the reals such that:
For any event A
0 < Prob(A) < 1
Prob (all possible events) = 1
If A, B are mutually exclusive events, then
Prob(A  B) = Prob (A) + Prob (B)

5. Conditional Probability
Define

6. Bayes’ Theorem
If A1, …, An are mutually exclusive and contain all events then

7. Random Variable A (Over Real Numbers)
Density Function

8. Random Variable A (cont’d)
Prob Distribution Function

9. Random Variable A (cont’d)
If for Random Variables A,B
Then “A upper bounds B” and “B lower bounds A”

10. Expectation of Random Variable A
Ā is also called “average of A” and “mean of A” = μA

11. Variance of Random Variable A

12. Variance of Random Variable A (cont’d)

13. Discrete Random Variable A

14. Discrete Random Variable A (cont’d)

15. Discrete Random Variable A Over Nonnegative Numbers
Expectation

16. Pair-Wise Independent Random Variables
A,B independent if
Prob(A ∧ B) = Prob(A) * Prob(B)
Equivalent definition of independence

17. Bounding Numbers of Permutations
n! = n * (n-1) * 2 * 1 = number of permutations of n objects
Stirling’s formula
n! = f(n) (1+o(1)), where

18. Bounding Numbers of Permutations (cont’d)
Note
Tighter bound
= number of permutations of n objects taken k at a time

19. Bounding Numbers of Combinations
= number of (unordered) combinations of n objects taken k at a time
Bounds (due to Erdos & Spencer, p. 18)

20. Bernoulli Variable Ai is 1 with prob P and 0 with prob 1-P
Binomial Variable
B is sum of n independent Bernoulli variables Ai each with some probability p

22. B is Binomial Variable with Parameters n,p

23. B is Binomial Variable with Parameters n,p (cont’d)

24. Poisson Trial Ai is 1 with prob Pi and 0 with prob 1-Pi
Suppose B’ is the sum of n independent Poisson trials Ai with probability Pi for i > 1, …, n

25. Hoeffding’s Theorem B’ is upper bounded by a Binomial Variable B
Parameters n,p where

26. Geometric Variable V parameter p GEOMETRIC parameter p V 0
loop with probability 1-p
exit
procedure
begin
goto

27. Probabilistic Inequalities
For Random Variable A

28. Markov and Chebychev Probabilistic Inequalities
Markov Inequality (uses only mean)
Chebychev Inequality
(uses mean and variance)

29. Example of use of Markov and Chebychev Probabilistic Inequalities
If B is a Binomial with parameters n,p

30. Gaussian Density Function

31. Normal Distribution
Bounds x > 0 (Feller, p. 175)

32. Sums of Independently Distributed Variables
Let Sn be the sum of n independently distributed variables A1, …, An
Each with mean and variance
So Sn has mean μ and variance σ2

33. Strong Law of Large Numbers: Limiting to Normal Distribution
The probability density function of to normal distribution Φ(x)
Hence
Prob

34. Strong Law of Large Numbers (cont’d)So
Prob
(since 1- Φ(x) < Ψ(x)/x)

35. Moment Generating Functions and Chernoff Bounds
Advanced Material
Moment Generating Functions and Chernoff Bounds

36. Moments of Random Variable A (cont’d)
n’th Moments of Random Variable A
Moment generating function

37. Moments of Random Variable A (cont’d)
Note
S is a formal parameter

38. Moments of Discrete Random Variable A
n’th moment

39. Probability Generating Function of Discrete Random Variable A

40. Moments of AND of Independent Random Variables
If A1, …, An independent with same distribution

41. Generating Function of Binomial Variable B with Parameters n,p
Interesting fact

42. Generating Function of Geometric Variable with parameter p

43. Chernoff Bound of Random Variable A
Uses all moments
Uses moment generating function
By setting x = ɣ’ (s) 1st derivative minimizes bounds

44. Chernoff Bound of Discrete Random Variable A
Choose z = z0 to minimize above bound
Need Probability Generating function

45. Chernoff Bounds for Binomial B with parameters n,p
Above mean x > μ

46. Chernoff Bounds for Binomial B with parameters n,p (cont’d)
Below mean x < μ

47. Anguin-Valiant’s Bounds for Binomial B with Parameters n,p
Just above mean
Just below mean

48. Anguin-Valiant’s Bounds for Binomial B with Parameters n,p (cont’d)
Tails are bounded by Normal distributions

49. Binomial Variable B with Parameters p,N and Expectation μ= np
By Chernoff Bound for p < ½
Raghavan-Spencer bound for any ∂ > 0

50. Prepared by John Reif, Ph.D.
Probability Theory
Analysis of Algorithms
Prepared by
John Reif, Ph.D.