1. Probability Theory Overview and Analysis of Randomized Algorithms Prepared by John Reif, Ph.D. Analysis of Algorithms

2. Probability Theory Topics a)Random Variables: Binomial and Geometric b)Useful Probabilistic Bounds and Inequalities

3. Readings Main Reading Selections: –CLR, Chapter 5 and Appendix C

4. Probability Measures A probability measure (Prob) is a mapping from a set of events to the reals such that: 1)For any event A 0  Prob(A)  1 2)Prob (all possible events) = 1 3)If A, B are mutually exclusive events, then Prob(A  B) = Prob (A) + Prob (B)

5. Conditional Probability Define

6. Bayes’ Theorem If A 1, …, A n are mutually exclusive and contain all events then

7. Random Variable A (Over Real Numbers) Density Function

8. Random Variable A (cont’d) Prob Distribution Function

9. Random Variable A (cont’d) If for Random Variables A,B Then “A upper bounds B” and “B lower bounds A”

10. Expectation of Random Variable A Ā is also called “average of A” and “mean of A” =  A

11. Variance of Random Variable A

12. Variance of Random Variable A (cont’d)

13. Discrete Random Variable A

14. Discrete Random Variable A (cont’d)

15. Discrete Random Variable A Over Nonnegative Numbers Expectation

16. Pair-Wise Independent Random Variables A,B independent if Prob(A  B) = Prob(A) X Prob(B) Equivalent definition of independence

17. Bounding Numbers of Permutations n! = n x (n-1) x 2 x 1 = number of permutations of n objects Stirling’s formula n! = f(n) x (1+o(1)), where

18. Bounding Numbers of Permutations (cont’d) Note –Tighter bound = number of permutations of n objects taken k at a time

19. Bounding Numbers of Combinations = number of (unordered) combinations of n objects taken k at a time Bounds (due to Erdos & Spencer, p. 18)

20. Bernoulli Variable A i is 1 with prob P and 0 with prob 1-P Binomial Variable B is sum of n independent Bernoulli variables A i each with some probability p

22. B is Binomial Variable with Parameters n,p

23. B is Binomial Variable with Parameters n,p (cont’d)

24. Poisson Trial A i is 1 with prob P i and 0 with prob 1-P i Suppose B’ is the sum of n independent Poisson trials A i with probability P i for i > 1, …, n

25. Hoeffding’s Theorem B’ is upper bounded by a Binomial Variable B Parameters n,p where

26. Geometric Variable V parameter p GEOMETRIC parameter p V 0 loop with probability 1-p exit procedure begin goto 

27. Probabilistic Inequalities For Random Variable A

28. Markov and Chebychev Probabilistic Inequalities Markov Inequality (uses only mean) Chebychev Inequality (uses mean and variance)

29. Example of use of Markov and Chebychev Probabilistic Inequalities If B is a Binomial with parameters n,p

30. Gaussian Density Function

31. Normal Distribution Bounds x > 0 (Feller, p. 175)

32. Sums of Independently Distributed Variables Let S n be the sum of n independently distributed variables A 1, …, A n Each with mean and variance So S n has mean  and variance  2

33. Strong Law of Large Numbers: Limiting to Normal Distribution The probability density function of to normal distribution (x) Hence Prob

34. Strong Law of Large Numbers (cont’d) So Prob (since 1- (x)  (x)/x)

35. Moment Generating Functions and Chernoff Bounds Advanced Material

36. Moments of Random Variable A (cont’d) n’th Moments of Random Variable A Moment generating function

37. Moments of Random Variable A (cont’d) Note S is a formal parameter

38. Moments of Discrete Random Variable A n’th moment

39. Probability Generating Function of Discrete Random Variable A

40. If A 1, …, A n independent with same distribution Moments of AND of Independent Random Variables

41. Generating Function of Binomial Variable B with Parameters n,p Interesting fact

42. Generating Function of Geometric Variable with parameter p

43. Chernoff Bound of Random Variable A Uses all moments Uses moment generating function By setting x = ’ (s) 1 st derivative minimizes bounds

44. Chernoff Bound of Discrete Random Variable A Choose z = z 0 to minimize above bound Need Probability Generating function

45. Chernoff Bounds for Binomial B with parameters n,p Above mean x  

46. Chernoff Bounds for Binomial B with parameters n,p (cont’d) Below mean x  

47. Anguin-Valiant’s Bounds for Binomial B with Parameters n,p Just above mean Just below mean

48. Anguin-Valiant’s Bounds for Binomial B with Parameters n,p (cont’d) Tails are bounded by Normal distributions

49. Binomial Variable B with Parameters p,N and Expectation  = pN By Chernoff Bound for p < ½ Raghavan-Spencer bound for any  > 0

50. Probability Theory Prepared by John Reif, Ph.D. Analysis of Algorithms