Background Knowledge Brief Review on Counting,Counting, Probability,Probability, Statistics,Statistics, I. TheoryI. Theory

Counting: Permutations  Permutations:  The number of possible permutations of r objects from n objects is from n objects is n ( n-1) (n-2) … (n –r +1) = n! / (n-r)! n ( n-1) (n-2) … (n –r +1) = n! / (n-r)!  We denote this number as nPr Remember the factorial of a number x = x! is defined as x! = (x) (x-1) (x-2) …. (2)(1) x! = (x) (x-1) (x-2) …. (2)(1)

Counting: Permutations  Permutations with indistinguishable objects:  Assume we have a total of n objects.  r1 are alike, r2 are alike,.., rk are alike.  The number of possible permutations of n objects is n! / r1! r2! … rk! n! / r1! r2! … rk!

Counting: Combinations  Combinations:  Assume we wish to select r objects from n objects.  In this case we do not care about the order in which we select the r objects. which we select the r objects.  The number of possible combinations of r objects from n objects is from n objects is n ( n-1) (n-2) … (n –r +1) / r! = n! / (n-r)! r! n ( n-1) (n-2) … (n –r +1) / r! = n! / (n-r)! r!  We denote this number as C(n,r)

Statistical and Inductive Probability  Statistical:  Relative frequency of occurrence after many trials  Inductive:  Degree of belief on certain event We will be concerned with the statistical view only. 0.5 Number of flips of a coin Proportion of heads Law of large numbers

The Sample Space  The space of all possible outcomes of a given process or situation is called the sample space S. or situation is called the sample space S. Example: cars crossing a check point based on color and size: S red & small blue & small red & large blue & large

An Event  An event is a subset of the sample space. Example: Event A: red cars crossing a check point irrespective of size S red & small blue & small red & large blue & large A

The Laws of Probability  The probability of the sample space S is 1, P(S) = 1  The probability of any event A is such that 0 <= P(A) <= 1.  Law of Addition If A and B are mutually exclusive events, then the probability that either one of them will occur is the sum of the individual probabilities: P(A or B) = P(A) + P(B) P(A or B) = P(A) + P(B) If A and B are not mutually exclusive: If A and B are not mutually exclusive: P(A or B) = P(A) + P(B) – P(A and B) P(A or B) = P(A) + P(B) – P(A and B) A B

Conditional Probabilities  Given that A and B are events in sample space S, and P(B) is different of 0, then the conditional probability of A given B is different of 0, then the conditional probability of A given B is P(A|B) = P(A and B) / P(B) P(A|B) = P(A and B) / P(B)  If A and B are independent then P(A|B) = P(A)

The Laws of Probability  Law of Multiplication What is the probability that both A and B occur together? P(A and B) = P(A) P(B|A) P(A and B) = P(A) P(B|A) where P(B|A) is the probability of B conditioned on A. where P(B|A) is the probability of B conditioned on A. If A and B are statistically independent: If A and B are statistically independent: P(B|A) = P(B) and then P(B|A) = P(B) and then P(A and B) = P(A) P(B) P(A and B) = P(A) P(B)

Random Variable Definition: A variable that can take on several values, each value having a probability of occurrence.  There are two types of random variables:  Discrete. Take on a countable number of values.  Continuous. Take on a range of values. Discrete Variables Discrete Variables  For every discrete variable X there will be a probability function P(x) = P(X = x). P(x) = P(X = x).  The cumulative probability function for X is defined as F(x) = P(X <= x). F(x) = P(X <= x).

Random Variable Continuous Variables:  Concept of histogram.  For every variable X we will associate a probability density function f(x). The probability is the area lying between function f(x). The probability is the area lying between two values. two values. Prob(x1 < X <= x2) = ∫ x1 f(x) dx Prob(x1 < X <= x2) = ∫ x1 f(x) dx  The cumulative probability function is defined as F(x) = Prob( X <= x) = ∫ -infinity f(u) du F(x) = Prob( X <= x) = ∫ -infinity f(u) du x2 x

Multivariate Distributions  P(x,y) = P( X = x and Y = y).  P’(x) = Prob( X = x) = ∑ y P(x,y) It is called the marginal distribution of X It is called the marginal distribution of X The same can be done on Y to define the marginal The same can be done on Y to define the marginal distribution of Y, P”(y). distribution of Y, P”(y).  If X and Y are independent then P(x,y) = P’(x) P”(y) P(x,y) = P’(x) P”(y)

Expectations: The Mean  Let X be a discrete random variable that takes the following values: values: x1, x2, x3, …, xn. x1, x2, x3, …, xn. Let P(x1), P(x2), P(x3),…,P(xn) be their respective Let P(x1), P(x2), P(x3),…,P(xn) be their respective probabilities. Then the expected value of X, E(X), is probabilities. Then the expected value of X, E(X), is defined as defined as E(X) = x1P(x1) + x2P(x2) + x3P(x3) + … + xnP(xn) E(X) = x1P(x1) + x2P(x2) + x3P(x3) + … + xnP(xn) E(X) = Σi xi P(xi) E(X) = Σi xi P(xi)

The Binomial Distribution  What is the probability of getting x successes in n trials?  Assumption: all trials are independent and the probability of success remains the same. success remains the same. Let p be the probability of success and let q = 1-p Let p be the probability of success and let q = 1-p then the binomial distribution is defined as then the binomial distribution is defined as P(x) = n C x p x q n-x for x = 0,1,2,…,n P(x) = n C x p x q n-x for x = 0,1,2,…,n The mean equals n p The mean equals n p

The Multinomial Distribution  We can generalize the binomial distribution when the random variable takes more than just two values. random variable takes more than just two values.  We have n independent trials. Each trial can result in k different values with probabilities p1, p2, …, pk. values with probabilities p1, p2, …, pk.  What is the probability of seeing the first value x1 times, the second value x2 times, etc. second value x2 times, etc. P(x1,x2,…,xk) = [n! / (x1!x2!…xk!)] p1 x1 p2 x2 … pk xk P(x1,x2,…,xk) = [n! / (x1!x2!…xk!)] p1 x1 p2 x2 … pk xk

Other Distributions  Poisson P(x) = e -u u x / x! P(x) = e -u u x / x!  Geometric f(x) = p(1-p) x-1  Exponential f(x) = λ e -λx  Others:  Normal  χ 2, t, and F

Entropy of a Random Variable A measure of uncertainty or entropy that is associated to a random variable X is defined as H(X) = - Σ pi log pi where the logarithm is in base 2. This is the “average amount of information or entropy of a finite complete probability scheme” (Introduction to I. Theory by Reza F.).

Example of Entropy P(A) = 1/256, P(B) = 255/256 P(A) = 1/256, P(B) = 255/256 H(X) = bit H(X) = bit P(A) = 1/2, P(B) = 1/2 P(A) = 1/2, P(B) = 1/2 H(X) = 1 bit H(X) = 1 bit P(A) = 7/16, P(B) = 9/16 P(A) = 7/16, P(B) = 9/16 H(X) = bit H(X) = bit There are two possible complete events A and B (Example: flipping a biased coin).

Entropy of a Binary Source It is a function concave downward bit

Derived Measures Average information per pairs H(X,Y) = - ΣxΣy P(x,y) log P(x,y) H(X,Y) = - ΣxΣy P(x,y) log P(x,y) Conditional Entropy: H(X|Y) = - ΣxΣy P(x,y) log P(x|y) Mutual Information: I(X;Y) = ΣxΣy P(x,y) log [P(x,y) / (P(x) P(y))] = H(X) + H(Y) – H(X,Y) = H(X) + H(Y) – H(X,Y) = H(X) – H(X|Y) = H(X) – H(X|Y) = H(Y) – H(Y|X) = H(Y) – H(Y|X)