1. Chapter 2 Concepts of Prob. Theory
Random Experiment: an experiment in which outcome varies in an unpredictable fashion when the experiment is
repeated under the same condition
Specified by:
1. An experimental procedure
2. One or more measurements or observations
tch-prob

2. EXAMPLE 2.1 :
Experiment E1 : Select a ball form an urn containing balls numbered 1 to 50. Note the number of the ball.
Experiment E2 : Select a ball form an urn containing balls numbered 1 to 4. Suppose that balls 1 and 2 are black and that balls 3 and 4 are white. Note number and color of the ball you select.
Experiment E3 : Toss a coin three times and note the sequence of heads/ tails.
Experiment E4 : Toss a coin three times and note the number of heads.
Experiment E6 : A block of information is transmitted repeatedly over a noisy channel until an error-free block arrives at the receiver.
Experiment E7 : Pick a number at random between zero and one.
Experiment E12 : Pick two numbers at random between zero and one.
Experiment E13 : Pick a number X at random between zero and one, then pick a number Y at random between zero and X.
tch-prob

3. Discussions Compare E3 and E4: same procedure, different observations
A random experiment with multiple measurements or observations: E2, E3, E12, E13
sequential experiment consists of multiple subexperiments: E3, E4, E6, E12, E13
dependent subexperiments: E13
tch-prob

4. Sample Space Outcome, sample point:
one and only one per experiment
mutually exclusive, cannot occur simultaneously
Sample space: set of all possible outcomes
Example 2.2: sample spaces
number of outcomes:
Finite
countably infinite S6
unaccountably infinite S7
Discrete
sample space
Zeta is the symbol name
Continuous
Sample space
tch-prob

5. EXAMPLE 2.2 :
The sample spaces corresponding to the experiments in Example 2.1 are given below using set notation :
tch-prob

6. elementary event: event of a single outcome
Event: a subset of S, a collection of outcomes that satisfy certain conditions
Certain event: S
null event:
elementary event: event of a single outcome
Example 2.3:
A2: The ball is white and even-numbered
A4: The number of heads equals the number of tails
A7: The number selected is nonnegative
tch-prob

7. A B Set Operations Union Intersection mutually exclusive Complement
imply
equal A=B
- Commutative properties of set operations
- Associative properties of set operations
- Distributive properties of set operations
- De Morgan’s Rules
Venn diagrams: Fig. 2.2 A B
tch-prob

8. Example 2.5: Configuration of a three-component system
a. series all three
b. parallel at least one of three
c. two-out-of-three C1 C2 C3 C1 C2 C3
tch-prob

9. 2.2 The Axioms of Probability
A probability law for the experiment E is a rule that assigns to each event a number P(A), called the probability of A, that satisfies the following axioms:
Axiom I: nonnegative
Axiom II: P(S)= total=1
Axiom III: If , then
Axiom III’: If for all , then
Axiom: a statement accepted as true as the basis for argument or inference
an established rule or principle or a self-evident truth
A set of consistency rules that any valid probability assignment must satisfy.
tch-prob

10. Corollary 4. are pairwise mutually exclusive, then for
pf:
Corollary 2.
pf: from Cor.1,
Corollary 3.
pf: Let A=S, in Cor.1.
Corollary are pairwise mutually exclusive,
then
for
Corollary: a proposition inferred immediately from a proved proposition with little or no additional proof
tch-prob

11. Corollary 5.
pf:
Corollary 6.
Corollary 7. If , then
pf.
tch-prob

12. Discrete Sample Spaces find the prob.of elementary events;
Axioms + corollaries  provide rules for computing the probability of certain events in terms of other events. However, we still need initial probability assignment
Discrete Sample Spaces
find the prob.of elementary events;
all distinct elementary events are mutually exclusive,
Example 2.6., 2.7
2. Continuous Sample Spaces
assign prob. to intervals of the real line or rectangular regions in the plane
Example 2.11
tch-prob

13. Discrete Sample Spaces
First, suppose that the sample space is finite, and is given by
(by corollary ?)
If S is countably infinite, then Axiom Ⅲ’ implies that the probability of an event such as is given by
tch-prob

14. The probability of any event that consists of k outcomes, say , is
If the sample space has n elements, , a probability assignment of particular interest is the case of equally likely outcomes. The probability of the elementary events is
The probability of any event that consists of k outcomes, say , is
Thus, if outcomes are equally likely, then the probability of an event is equal to the number of outcomes in the event divided by the total number of outcomes in the sample space.
(remember the classical definition?)
tch-prob

15. EXAMPLE 2.6
An urn contains 10 identical balls numbered 0,1,…, 9. A random experiment involves selecting a ball from the urn and noting the number of the ball. Find the probability of the following events :
A = “number of ball selected is odd,” B = “number of ball selected is a multiple of 3,” C = “number of ball selected is less than 5,”
and of and
tch-prob

16. If we assume that the outcomes are equally likely, then
The sample space is , so the sets of outcomes corresponding to the above events are
, , and .
If we assume that the outcomes are equally likely, then .
From Corollary 5.
tch-prob

17. where we have used the fact that , so . From Corollary 6,
tch-prob

18. *If we count the number of heads in three tosses, then
EXAMPLE 2.7
Suppose that a coin is tossed three times. If we observe the sequence of heads and tails, then there are eight possible outcomes If we assume that the outcomes of S3 are equiprobable, then the probability of each of the eight elementary events is 1/8. This probability assignment implies that the probability of obtaining two heads in three tosses is, by Corollary 3,
Many probability models can be devised for the same sample space and events by varying the probability assignment.
However, the probability assignment should be selected to reflect experimental observations to the extent possible.
*If we count the number of heads in three tosses, then
tch-prob

19. Continuous Sample Spaces EXAMPLE 2.11
Consider Experiment E12, where we picked two number x and y at random between zero and one. The sample space is then the unit square shown in Fig. 2.8(a). If we suppose that all pairs of numbers in the unit square are equally likely to be selected, then it is reasonable to use a probability assignment in which the probability of any region R inside the unit square is equal to the area of R. Find the probability of the following events: , , and
Figures 2.8(b) through 2.8(c) show the regions corresponding to the events A, B, and C. Clearly each of these regions has area ½. Thus
tch-prob

20. FIGURE 2.8 a Two-dimensional sample space and three events.
(b) Event x y y x y S x
(a) Sample space
(c) Event x y
(d) Event x y
tch-prob

21. 2.3 Computing probabilities using counting methods
In many experiments with finite sample space, the outcomes can be
assumed to be equiprobable.
Prob Counting the number of outcomes in an event
Sampling with Replacement and with Ordering
choose k objects from a set A that has n distinct objects, with replacement and note the order. The experiment produces an ordered k-tuple
where and
The number of distinct ordered k-tuples =
Ex balls, select 2
Pr[2 balls with same number] =
tch-prob

22. Pr [1st number >2nd number ]= 10/20
Sampling without Replacement and with ordering
choose k objects from a set A that has n distinct objects, without replacement.
number of distinct ordered k-tuples = n (n-1)…(n-k+1)
Ex balls, select 2
Pr [1st number >2nd number ]= 10/20
tch-prob

23. Permutations of n Distinct Objects Consider the case when k=n
number of permutations of n objects = n!
For large n, stirling’s formula is useful
Ex balls randomly placed into 12 cells, where more than 1 ball is
allowed to occupy a cell.
Pr [ all cells are occupied ]= ~
tch-prob

24. 3. Sampling without Replacement and without ordering
~ ” combination of size k” =
each combination has k! possible orders
“binomial coefficient”, read “n choose k”
Ex The number of distinctive permutation of k white balls
and n-k black balls
From n distinct positions, pick k for white balls
Partition the set of n distinctive objects into two sets
tch-prob

25. Ex. 2.19 A batch of 50 items contains 10 defective items.
Select 10 items at random, Pr [ 5 out of 10 defective ]=?
Exercise prob. 47: Multinomial
Ex Toss a die 12 times. Pr [ each number appears twice ]= ?
{1,1,2,2,3,3, …….6,6} permutation
Ex. 2.19: get 10 items in a shot (Permutation of similar items results in combinatorial (Binomial or multinomial) formula)
Ex. 2.20: the die is tossed 12 times, a sequential experiment.
use coin, tossed 4 times to explain
tch-prob

26. 4. Sampling with Replacement and without ordering
Suppose k=5, n=4
object
XX X XX
The result can be summarized as XX // X / XX
= k: x
n-1: / =
Ex Place k balls into n cells.
Pick k balls from an urn of
n balls. (k may be greater than n)
tch-prob

27. 2.4 Conditional Probability
We are interested in knowing whether A and B are related, i.e. ,
if B occurs, does it alter the likelihood of A occurs.
Define the conditional probability as
for P[B]>0
- renormalize
tch-prob

28. Example 2.21. A ball is selected from an urn containing 2 black balls (1,2)
and 2 white balls (3,4).
A: Black ball selected ½
B: even-numbered ball ½
C: number > ½
Not related
Related
tch-prob

29. is useful in finding prob. in sequential experiments
Ex An urn contains 2 black, 3 white balls.
Two balls are selected without replacement.
Sequence of colors is noted.
Sol. 1. b b two subexperiments
b w
w b
w w
Tree diagram
Cannot use 2.3.3, sampling without replacement and with ordering. Why?
Some balls are not distingushable.
tch-prob

30. S B1 B2 Bn A Let be mutually exclusive, whose union = S ,
partition of S.
“ Theorem on total probability”
Ex In prev. example (2.22) find P[W2]=Pr [second ball is white] S B1 B2 Bn A
In 2.24, find out what A and Bi correspond to.
tch-prob

31. Before experiment: P[ Bj ]= a priori probability
Suppose event A occurs, what is the prob. of event Bj?
Bayes’ Rule
Before experiment: P[ Bj ]= a priori probability
After experiment: A occurs
P[ Bj/A ]= a posteriori probability
tch-prob

32. Ex.2.26. Binary symmetric channel Suppose
Ak is input symbol event
Bk is output symbol event
tch-prob

33. Two events A and B are independent if
2.5 Independence of Events
If knowledge of the occurrence of an event B does not alter the probability
of some other event A, then A is independent of B.
Two events A and B are independent if
tch-prob

34. Ex. 2.28. A ball is selected from an urn of 4 balls
{1,b},{2,b},{3,w},{4,w}
A: black ball is selected
B: even-numbered ball
C: number >2
P[A]=P[B]=P[C]=0.5
If two events have non zero prob., and are mutually exclusive, then they cannot be independent.
tch-prob

35. Three events A, B, and C are independent ifEx
Common application of the independence concept is in making the assumption that the event of separate experiments are independent.
Ex A fair coin is tossed three times. P[HHH]=1/8, ….
I, 2, 3 do not necessarily imply 4
tch-prob

36. 2.6. Sequential Experiments Sequence of independent experiments
If sub experiments are independent, and event concerns outcomes of
the kth subexperiment, then are independent.
Bernoulli trial:
Perform an experiment once, note whether an event A occurs.
success, if A occurs prob= p
failure, otherwise prob= 1-p
Pn(k)= prob. of k successes in n independent repetitions of a Bernoulli trial = ?
tch-prob

37. Ex.2.34. toss a coin 3 times. P[ H ]= p
Binomial probability law
k=0,1,…,n
Bernoulli repeats n times  binomial
tch-prob

38. Ex.2.37. A binary channel with BER . 3 bits are transmitted.
Binomial theorem
Let a=b=1 ,
Let
Ex A binary channel with BER bits are transmitted.
majority decoding is used. What is prob. of error?
tch-prob

39. Multinomial Probability Law Bj’s: partition of S; mutually exclusive
n independent repetitions of experiment
number of times Bj occurs.
Ex dart areas
throw dart 9 times, P[3 on each area]=?
tch-prob

40. P[The initial k trials are failures]
Geometric Probability Law
Repeat independent Bernoulli trials until the occurrence of the first success.
P[ more than k trials are required before a success occurs ]
P[The initial k trials are failures]
tch-prob

41. Urn 1 Urn 2 Trellis diagram Sequence of Dependent Experiments
Ex A sequential experiment involves repeatedly drawing a ball from one of two urns, noting the number on the ball, and replacing the ball in its urn.
Urn 0 : 1 ball #1, 2 balls #0
Urn 1 : 5 balls #1, 1 ball #0
Initially, flip a coin. Thereafter, the urn used in the subexperiment corresponds to the number selected in the previous subexperiment.
Urn 1
Urn 2
Trellis diagram
tch-prob

42. conditional prob.
Markov property
Ex calculate
with
tch-prob

43. which four are aces and four are kings.
Problem 95 on Page 83
Consider a well-shuffled deck of cards consisting of 52 distinct cards, of
which four are aces and four are kings.
a. Pr[Obtaining an ace in the first draw]=
b. Draw a card from the deck and look at it.
Pr[Obtaining an ace in the second draw]=
If does not know the previous outcome
tch-prob

44. c. Suppose we draw 7 cards from the deck. Pr[7 cards include 3 aces]=
Pr[7 cards include 2 kings]=
Pr[7 cards includes 3 aces and 2 kings]=
d. Suppose the entire deck of cards is distributed equally among four
players. Pr[each player gets an ace]=
tch-prob

45. p =No two people in a group of n people will have a common birthday.
Birthday problem.
p =No two people in a group of n people will have a common birthday.
Lotto Problem.
Six numbers and one special number are picked from a pool of 42 numbers. …
What is the probability of winning the grand prize?
What is the probability of winning 2nd prize?
What is the probability that number 37 is drawn next time?
1/(42,6)
(6,5)*1/(42,6)
7/42 or 6/42
tch-prob