1. Lecture 2 Basic Concepts on Probability (Section 0.2)
Theory of Information Lecture 2
Theory of Information
Lecture 2 Basic Concepts on Probability (Section 0.2)

2. Sample Spaces and Events
Theory of Information Lecture 2
DEFINITION The set of all possible outcomes of a process is called
the sample space of that process. Any subset of the sample space, i.e.
any set of outcomes, is called an event.
Example 1: Process = flipping a coin;
sample space = {H,T}.
events: {H,T}, {H}, {T}, {}
Example 2: Process = rolling a pair of dice;
sample space = {(1,1), (1,2), …, (2,1), …, (6,6)}
events: {(1,1),(2,2),(3,3),(4,4),(5,5),(6,6)},
{(1,3), (3,1), (2,2)},
etc.

3. Probability Distribution/Law
Theory of Information Lecture 2
DEFINITION Let S={s1,…,sn} be an (ordered) sample space. A
probability law for S is an assignment P that sends each si to a real
number pi (0  pi1) such that p1+…+pn=1. The sequence p1,…,pn
is said to be a probability distribution for S.
Example: For the process of flipping a fair coin, an adequate
probablity law is P(H)=1/2, P(T)=1/2.
If the coin is not fair and H occurs twice as often as T, then we would
have .

4. Probability of an Event
Theory of Information Lecture 2
DEFINITION Let E be an event in a sample space S, and P a
probability law for P. Then the probability of E, denoted by P(E), is
the sum of the probabilities of each outcome in the event. (The empty
sum is =0).
Example: For fair coin flipping, we have the following probabilities for
the following events:
P({H})=
P({T})=
P({H,T})=
P({})=

5. Probability of an Event
Theory of Information Lecture 2
Example: Consider the process of rolling a fair pair of dice.
What is P(i,j) for each pair i,j?
What is the event of the second die being twice as big as the first one?
What is the probability of the above event?
What is the probability of the event that both dice are the same?
What is the probability of the event that the two dice are different?

6. Uniform Probability Distribution
Theory of Information Lecture 2
DEFINITION A uniform probability distribution for a sample space of size n is 1/n,…,1/n.
THEOREM Let S be a sample space with uniform probability law. If E is an event in E,
|E|
P(E) = -----
|S|

7. The Basic Properties of Probability
Theory of Information Lecture 2
THEOREM Let S be a sample space. Then
P({}) =
P(S) =
 P(E)  , for all events ES
P(Ec) = , for all events ES
If events E and F are disjoint (mutually exclusive), then
P(EF) =
More generally, if E1,…,En are mutually disjoint, then
P(E1  …  En) =

8. Theory of Information Lecture 2
Example 0.2.3
Theory of Information Lecture 2
EXAMPLE A black box emits 4 bits, with each bit with equal
probability. What is the probability of emitting an even number of 1s?
What is the size of the sample space?
How many 4-bit strings are there with k (0 k 4) 1s?
How many 4-bit strings are there with an even number of 1s?
So, P(even number of 1s) =

9. Theory of Information Lecture 2
Example 0.2.4
Theory of Information Lecture 2
EXAMPLE Five cards are drawn, each with equal probability,
from a deck of 52 cards. What is the probability of getting exactly 3
aces (event E)?
What is the size of the sample space S?
What is the size of the event E?
P(E) =

10. Theory of Information Lecture 2
Example 0.2.5
Theory of Information Lecture 2
EXAMPLE Five decimal digits a1,…,a5 are chosen at random.
What is the probability that a5 is the same as one of the previous 4
digits (event E)?
What is the size of the sample space S?
What is the size of Ec?
P(Ec) =
P(E) =

11. Theory of Information Lecture 2
Homework
Theory of Information Lecture 2
2.1. A fair coin is flipped 3 times.
What is the sample space here? (list all of its elements).
List all of the elements of the event that all flips result in the same outcome (all heads, or all tails).
What is the probability of the above event?
2.2. What is the probability that the last card of a deck of 52 cards is an ace? In your
answer, indicate the size of the sample space and the size of the event.
2.3. Four cards are drawn, each with equal probability, from a deck of 52 cards. What is the probability of getting exactly 2 aces?
2.4. Five (binary) bits b1,…,b5 are chosen at random. What is the probability that b5 is the same as one of the previous 4 bits?