1. PROBABILITY AND STATISTICS FOR ENGINEERING
Hossein Sameti
Sharif University of Technology
Spring 2008

2. Sharif University of Technology
Text Book
A. Papoulis and S. Pillai,
Probability, Random Variables and Stochastic Processes,
4th Edition, McGraw Hill, 2000
4/17/2017
Sharif University of Technology

3. Source AT: www.mhhe.com/papoulis
TABLE OF CONTENTS
PROBABILITY THEORY
Lecture – Basics
Lecture – Independence and Bernoulli Trials
Lecture – Random Variables
Lecture – Binomial Random Variable Applications, Conditional
Probability Density Function and Stirling’s Formula.
Lecture – 5 Function of a Random Variable
Lecture – 6 Mean, Variance, Moments and Characteristic Functions
Lecture – Two Random Variables
Lecture – One Function of Two Random Variables
Lecture – 9 Two Functions of Two Random Variables
Lecture – 10 Joint Moments and Joint Characteristic Functions
Lecture – 11 Conditional Density Functions and Conditional Expected Values
Lecture – 12 Principles of Parameter Estimation
Lecture – 13 The Weak Law and the Strong Law of Large numbers
STOCHASTIC PROCESSES
Lecture – 14 Stochastic Processes - Introduction
Lecture – 15 Poisson Processes
Lecture – 16 Mean square Estimation
Lecture – 17 Long Term Trends and Hurst Phenomena
Lecture – 18 Power Spectrum
Lecture – 19 Series Representation of Stochastic processes
Lecture – 20 Extinction Probability for Queues and Martingales
Source AT:

4. Sharif University of Technology
Teaching Assistants:
Hoda Akbari
Course Evaluation:
Homeworks: 10-20%
Midterm: 25-30%
Final Exam: 50-60%
4/17/2017
Sharif University of Technology

5. Probability And Statistics
Source:
4/17/2017
Sharif University of Technology

6. PROBABILITY THEORY
1.Basics

7. Random Phenomena, Experiments
Study of random phenomena
Different outcomes
Outcomes that have certain underlying patterns about them
Experiment
repeatable conditions
Certain elementary events Ei occur in different but completely uncertain ways.
probability of the event Ei : P(Ei )>=0
4/17/2017
Sharif University of Technology

8. Probability Definitions
Laplace’s Classical Definition
without actual experimentation
provided all these outcomes are equally likely.
Example
a box with n white and m red balls
elementary outcomes: {white , red}
Probability of “selecting a white ball”:
4/17/2017
Sharif University of Technology

9. Probability Definitions
Relative Frequency Definition
The probability of an event A is defined as
nA is the number of occurrences of A
n is the total number of trials
4/17/2017
Sharif University of Technology

10. Probability Definitions
Example
The probability that a given number is divisible by a prime p:
4/17/2017
Sharif University of Technology

11. Sharif University of Technology
Counting - Remark
General Product Rule
if an operation consists of k steps each of which can be performed in ni ways (i = 1, 2, …, k), then the entire operation can be performed in ni ways.
Number of PINs
Number of elements in a Cartesian product
Number of PINs without repetition
Number of Input/Output tables for a circuit with n input signals
Number of iterations in nested loops
Example
4/17/2017
Sharif University of Technology

12. Permutations and Combinations - Remark
If order matters choose k from n:
Permutations :
If order doesn't matters choose k from n:
Combinations :
Example
A fair coin is tossed 7 times. What is the probability of obtaining 3 heads? What is the probability of obtaining at most 3 heads?
4/17/2017
Sharif University of Technology

13. Example: The Birthday Problem
Suppose you have a class of 23 students. Would you think it likely or unlikely that at least two students will have the same birthday?
It turns out that the probability of at least two of 23 people having the same birthday is about 0.5 (50%).
4/17/2017
Sharif University of Technology

14. Axioms of Probability- Basics
The axiomatic approach to probability, due to Kolmogorov, developed through a set of axioms
The totality of all events known a priori, constitutes a set Ω, the set of all experimental outcomes.
4/17/2017
Sharif University of Technology

15. Axioms of Probability- Basics
A and B are subsets of Ω . A B A B A
4/17/2017
Sharif University of Technology

16. Mutually Exclusiveness and Partitions
A and B are said to be mutually exclusive if
A partition of  is a collection of mutually exclusive(ME) subsets of  such that their union is . A B
4/17/2017
Sharif University of Technology

17. Sharif University of Technology
De-Morgan’s Laws A B A B A B A B
4/17/2017
Sharif University of Technology

18. Sharif University of Technology
Events
Often it is meaningful to talk about at least some of the subsets of  as events
we must have mechanism to compute their probabilities.
Example
Tossing two coins simultaneously:
A: The event of “Head has occurred at least once” .
4/17/2017
Sharif University of Technology

19. Events and Set Operators
“Does an outcome belong to A or B”
“Does an outcome belong to A and B”
“Does an outcome fall outside A”?
These sets also qualify as events.
We shall formalize this using the notion of a Field.
4/17/2017
Sharif University of Technology

20. Sharif University of Technology
Fields
A collection of subsets of a nonempty set  forms a field F if
Using (i) - (iii), it is easy to show that the following also belong to F.
4/17/2017
Sharif University of Technology

21. Sharif University of Technology
Fields If
then
We shall reserve the term event only to members of F.
Assuming that the probability P(Ei ) of elementary outcomes Ei of Ω are apriori defined.
The three axioms of probability defined below can be used to assign probabilities to more ‘complicated’ events.
4/17/2017
Sharif University of Technology

22. Sharif University of Technology
Axioms of Probability
For any event A, we assign a number P(A), called the probability of the event A.
Conclusions:
4/17/2017
Sharif University of Technology

23. Probability of Union of to Non-ME Sets
4/17/2017
Sharif University of Technology

24. Sharif University of Technology
Union of Events
Is Union of denumerably infinite collection of pairwise disjoint events Ai an event?
If so, what is P(A ) ?
We cannot use third probability axiom to compute P(A), since it only deals with two (or a finite number) of M.E. events.
4/17/2017
Sharif University of Technology

25. An Example for Intuitive Understanding
in an experiment, where the same coin is tossed indefinitely define:
A = “head eventually appears”.
Our intuitive experience surely tells us that A is an event. If
We have:
Extension of previous notions must be done based on our intuition as new axioms.
4/17/2017
Sharif University of Technology

26. σ-Field (Definition):
A field F is a σ-field if in addition to the three mentioned conditions, we have the following:
For every sequence of pairwise disjoint events belonging to F, their union also belongs to F
4/17/2017
Sharif University of Technology

27. Extending the Axioms of Probability
If Ai s are pairwise mutually exclusive
from experience we know that if we keep tossing a coin, eventually, a head must show up:
But:
Using the fourth probability axiom we have:
4/17/2017
Sharif University of Technology

28. Sharif University of Technology
Reasonablity
In previously mentioned coin tossing experiment:
So the fourth axiom seems reasonable.
4/17/2017
Sharif University of Technology

29. Summary: Probability Models
The triplet (, F, P)
 is a nonempty set of elementary events
F is a -field of subsets of .
P is a probability measure on the sets in F subject the four axioms
The probability of more complicated events must follow this framework by deduction.
4/17/2017
Sharif University of Technology

30. Conditional Probability
In N independent trials, suppose NA, NB, NAB denote the number of times events A, B and AB occur respectively.
According to the frequency interpretation of probability, for large N,
Among the NA occurrences of A, only NAB of them are also found among the NB occurrences of B.
Thus the following is a measure of “the event A given that B has already occurred”:
4/17/2017
Sharif University of Technology

31. Satisfying Probability Axioms
We represent this measure by P(A|B) and define:
As we will show, the above definition is a valid one as it satisfies all probability axioms discussed earlier.
4/17/2017
Sharif University of Technology

32. Satisfying Probability Axioms
4/17/2017
Sharif University of Technology

33. Properties of Conditional Probability
Example
In a dice tossing experiment,
A : outcome is even
B: outcome is 2.
The statement that B has occurred
makes the odds for A greater
4/17/2017
Sharif University of Technology

34. Law of Total Probability
We can use the conditional probability to express the probability of a complicated event in terms of “simpler” related events.
Suppose that
So,
4/17/2017
Sharif University of Technology

35. Conditional Probability and Independence
A and B are said to be independent events, if
This definition is a probabilistic statement, not a set theoretic notion such as mutually exclusiveness.
If A and B are independent,
Thus knowing that the event B has occurred does not shed any more light into the event A.
4/17/2017
Sharif University of Technology

36. Independence - Example
From a box containing 6 white and 4 black balls, we remove two balls at random without replacement.
What is the probability that the first one is white and the second one is black?
4/17/2017
Sharif University of Technology

37. Sharif University of Technology
Example - continued
Are W1 and B2 independent?
Removing the first ball has two possible outcomes:
These outcomes form a partition because:
So,
Thus the two events are not independent.
4/17/2017
Sharif University of Technology

38. General Definition of Independence
Independence between 2 or more events:
Events A1,A2, ..., An are mutually independent if, for all possible subcollections of k ≤ n events:
Example
In experiment of rolling a die,
A = {2, 4, 6}
B = {1, 2, 3, 4}
C = {1, 2, 4}.
Are events A and B independent?
What about A and C?
Source:
4/17/2017
Sharif University of Technology

39. Sharif University of Technology
Bayes’ Theorem
We have:
Thus,
Also,
4/17/2017
Sharif University of Technology

40. Bayesian Updating: Application Of Bayes’ Theorem
Suppose that A and B are dependent events and A has apriori probability of P(A ) .
How does Knowing that B has occurred affect the probability of A?
The new probability can be computed based on Bayes’ Theorm.
Bayes’ Theorm shows how to incorporate the knowledege about B’s occuring to calculate the new probability of A.
4/17/2017
Sharif University of Technology

41. Bayesian Updating - Example
Suppose there is a new music device in the market that plays a new digital format called MP∞. Since it’s new, it’s not 100% reliable.
You know that
20% of the new devices don’t work at all,
30% last only for 1 year,
and the rest last for 5 years.
If you buy one and it works fine, what is the probability that it will last for 5 years?
Source:
4/17/2017
Sharif University of Technology

42. Generalization of Bayes’ Theorem
A more general version of Bayes’ theorem involves partition of Ω :
In which,
Represents a collection of mutually exclusive events with assiciated apriori probabilities:
With the new information “B has occurred”, the information about Ai can be updated by the n conditional probabilities:
4/17/2017
Sharif University of Technology

43. Bayes’ Theorem - Example
Two boxes, B1 and B2 contain 100 and 200 light bulbs respectively. The first box has 15 defective bulbs and the second 5. Suppose a box is selected at random and one bulb is picked out.
What is the probability that it is defective?
4/17/2017
Sharif University of Technology

44. Sharif University of Technology
Example - Continued
Suppose we test the bulb and it is found to be defective. What is the probability that it came from box 1?
Note that initially,
But because of greater ratio of defective bulbs in B1 ,this probability is increased after the bulb determined to be defective..
4/17/2017
Sharif University of Technology