1. Computational Learning Theory
Introduction
The PAC Learning Framework
Finite Hypothesis Spaces
Examples of PAC Learnable Concepts
What is machine learning?

2. Computational learning theory (or CLT):
Introduction
Computational learning theory (or CLT):
Provides a theoretical analysis of learning
Shows when a learning algorithm can be expected to succeed
Shows when learning may be impossible
What is machine learning?

3. There are typically three areas comprised by CLT:
Introduction
There are typically three areas comprised by CLT:
Sample Complexity. How many examples we need
to find a good hypothesis?
Computational Complexity. How much computational
power we need to find a good hypothesis?
Mistake Bound. How many mistakes we will make
before finding a good hypothesis?
What is machine learning?

4. Computational Learning Theory
Introduction
The PAC Learning Framework
Finite Hypothesis Spaces
Examples of PAC Learnable Concepts
What is machine learning?

5. The PAC Learning Framework
Let’s start with a simple problem:
Assume a two dimensional space with positive and
negative examples. Our goal is to find a rectangle that
includes the positive examples but not the negatives
(input space is R2):
true concept
What is machine learning? - - + - + - + + - + - + - - -

6. Distribution D. Assume instances are generated at random
Definitions
Definitions:
Distribution D. Assume instances are generated at random
from a distribution D.
Class of Concepts C. Let C be a class of concepts that we
wish to learn. In our example C is the family of all rectangles
in R2.
Class of Hypotheses H. The hypotheses our algorithm
considers while learning the target concept.
True error of a hypothesis h
errorD(h) = Pr[c(x) = h(x)]D
What is machine learning?

7. True error is the probability of regions A and B.
The true error:
true concept c
Region A - - + - + - + + - + -
What is machine learning? +
hypothesis h - - -
Region B
True error is the probability of regions A and B.
Region A : false negatives
Region B : false positives

8. Considerations
Two considerations:
We don’t want a hypothesis or approximation with
zero error. There might be some error as long as it
is small (bounded by a constant ε).
We don’t need to always produce a good hypothesis
for every sample of examples (the probability of failure
will be bounded by a constant δ).

9. The PAC Learning Framework
PAC learning stands for probably approximate correct.
Roughly, it tells us a class of concepts C (defined over an
input space with examples of size n) is PAC learnable by
a learning algorithm L, if for arbitrary small δ and ε, and
for all concepts c in C, and for all distributions D over the
input space, there is a 1-δ probability that the hypothesis h
selected from space H by learning algorithm L is
approximately correct (has error less than ε). L must run
in time polynomial in 1/ ε , 1/ δ, n, and the size of c.

10. Example
true concept c - - + - + - + - + + - +
hypothesis h (most specific) - - -
Assume a learning algorithm that outputs the rectangle
that is most specific (touches the positive examples at the
border of the rectangle).
Question: Is this class of problems (rectangles in R2) PAC
learnable by L?

11. Example
error
true concept c - - + - + - - + + + - +
hypothesis h (most specific) - - -
The error is the probability of the area between h and the
true target rectangle c. How many examples do we need to
make this error less than ε?

12. Example
error
true concept c - - + - + - - + + + - +
hypothesis h (most specific) - - -
In general, the probability that m independent examples
have NOT fallen within the error region is (1- ε) m which we
want to be less than δ.

13. Example
In other words, we want:
(1- ε) m <= δ
Since (1-x) <= e–x we have that
e – εm <= δ or
m >= (1/ ε) ln (1/ δ)
The result grows linearly in 1/ ε and logarithmically 1/ δ

14. Example
error
true concept c - - + - + - - + + + - +
hypothesis h (most specific) - - -
The analysis above can be applied by considering each of the
four stripes of the error region. Can you finish the analysis
considering each stripe separately and then joining their
probabilities?

15. Computational Learning Theory
Introduction
The PAC Learning Framework
Finite Hypothesis Spaces
Examples of PAC Learnable Concepts
What is machine learning?

16. Sample Complexity for Finite Hypothesis Spaces
Definition:
A consistent learner outputs the hypothesis h in H that
perfectly fits the training examples (if possible).
How many examples do we need to be approximately
correct in finding a hypothesis output by a consistent
learner that has low error?

17. Version Space ε-exhausted
This is the same as asking how many examples we need to
make the Version Space contain no hypothesis with error
greater than ε.
When a Version Space VS is such that no hypothesis has error
greater than ε, we say the version space is ε-exhausted.
How many examples do we need to make a version space VS
be ε-exhausted?

18. Probability of Version Space being ε-exhausted
The probability that the version space is not ε-exhausted
after seeing m examples is bounded by the probability
than all hypotheses in VS have error greater than ε. Since
the size of the VS is less than the size of the whole hypothesis
space H, then that probability is clearly less than
|H|e –εm
If we make this less than δ , then we have that
m >= 1/ ε (ln |H| + ln (1/ δ ))

19. Agnostic Learning
What happens if our hypothesis space H does not
contain the target concept c?;
Then clearly we can never find a hypothesis h with
zero error.
Here we want an algorithm that simply outputs
the hypothesis with minimum training error.

20. Computational Learning Theory
Introduction
The PAC Learning Framework
Finite Hypothesis Spaces
Examples of PAC Learnable Concepts
What is machine learning?

21. Examples of PAC-Learnable Concepts
Can we say that concepts described by conjunctions of
Boolean literals are PAC learnable?
First, how large is the hypothesis space when we have n
Boolean attributes?
Answer: |H| = 3n

22. Examples of PAC-Learnable Concepts
If we substitute this in our analysis of sample complexity
for finite hypothesis spaces we have that:
m >= 1/ ε (n ln 3 + ln (1/ δ) )
Thus the set of conjunctions of Boolean literals is
PAC learnable.

23. K-Term DNF not PAC Learnable
Consider now the class of functions of k-term DNF expressions.
These are expressions of the form
T1 V T2 V … V Tk where V stands for disjunction and
each term Ti is a conjunction of n Boolean attributes.

24. K-Term DNF not PAC Learnable
The size of |H| is k3n Using the equation for the sample
complexity of finite hypothesis spaces:
m >= 1/ ε (n ln 3 + ln (1/ δ) + ln k)
Although the sample complexity is polynomial in the
main parameters the problem is known to be NP complete.

25. K-Term CNF is PAC Learnable
But it is interesting to see that another family of
functions, the class of k-term CNF expressions is
PAC learnable.
This is interesting because the class of k-term CNF
expressions is strictly larger than the class of k-term
DNF expressions.