1. Concept Learning Machine Learning by T. Mitchell (McGraw-Hill) Chp. 2

2. Much of learning involves acquiring general concepts from specific training examples
e.g. what is a bird? what is a chair?
Concept learning: Inferring a boolean-valued function from training examples of its input and output.

3. A Concept Learning Task Example
Target Concept
“Days on which Aldo enjoys his favorite water sport”
(you may find it more intuitive to think of
“Days on which the beach will be crowded” concept)
Task
Learn to predict the value of EnjoySport/Crowded for an arbitrary day
Training Examples for the Target Concept
6 attributes (Nominal-valued (symbolic) attributes):
Sky (SUNNY, RAİNY, CLOUDY), Temp (WARM,COLD), Humidity (NORMAL, HIGH),
Wind (STRONG, WEAK), Water (WARM, COOL), Forecast (SAME, CHANGE)

4. A Learning Problem x1
Unknown
Function x2
y = f (x1, x2, x3, x4 ) x3 x4
Hypothesis Space (H): Set of all possible hypotheses that the learner may consider during learning the target concept.
How many?

5. Hypothesis Space: Unrestricted Case
| A  B | = | B | | A |
|H4  H | = | {0,1}  {0,1}  {0,1}  {0,1}  {0,1} | = 224 = function values
After 7 examples, still have 29 = 512 possibilities (out of 65536) for f
Is learning possible without any assumptions?

6. A Concept Learning Task
In this notation, a conjunction such as <warm,?,stong>
indicaties
A1=Warm AND A3=strong
Hypothesis h: Conjunction of Constraints on Attributes
Constraint Values:
Specific value (e.g., Water = Warm)
All values allowed for that attribute (e.g., “Water = ?”)
No value allowed for that attribute (e.g., “Water = Ø”)
Hypothesis Representation –
Example Hypothesis for EnjoySport:
‘’Aldo enjoys his favorite sport only on sunny days with Strong wind’’:
Sky AirTemp Humidity Wind Water Forecast
<Sunny ? ? Strong ? ? >
The most general hypothesis – every day is a positive example of this concept
<?, ?, ?, ?, ?, ?>
A very specific hypothesis – no day is a positive ex. of this concept
< Sunny, Warm, Normal, Strong, Warm, Same>
The most specific possible hypothesis – no day is a positive ex. of this concept
<  ,, , , , >
Is this hypothesis consistent with the training examples?
What are some hypotheses that are consistent with the examples?

7. A Concept Learning Task(2)
The instance space, X (book uses ‘’set of instances’’)
all possible days represented by attributes Sky, AirTemp,...
Target concept, c :
Any boolean-valued function defined over the instance space X
c : X  {0, 1} (I.e. if EnjoySport = Yes, then c(x) = 1 )
Training Examples (denoted by D): ordered pair <x, c(x)>
Positive example: member of the target concept, c(x) = 1
Negative example: nonmember of the target concept, c(x) = 0
Assumption: no missing X values
No noise in values of c (contradictory labels).
Hypotheses Space, H:
Often picked by the designer, for instance only conjunctions (e.g. A1  A5)
H is the set of Boolean valued functions defined over X
Or you may narrow it down to conjunction of constraints on attributes
The goal of the learner
Hypothesize or estimate c:
Find a hypothesis h such that h(x) = c(x) for all x in X

8. A Concept Learning Task(3)
Although the learning task is to determine a hypothesis h identical to c, over the entire set of instances X, the only information available about c is its value over the training instances D.
Inductive Learning Hypothesis
Any hypothesis found to approximate the target function well over a sufficiently large set of training examples will also approximate the target function well over other unobserved examples.

9. Concept Learning As Search
Concept learning can be viewed as the task of searching through a large space of hypotheses implicitly defined by the hypothesis representation.
The goal of this search is to find the hypothesis that (best) fits the training examples.
Sky AirTemp Humidity Wind Water Forecast
<Sunny/Rainy/Cloudy Warm/Cold Normal/High Weak/Strong Warm/Cold Change/Same >
EnjoySport Learning Task
Size of the instance space X
3  2  2  2  2  2 = 96
Syntactically distinct hypotheses (including ?, Ø)
5  4  4  4  4  4 = 5120
Semantically distinct hypotheses (+1 for no positive example)
1 + (4  3  3  3  3  3) = 973
Often much larger, sometimes infinite hypotheses spaces

10. Inductive Bias A biased Hypothesis space EnjoySport example
Restriction : only conjunctions of attribute values.
No representation for a disjunctive target concept
(Sky = Sunny) or (Wind = Weak)
Potential problem
We biased the learner (inductive bias) to consider only conjunctive hypotheses
But the concept may require a more expressive hypothesis space

11. UnBiased Learner An Unbiased Learner
Obvious solution: Provide a hypothesis space capable of representing every teachable concept = every possible subset of the instance space X
The set of all subsets of a set X is called the power set of X
EnjoySport Example
|Instance space| : 96
|Power set of X| : 296 =
|Conjunctive hypothesis space| : 973
Very biased hypothesis space indeed!!

12. Need for Inductive Bias
An Unbiased Learner
Reformulate the EnjoySport learning task in an unbiased way
Defining a new hypothesis space H’ that can represent every subset of X
Allow arbitrary disjunctions, conjunctions, and negations
Example: “Sky = Sunny or Wind = Weak”
<Sunny, ?, ?, ?, ?, ?> OR <?, ?, ?, Weak, ?, ?>
New problem: We will be completely unable to generalize beyond the observed samples!
Explanation: If we allow full flexibility for the concept (conjunctions, disjunctions,…) , then even if one unseen sample is left, then there are still 2 hypotheses left that are consistent with the data (i.e. You still havent precisely found the right concept, eliminating all those that are inconsistent with the data you have seen!).

13. Need For Inductive Bias
Fundamental property of inductive inference:
A learner that makes no a priori assumptions regarding the identity of the target concept has no rational basis for classifying any unseen instances