1. Machine Learning Chapter 2
Machine Learning Chapter 2. Concept Learning and The General-to-specific Ordering Gun Ho Lee Soongsil University, Seoul

2. Outline Learning from examples
General-to-specific ordering over hypotheses
Version spaces and candidate elimination algorithm
Picking new examples
The need for inductive bias
Note: simple approach assuming no noise,
illustrates key concepts

3. A Concept Examples of Concepts Concept
“birds”, “car”, “situations” in which I should study more in order to pass the exam”
Concept
Some subset of objects or events defined over a larger set, or
A boolean-valued function defined over this larger set.
Concept “birds” is the subset of animals that constitute birds.

4. Adapted by Doug Downey from: Bryan Pardo, EECS 349 Fall 2007
A Concept
Let there be a set of objects, X.
X = {White Fang, Scooby Doo, Wile E, Lassie}
A concept C is…
A subset of X
C = dogs = {Lassie, Scooby Doo}
A function that returns 1 only for elements in the concept
C(Lassie) = 1, C(Wile E) = 0
Adapted by Doug Downey from: Bryan Pardo, EECS 349 Fall 2007 4

5. Instance Representation
Represent an object (or instance) as an n-tuple of attributes
Example: Days (6-tuples)
Instance Sky AirTemp Humidity Wind Water Forecast EnjoySport
Sunny Warm Normal Strong Warm Same No
Sunny Warm High Strong Warm Same Yes
Rainy Cold High Strong Warm Change No
Sunny Warm High Strong Cool Change Yes 5

6. Concept Learning Learning Concept learning
Inducing general functions from specific training examples
Concept learning
Acquiring the definition of a general category given a sample of positive and negative training examples of the category
Inferring a boolean-valued function from training examples of its input and output.

7. A Concept Learning Task
Target concept EnjoySport
“days on which 박지성 enjoys water sport”
Hypothesis
A vector of 6 constraints, specifying the values of the six attributes
Sky, AirTemp, Humidity, Wind, Water, Forecast.
<?, Cold, High, ?, ?, ?> expresses the hypothesis that 박지성 enjoys his favorite sport only on cold days with high humidity.
Sky, AirTemp, Humidity, Wind, Water, Forecast

8. Representing Hypotheses
Many possible representations
Here, h is conjunction of constraints on attributes
Each constraint can be
a specific value (e.g., Water = Warm)
don’t care (e.g., “Water =?”)
no value allowed (e.g., “Water=Φ”)
For example,
Sky AirTemp Humid Wind Water Forecst
<Sunny ? ? Strong ? Same>

9. Task: Learn a hypothesis from a dataset

10. Example Concept Function
“Days on which my friend 박지성 enjoys his favorite water sport”
INPUT
OUTPUT
Sky
Temp
Humid
Wind
Water
Forecast
C(x)
sunny
warm
normal
strong
same 1
high
rainy
cold
change
cool 10

11. The Learning Task Given: Determine:
Hypotheses space H: conjunction of constraints on attributes.
E.g. conjunction of literals: < Sunny ? ? Strong ? Same >
Target concept c: E.g., EnjoySport X  {0,1}
Instances X: set of items over which the concept is defined.
E.g., days decribed by attributes: Sky, Temp, Humidity, Wind, Water, Forecast
Training examples (positive/negative): <x,c(x)>
Training set D: positive, negative examples of the target function:
<x1,c(x1)>,…, <xn,c(xn)>
Determine:
A hypothesis h in H such that h(x) = c(x), for all x in X

12. Assumption 1 We will explore the space of all conjunctions.
We assume the target concept falls within this space.
H, Hypotheses space
Target concept c

13. Assumption 2 A hypothesis close to target concept c obtained after
seeing many training examples will result in high
accuracy on the set of unobserved examples.
Training set D
Hypothesis h is good
Complement set D’
Hypothesis h is good
 Inductive learning hypothesis

14. Inductive Learning Hypothesis
Learning task is to determine h identical to c over the entire set of instances X.
But the only information about c is its value over D (training set).
Inductive learning algorithms can at best guarantee that the induced h fits c over D.
Inductive learning hypothesis
Any good hypothesis over a sufficiently large set of training examples will also approximate the target function. well over unseen examples.

15. Concept Learning as Search
Find a hypothesis that best fits training examples
Efficient search in hypothesis space (finite/infinite)
Search space in EnjoySport
• Sky has 3 (Sunny, Cloudy, and Rainy)
• Temp has 2 (Warm and Cold)
• Humidity has 2 (Normal and High)
• Wind has 2 (Strong and Weak)
• Water has 2 (Warm and Cool)
• Forecast has 2 (Same and Change)
3*2*2*2*2*2 = 96 distinct instances
5*4*4*4*4*4 = 5120 syntactically distinct hypotheses within H (considering Φ and ? in addition)
1+4*3*3*3*3*3 = 973 semantically distinct hypotheses
(count just one Φ for each attribute since every hypo having one or more Φ symbols is the empty set of instances) -

16. Adapted by Doug Downey from: Bryan Pardo, EECS 349 Fall 2007
Concept Generality
A concept P is more general than or equal to another concept Q iff the set of instances represented by P includes the set of instances represented by Q.
Mammal
Canine
Wolf
Pig
Dog
Lassie
White_fang
Wilbur
Scooby_doo
Charlotte
Adapted by Doug Downey from: Bryan Pardo, EECS 349 Fall 2007 16

17. General to Specific Order
Consider two hypotheses:
h1=< Sunny,?,?,Strong,?,?>
h2=< Sunny,?,?,?,?,?>
Definition: hj is more general than or equal to hk iff:
This imposes a partial order on a hypothesis space.
Adapted by Doug Downey from: Bryan Pardo, EECS 349 Fall 2007 17

18. Instance, Hypotheses, and More- General-Than
Instances
Hypotheses
specific x1 h1 h3
h2  h1
h2  h3 h2 x2
general
x1=< Sunny,Warm,High,Strong,Cool,Same>
h1=< Sunny,?,?,Strong,?,?>
x2=< Sunny,Warm,High,Light,Warm,Same>
h2=< Sunny,?,?,?,?,?>
h3=< Sunny,?,?,?,Cool,?>
The Most General Hypothesis : < ?, ?, ?, ?, ?, ? >
The Most Specific Hypothesis : < Ø, Ø, Ø, Ø, Ø, Ø >
Adapted by Doug Downey from: Bryan Pardo, EECS 349 Fall 2007

19. Find-S Algorithm 1. Initialize h to the most specific hypothesis in H
2. For each positive training instance x
For each attribute constraint ai in h
If the constraint ai in h is satisfied by x
Then do nothing
Else replace ai in h by the next more general constraint
satisfied by x
3. Output hypothesis h
Finding a Maximally Specific Hypothesis

20. Hypothesis Space Search by Find-S
Instances
Hypotheses h1
x1=<Sunny,Warm,Normal,Strong,Warm,Same> + x1
h1=< Sunny,Warm,Normal,
Strong,Warm,Same>
specific
x3=<Rainy,Cold,High,Strong,Warm,Change> - x3 h0
h0=< Ø, Ø, Ø, Ø, Ø, Ø,>
h2,3
x2=<Sunny,Warm,High,Strong,Warm,Same> + x2
h2,3=< Sunny,Warm,?,
Strong,Warm,Same> h4
x4=<Sunny,Warm,High,Strong,Cool,Change> + x4
h4=< Sunny,Warm,?,
Strong,?,?>
general

21. Properties of Find-S
Ignores every negative example (no revision to h required in response to negative examples).
Guaranteed to output the most specific hypothesis consistent with the positive training examples (for conjunctive hypothesis space).
Final h also consistent with negative examples provided the target c is in H and no error in D.

22. Weaknesses of Find-S
Has the learner converged to the correct target concept ? No way to know whether the solution is unique.
Why prefer the most specific hypothesis? How about the most general hypothesis?
Are the training examples consistent ? Training sets containing errors or noise can severely mislead the algorithm Find-S.
What if there are several maximally specific consistent hypotheses? No backtrack to explore a different branch of partial ordering.

23. Partial order of hypotheses I

24. Partial order of hypotheses II

25. Partial order of hypotheses III

26. The space of hypotheses

27. The space of hypotheses I

28. The space of hypotheses II

29. The space of hypotheses III

30. Consistent(h, D) ≡ (∀<x, c(x)>∈D) h(x) = c(x)
A hypothesis h is consistent with a set of training examples D of target concept c if and only if h(x) = c(x) for each training example <x, c(x)> in D.
Consistent(h, D) ≡ (∀<x, c(x)>∈D) h(x) = c(x)

31. VSH,D ≡ {h ∈ H | Consistent(h, D)}
Version Space
The version space, VSH,D, with respect to hypothesis space H and training examples D, is the subset of hypotheses from H consistent with all training examples in D.
VSH,D ≡ {h ∈ H | Consistent(h, D)}

32. Version Spaces
A hypothesis h is consistent with a set of training examples D of target concept c if and only if h(x) = c(x) for each training example <x, c(x)> in D.
Consistent(h, D) ≡ (∀<x, c(x)>∈D) h(x) = c(x)
The version space, V SH,D, with respect to hypothesis space H and training examples D, is the subset of hypotheses from H consistent with all training examples in D.
VSH,D ≡ {h ∈ H | Consistent(h, D)}

33. The List-Then-Eliminate Algorithm:
VersionSpace  a list containing every hypothesis in H
2. For each training example, <x, c(x)>
remove from VersionSpace any hypothesis h for which h(x)  c(x)
3. Output the list of hypotheses in VersionSpace

34. Drawbacks of List-Then-Eliminate
The algorithm requires exhaustively enumerating all hypotheses in H
An unrealistic approach ! (full search)
If insufficient (training) data is available, the algorithm will output a huge set of hypotheses consistent with the observed data

35. Adapted by Doug Downey from: Bryan Pardo, EECS 349 Fall 2007
Example Version Space S:
{<Sunny,Warm,?,Strong,?,?>}
<Sunny,?,?,Strong,?,?>
<Sunny,Warm,?,?,?,?>
<?,Warm,?,Strong,?,?> G:
{<Sunny,?,?,?,?,?>, <?,Warm,?,?,?>, }
x1 = <Sunny Warm Normal Strong Warm Same> +
x2 = <Sunny Warm High Strong Warm Same> +
x3 = <Rainy Cold High Strong Warm Change> -
x4 = <Sunny Warm High Strong Cool Change> +
Adapted by Doug Downey from: Bryan Pardo, EECS 349 Fall 2007

36. Representing Version Spaces
The General boundary, G, of version space VSH,D is the set of its maximally general members
The Specific boundary, S, of version space VSH,D is the set of its maximally specific members
Every member of the version space lies between these boundaries
VSH,D = {h ∈ H | (∃s ∈ S)(∃g ∈ G) (g ≥ h ≥ s)}
where x ≥ y means x is more general or equal to y

37. Relevant bounds

38. Basic Idea of Candidate Elimination Algorithm
Initialize G to the set of maximally general hypotheses in H
Initialize S to the set of maximally specific hypotheses in H
For each training example x, do
If x is positive: generalize S if necessary
If x is negative: specialize G if necessary

39. Candidate Elimination Algorithm (1/2)
G ← maximally general hypotheses in H
S ← maximally specific hypotheses in H
For each training example d, do
If d is a positive example
Remove from G any hypothesis inconsistent with d
For each hypothesis s in S that is not consistent with d
Remove s from S
Add to S all minimal generalizations h of s such that
1. h is consistent with d, and
2. some member of G is more general than h
Remove from S any hypothesis that is more general than another hypothesis in S S:
Add minimal generalizations G: h
inconsistent with d

40. Candidate Elimination Algorithm (2/2)
If d is a negative example
Remove from S any hypothesis inconsistent with d
For each hypothesis g in G that is not consistent with d
Remove g from G
Add to G all minimal specializations h of g such that
1. h is consistent with d, and
2. some member of S is more specific than h
Remove from G any hypothesis that is less general than another hypothesis in G
inconsistent with d S: h
Add minimal specializations G:

41. Candidate-Elimination Algorithm
When does this halt?
If S and G are both singleton sets, then:
if they are identical, output value and halt.
if they are different, the training cases were inconsistent. Output this and halt.
Else continue accepting new training examples.

42. Example Trace First initialize the S and G sets:

43. Given 1st Example S0 : {<0,0,0,0,0,0> }
The first example is positive:
< <Sunny, Warm, Normal, Strong, Warm, Same>, Yes>
S0 : {<0,0,0,0,0,0> }
S1 : { <Sunny, Warm, Normal, Strong, Warm, Same> }
G0, G1 : { < ?,?,?,?,?,?> }

44. Given 2nd Example
The second example is positive:
< <Sunny,Warm,High, Strong,Warm, Same>, Yes>
S1 : { <Sunny, Warm, Normal, Strong, Warm, Same> }
S2 : { <Sunny, Warm, ?, Strong, Warm, Same> }
G1, G2 : { < ?,?,?,?,?,?> }

45. Given 3rd Example
The third example is negative:
< <Rainy, Cold, High, Strong, Warm, Change>, No>
S2, S3 : {<Sunny, Warm, ?, Strong, Warm, Same> }
G3 : { <Sunny, ?, ?, ?, ?, ?>, <?, Warm, ?, ?, ?, ?>, <?, ?, ?, ?, ?, Same>}
G2: { < ?,?,?,?,?,?> }
Why is <?,?, Normal,?,?,? > not included in G ?

46. Given 4th Example S3 : {<Sunny, Warm, ?, Strong, Warm, Same> }
The 4th example is negative:
< <Sunny,Warm,High, Strong,Cool,Change ,Yes>
S3 : {<Sunny, Warm, ?, Strong, Warm, Same> }
S4 : {<Sunny, Warm, ?, Strong, ?, ?> }
G4 : { <Sunny, ?, ?, ?, ?, ?>, <?, ?, Warm, ?, ?, ?>}
G3 : { <Sunny, ?, ?, ?, ?, ?>, <?, ?, Warm, ?, ?, ?>, <?, ?, ?, ?, ?, Same> }

47. Example Trace S0 G0 S1 = G1 S2 = G2 = S3 G3 S4 G4
d1: <Sunny, Warm, Normal, Strong, Warm, Same, Yes>
<Ø, Ø, Ø, Ø, Ø, Ø> S0
<?, ?, ?, ?, ?, ?> G0
<Sunny, Warm, Normal, Strong, Warm, Same> S1
= G1
d2: <Sunny, Warm, High, Strong, Warm, Same, Yes>
d3: <Rainy, Cold, High, Strong, Warm, Change, No>
<Sunny, Warm, ?, Strong, Warm, Same> S2
= G2
d4: <Sunny, Warm, High, Strong, Cool, Change, Yes>
= S3
<Sunny, ?, ?, ?, ?, ?> <?, Warm, ?, ?, ?, ?> <?, ?, ?, ?, ?, Same> G3
<Sunny, Warm, ?, Strong, ?, ?> S4 G4
<Sunny, ?, ?, ?, ?, ?>
<?, Warm, ?, ?, ?, ?>
<Sunny, ?, ?, Strong, ?, ?>
<Sunny, Warm, ?, ?, ?, ?>
<?, Warm, ?, Strong, ?, ?>

48. The hypothesis space after four cases

49. The hypothesis space after four cases

50. Remarks on Candidate Elimination
What training example should the learner request next?
Will the CE algorithm converge to the correct hypothesis ?
How can partially learned concepts be used?

51. What Next Training Example?

52. Who Provides Examples?
What training example should the learner request next?
Two methods
Fully supervised learning: External teacher provides all training examples (input + correct output)
Learning by query: The learner generates instances (queries) by conducting experiments, then obtains the correct classification for this instance from an external oracle (nature or a teacher).
Negative training examples specializes G, positive ones generalize S.

53. When Does CE Converge?
Will the Candidate-Elimination algorithm converge to the correct hypothesis?
Prerequisites
1. No error in training examples
2. The target hypothesis exists which correctly
describes c(x).
If S and G boundary sets converge to an empty set, this means there is no hypothesis in H consistent with observed examples.
(S와 G 모두 empty set에 이르게 된다면 학습된 예제들과는 부합하는 hypothesis이 없다는 것을 의미한다.)

54. Can a partially learned classifier be used?

55. How to Use Partially Learned Concepts?
{<Sunny,Warm,?,Strong,?,?>}
<Sunny,?,?,Strong,?,?>
<Sunny,Warm,?,?,?,?>
<?,Warm,?,Strong,?,?> G:
{<Sunny,?,?,?,?,?>, <?,Warm,?,?,?>, }
Suppose the learner is asked to classify the four new instances
shown in the following table.
Instance Sky AirTemp Humidity Wind Water Forecast EnjoySport
A Sunny Warm Normal Strong Cool Change /0
B Rainy Cold Normal Light Warm Same /6
C Sunny Warm Normal Light Warm Same ? 3/3
D Sunny Cold Normal Strong Warm Same ? 2/4

56. Can a partially learned classifier be used?

57. Example Candidate Elimination
Instance space: integer points in the x,y plane with
0  x,y  10
hypothesis space : rectangles, that means hypotheses
are of the form a  x  b , c  y  d , assume d c a b
Adapted by Doug Downey from: Bryan Pardo, EECS 349 Fall 2007 57

58. Example Candidate Elimination
examples = {ø}
G={0,10,0,10}
S={ø} G

59. Example Candidate Elimination
examples = {(3,4),+}
G={(0,10,0,10)}
S={(3,3,4,4)} G S +

60. An UNBiased Learner
Idea: Choose H that expresses every teachable concept (i.e., H is the
power set of X)
Consider H' = disjunctions, conjunctions, negations over previous H.
E.g.,
<Sunny Warm Normal ???> ∨<?????Change>
What are S, G in this case?
S ←
G ←

61. Unbiased Learner Assume positive examples (x1, x2, x3) and
negative examples (x4, x5)
G : {  (x4 v x5) }
S : { (x1 v x2 v x3) }
How would we classify some new instance x6?
For any instance not in the training examples
half of the version space says +
the other half says –
=> To learn the target concept, one would have to present every single instance in X as a training example (Rote learning)

62. What Justifies this Inductive Leap?
New examples
d1: <Sunny Warm Normal Strong Cool Change>, Yes
d2: <Sunny Warm Normal Light Warm Same>, No
S : {<Sunny Warm Normal ? ? ?> +}
Overly general hypothesis space

63. Inductive bias Our hypothesis space is unable to represent a simple
disjunctive target concept :
(Sky=Sunny) v (Sky=Cloudy)
x1 = <Sunny Warm Normal Strong Cool Change> +
S1 : { <Sunny, Warm, Normal, Strong, Cool, Change> }
x2 = <Cloudy Warm Normal Strong Cool Change> +
S2 : { <?, Warm, Normal, Strong, Cool, Change> }
x3 = <Rainy Warm Normal Strong Cool Change> -
S3 : {}
The third example x3 contradicts the already overly
general hypothesis space specific boundary S2.
Overly general hypothesis space

64.  Why Inductive learning hypothesis ?
Why believe we can classify the unseen ?
S : {<Sunny Warm Normal ? ? ?> +}
Unseen example: <Sunny Warm Normal Strong Warm Same>, ….
 Why Inductive learning hypothesis ?

65. Inductive bias Example
The inductive bias of a learning algorithm is the set of assumptions that the learner uses to predict outputs given inputs that it has not encountered (Mitchell, 1980).
Example
Occam’s Razor
Target concept c ∈ H (hypothesis space) of candidate-elimination algorithm

66. Inductive Bias Consider Definition: concept learning algorithm L
instances X, target concept c
training examples Dc = {<x, c(x)>}
let L(xi, Dc) denote the classification assigned to the instance xi by L after training on data Dc.
Definition:
The inductive bias of L is any minimal set of assertions B such
that for any target concept c and corresponding training
examples Dc
(∀xi ∈ X)[(B ∧ Dc ∧ xi) ├ L(xi, Dc)]
where A├ B means A logically entails B

67. Inductive bias II

68. Inductive bias II

69. Inductive Systems and Equivalent Deductive Systems
Candidate Elimination
Algorithm
Using Hypothesis
Space H
Inductive System
Training Examples
New Instance
Classification of New Instance
(or “Don’t Know”)
Theorem Prover
Equivalent Deductive System
Training Examples
New Instance
Assertion { c  H }
Classification of New Instance
(or “Don’t Know”)
Inductive bias made explicit

70. Three Learners with Different Biases
Rote Learner
Weakest bias: anything seen before, i.e., no bias
Store examples
Classify x if and only if it matches previously observed example
Version Space Candidate Elimination Algorithm
Stronger bias: concepts belonging to conjunctive H
Store extremal generalizations and specializations
Classify x if and only if it “falls within” S and G boundaries (all members agree)
Find-S
Even stronger bias: most specific hypothesis
Prior assumption: any instance not observed to be positive is negative
Classify x based on S set

71. Summary Points 1. Concept learning as search through H
2. General-to-specific ordering over H
3. Version space candidate elimination algorithm
4. S and G boundaries characterize learner’s uncertainty
5. Learner can generate useful queries
6. Inductive leaps possible only if learner is biased
7. Inductive learners can be modelled by equivalent deductive systems