1. CSE543: Machine Learning Lecture 2: August 6, 2014

2. Example
Labels

3. Example Learning Phase 2/9 3/5 4/9 0/5 3/9 2/5 2/9 2/5 4/9 3/9 1/5 3/9
Outlook
Play=Yes
Play=No
Sunny
2/9
3/5
Overcast
4/9
0/5
Rain
3/9
2/5
Temperature
Play=Yes
Play=No
Hot
2/9
2/5
Mild
4/9
Cool
3/9
1/5
Humidity
Play=Yes
Play=No
High
3/9
4/5
Normal
6/9
1/5
Wind
Play=Yes
Play=No
Strong
3/9
3/5
Weak
6/9
2/5
P(Play=Yes)
= 9/14
P(Play=No)
= 5/14

4. Test Phase Given a new instance
x’=(Outlook=Sunny, Temperature=Cool, Humidity=High, Wind=Strong)
Is Play possible?

5. Test Phase Given a new instance
x’=(Outlook=Sunny, Temperature=Cool, Humidity=High, Wind=Strong)
Is Play possible?
This is the task of ML algorithms …
Lets start with CONCEPT LEARNING

6. Concept Learning
Concept learning can be formulated as a problem of searching through a predefined space of potential hypotheses for the hypothesis that best fits the training examples.
Inferring a boolean-valued function from training examples of its input and output: Learning Concepts from Examples
“Concept” typically means categorization based on features

7. Concept Learning Task
Days on which Aldo enjoys his favorite water sport … we need to learn this task ..
The task is -> EnjoySport – i.e. Aldo likes his favorite sport
Training set
Example
Sky
AirTemp
Humidity
Wind
Water
Forecast
Enjoy Sport 1
Sunny
Warm
Normal
Strong
Same
Yes 2
High 3
Rainy
Cold
Change No 4
Cool

8. Concept Learning Task Target concept:
Example
Sky
AirTemp
Humidity
Wind
Water
Forecast
Enjoy Sport 1
Sunny
Warm
Normal
Strong
Same
Yes 2
High 3
Rainy
Cold
Change No 4
Cool
Concept Learning Task
Target concept:
“Days on which Aldo enjoys his favourite water sport”
Two categories: yes/no
Some possible concepts:
Enjoy sport when Sky = Sunny
Enjoy sport when Airtemp = Warm

9. Concept Learning Task
In Concept learning there is a notion of all possible hypotheses and best hypothesis
Lets talk about how we generate hypothesis
“?” represents that any value is acceptable
we specify a single required value
 for no value is acceptable
If some instance x satisfies all the constraints of hypothesis h, then h classifies x as a positive example (h(x) = 1)

10. How do we represent hypotheses
Sample hypothesis:
H1: Enjoy sport only on cold days with high humidity
(?,cold, high, ?,?,?)
H2: Most general hypothesis - everyday is positive day for game
(?,?,?,?,?,?)
H3: Most specific possible hypothesis – no day is good day for game
(, , , , , )

11. EnjoySport Concept Learning Task
Given
Instances X: Possible days, described by attributes
Sky (with possible values Sunny, Cloudy, and Rainy)
AirTemp (with values Warm and Cold)
Humidity (with values Normal and High)
Wind (with values Strong and Weak)
Water (with values Warm and Cool)
Forecast (with values Same and Change)

12. EnjoySport Concept Learning Task
Hypotheses H: Each hypothesis is described by a conjunction of the attributes Sky, AirTemp, Humidity, Wind, Water, and Forecast
The constraints may be "?" (any value is acceptable), "∅" (no value is acceptable), or a specific value

13. EnjoySport Concept Learning Task
Target concept c: EnjoySport: X→{0,1}
Training examples D <x, c(x)>: Positive and negative examples of target function
For positive examples, c(x) = 1 and for negative examples, c(x) = 0

14. EnjoySport Concept Learning Task
Determine
A hypothesis h such that h(x)=c(x) for all x in X

15. EnjoySport Concept Learning Task
Sky (with possible values Sunny, Cloudy, and Rainy)
AirTemp (with values Warm and Cold)
Humidity (with values Normal and High)
Wind (with values Strong and Weak)
Water (with values Warm and Cool)
Forecast (with values Same and Change)
What is the size of instance space
How many different hypothesis exist?
Syntactically?
Semantically?

16. EnjoySport Concept Learning Task
Sky (with possible values Sunny, Cloudy, and Rainy)
AirTemp (with values Warm and Cold)
Humidity (with values Normal and High)
Wind (with values Strong and Weak)
Water (with values Warm and Cool)
Forecast (with values Same and Change)
What is the size of instance space: 96
How many different hypothesis exist?:
Syntactically? : 5120
Semantically? : 973

17. Inductive Learning Hypothesis
Inductive Learning: Learning by example – the system tries to induce a general rule by the set of observed examples
Inductive Learning Hypothesis: Any hypothesis found to approximate the target function well over a sufficiently large set of training examples will also approximate the target function well over other unobserved samples

18. How you create hypothesis from instances

19. FIND-S Algorithm Initialize h to the most specific hypothesis in H
For each positive training instance x
For each attribute constraint ai in h
If the constraint ai is satisfied by x
Then do nothing
Else replace ai in h by the next more general constraint that is satisfied by x
Output hypothesis h

20. Step1: h<-- (, , , , , )
Example
Sky
AirTemp
Humidity
Wind
Water
Forecast
EnjoySport 1
Sunny
Warm
Normal
Strong
Same
Yes 2
High 3
Rainy
Cold
Change No 4
Cool
Step1: h<-- (, , , , , )
Upon observing the first training example, current hypothesis is too specific. So replace by the next more general constraint
h<--<Sunny,Warm,Normal,Strong,Warm,Same>

21. After observing the second training sample, the hypothesis becomes:
Example
Sky
AirTemp
Humidity
Wind
Water
Forecast
EnjoySport 1
Sunny
Warm
Normal
Strong
Same
Yes 2
High 3
Rainy
Cold
Change No 4
Cool
After observing the second training sample, the hypothesis becomes:
h<--<Sunny,Warm,?,Strong,Warm, Same>
Ignore the third sample (which is negative)
After the fourth sample
h<--<Sunny,Warm,?,Strong,?>

22. Find S Algorithm

23. Hypothesis found by FIND-S
h<--<Sunny,Warm,?,Strong,?>
Example
Sky
AirTemp
Humidity
Wind
Water
Forecast
EnjoySport 1
Sunny
Warm
Normal
Strong
Same
Yes 2
High 3
Rainy
Cold
Change No 4
Cool

24. Any other hypothesis for same data?
There are other hypotheses consistent with the data, e.g., <Sunny,?,?,Strong,?,?>
Example
Sky
AirTemp
Humidity
Wind
Water
Forecast
EnjoySport 1
Sunny
Warm
Normal
Strong
Same
Yes 2
High 3
Rainy
Cold
Change No 4
Cool
Find-S finds the most specific hypothesis in H which is consistent with the positive examples

25. Major drawback of FIND-S
No negative training
What happens with a negative test case?

26. Major drawback of FIND-S
No negative training
What happens with a negative test case?
Can we come up with a better algorithm?

27. Version Spaces and The Candidate Elimination Algorithm
Next Class:
Version Spaces and The Candidate Elimination Algorithm