Presentation is loading. Please wait.

Presentation is loading. Please wait.

Introduction to Artificial Intelligence CS440/ECE448 Lecture 21

Published byRobin Conant Modified over 10 years ago

Similar presentations

Presentation on theme: "Introduction to Artificial Intelligence CS440/ECE448 Lecture 21"— Presentation transcript:

1 Introduction to Artificial Intelligence CS440/ECE448 Lecture 21
Learning II Introduction to Artificial Intelligence CS440/ECE448 Lecture 21

2 Last lecture This lecture
The (in)efficiency of exact inference with Bayes nets The learning problem Decision trees This lecture Identification trees Neural networks: Perceptrons Reading Chapters 18 and 20

3

4

5

6 Inductive learning method
Construct/adjust h to agree with f on training set. (h is consistent if it agrees with f on all examples) E.g., curve fitting:

7 Inductive learning method
Construct/adjust h to agree with f on training set. (h is consistent if it agrees with f on all examples) E.g., curve fitting:

8 Inductive learning method
Construct/adjust h to agree with f on training set. (h is consistent if it agrees with f on all examples) E.g., curve fitting:

9 Inductive learning method
Construct/adjust h to agree with f on training set. (h is consistent if it agrees with f on all examples) E.g., curve fitting: Ockham’s razor: prefer the simplest consistent hypothesis.

10 Inductive Learning Given examples of some concepts and a description (features) for these concepts as training data, learn how to classify subsequent descriptions into one of the concepts. Concepts (classes) Features Training set Test set Here, the function has discrete outputs (the classes).

11 “Should I play tennis today?”
Decision Trees “Should I play tennis today?” Outlook Humidity Wind No Yes Sunny Rain Overcast High Low Strong Weak Note: A decision tree can be expressed as a disjunction of conjunctions (Outlook = sunny) (Humidity = normal)  (Outlook = overcast)  (Wind=Weak)

12 Learning Decision Trees
Inductive Learning. Given a set of positive and negative training examples of a concept, can we learn a decision tree that can be used to appropriately classify other examples? Identification Trees: ID3 [ Quinlan, 1979 ].

13 What on Earth causes people to get sunburns?
I don’t know, so let’s go to the beach and collect some data.

14 There are 3 x 3 x 3 x 2 = 54 possible feature vectors
Sunburn data Name Hair Height Swim Suit Color Lotion Result Sarah Blond Average Yellow No Sunburned Dana Tall Red Yes Fine Alex Brown Short Annie Emily Blue Pete John Katie There are 3 x 3 x 3 x 2 = 54 possible feature vectors

15 Exact Matching Method  106/512 0.4%
Construct a table recording observed cases. Use table lookup to classify new data. Problem: For realistic problems, exact matching can’t be used. 8 people and 54 possible feature vectors:  15% chance of finding an exact match. Another example: 106 Examples 12 features 5 values per feature  106/512 0.4%

16 How can we do the classification?
Nearest-neighbor method (but only if we can establish a distance between feature vectors). Use identification trees: An identification tree is a decision tree in which each set of possible conclusions is implicitly established by a list of samples of known class.

17 An ID tree consistent with the data
Hair Color Blond Brown Red Alex Pete John Emily Lotion Used Yes No Sarah Annie Dana Katie Sunburned Not Sunburned

18 Another consistent ID tree
Height Sunburned Not Sunburned Tall Dana Pete Short Average Hair Color Suit Color Brown Red Red Blond Yellow Blue Alex Hair Color Suit Color Sarah Red Blue Red Yellow Brown Annie Emily Katie John

19 An idea Select tests that divide as well as possible people into sets with homogeneous labels
Hair Color Lotion used No Blond Yes Red Brown Sarah Annie Emily Pete John Sarah Annie Dana Katie Dana Alex Katie Alex Pete John Emily Height Suit Color

20 Then among blonds... This is perfectly homogeneous... Height Katie
Annie Sarah Dana Short Av Tall Lotion used Sarah Annie Dana Katie No Yes This is perfectly homogeneous... Suit Color Sarah Katie Dana Annie Yellow Red Blue

21 Combining these two together …
Hair Color Blond Brown Red Alex Pete John Emily Lotion Used Yes No Sarah Annie Dana Katie Sunburned Not Sunburned

22 Decision Tree Learning Algorithm

23 Problem: For practical problems, it is unlikely that any test will produce one completely homogeneous subset. Solution: Minimize a measure of inhomogeneity or disorder. Available from information theory.

24 Information Let’s say we have a question which has n possible answers and call them vi. Let’s say that answer vi occurs with probability P(vi), then the information content (entropy) measured in bits of knowing the answer is: One bit of information is enough information to answer a yes or no question. E.g. consider flipping a fair coin, how much information do you have if you know which side comes up? I(½, ½) = - (½ log2½ + ½ log2½) = 1bit

25 Information at a node In our decision tree for a given feature (e.g. hair color), we have b: number of branches (e.g. possible values for the feature) Nb: number of samples in branch Np: number of samples in all branches Nbc: number of samples in class c in branch b. Using frequencies as an estimate of the probabilities, we have For a single branch, the information is simply

26 Example Consider a single branch (b=1) which only contains members of two classes A and B. If half of the points belong to A and half belong to B: What if all the points belong to A (or to B): We like the latter situation since the branches are homogeneous, so less information is needed to make a decision (maximize information gain).

27 Information = 4/8*1 + 1/8*0 + 3/8*0 = 0.5
What is the amount of information required for classification after we have used the hair test? Hair Color Blond Red Brown Sarah Annie Dana Katie Alex Pete John Emily -1 log21 -0 log20 = 0 - 0 log20 - 3/3 log23/3 = 0 - 2/4 log22/4 = 1 Information = 4/8*1 + 1/8*0 + 3/8*0 = 0.5

28 Selecting top level feature
Using the 8 samples we have so far, we get: Test Information Hair Height Suit Color Lotion Hair wins, least additional information needed for rest of classification. This is used to build the first level of the identification tree: Hair Color Sarah Annie Dana Katie Emily Alex Pete John Blond Red Brown

29 Selecting second level feature
Hair Color Sarah Annie Dana Katie Emily Alex Pete John Blond Red Brown Let’s consider the remaining features for the blond branch (4 samples) Test Information Height Suit Color 1 Lotion Lotion wins, least additional information.

30 Thus we get to the tree we had arrived at earlier
Hair Color Blond Brown Red Alex Pete John Emily Lotion Used Yes No Sarah Annie Dana Katie Sunburned Not Sunburned

31 Using the Identification tree as a classification procedure
Hair Color Blond Red Brown Lotion Used OK Sunburn Yes No Sunburn OK Rules: If Blond and uses lotion, then OK If Blond and does not use lotion, then gets burned If red-haired, then gets burned If brown hair, then OK

32 Performance measurement
How do we know that h ≈ f ? Use theorems of computational/statistical learning theory Try h on a new test set of examples (use same distribution over example space as training set) Learning curve = % correct on test set as a function of training set size

Download ppt "Introduction to Artificial Intelligence CS440/ECE448 Lecture 21"

Similar presentations

Learning from Observations Chapter 18 Section 1 – 3.

Learning from Observations Chapter 18 Section 1 – 3.
Data Mining Lecture 9.

Data Mining Lecture 9.
1 Machine Learning: Lecture 3 Decision Tree Learning (Based on Chapter 3 of Mitchell T.., Machine Learning, 1997)

1 Machine Learning: Lecture 3 Decision Tree Learning (Based on Chapter 3 of Mitchell T.., Machine Learning, 1997)
Decision Tree Learning - ID3

Decision Tree Learning - ID3
Decision Trees Decision tree representation ID3 learning algorithm

Decision Trees Decision tree representation ID3 learning algorithm
1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006 3. Decision Tree Learning 3.1 Introduction –Method for approximation of discrete-valued target functions.

1er. Escuela Red ProTIC - Tandil, de Abril, Decision Tree Learning 3.1 Introduction –Method for approximation of discrete-valued target functions.
Decision Tree Approach in Data Mining

Decision Tree Approach in Data Mining
Data Mining Classification: Alternative Techniques

Data Mining Classification: Alternative Techniques
ICS320-Foundations of Adaptive and Learning Systems

ICS320-Foundations of Adaptive and Learning Systems
Machine Learning Decision Trees. Exercise Solutions.

Machine Learning Decision Trees. Exercise Solutions.
Decision Tree Learning 主講人：虞台文 大同大學資工所 智慧型多媒體研究室.

Decision Tree Learning 主講人：虞台文 大同大學資工所 智慧型多媒體研究室.
Machine learning learning... a fundamental aspect of intelligent systems – not just a short-cut to kn acquisition / complex behaviour.

Machine learning learning... a fundamental aspect of intelligent systems – not just a short-cut to kn acquisition / complex behaviour.
Information Extraction Lecture 6 – Decision Trees (Basic Machine Learning) CIS, LMU München Winter Semester 2014-2015 Dr. Alexander Fraser, CIS.

Information Extraction Lecture 6 – Decision Trees (Basic Machine Learning) CIS, LMU München Winter Semester Dr. Alexander Fraser, CIS.
1 Chapter 10 Introduction to Machine Learning. 2 Chapter 10 Contents (1) l Training l Rote Learning l Concept Learning l Hypotheses l General to Specific.

1 Chapter 10 Introduction to Machine Learning. 2 Chapter 10 Contents (1) l Training l Rote Learning l Concept Learning l Hypotheses l General to Specific.
Learning: Identification Trees Larry M. Manevitz All rights reserved.

Learning: Identification Trees Larry M. Manevitz All rights reserved.
Decision Tree Learning

Decision Tree Learning
Induction of Decision Trees

Induction of Decision Trees
LEARNING DECISION TREES

LEARNING DECISION TREES
Decision Trees IDHairHeightWeightLotionResult SarahBlondeAverageLightNoSunburn DanaBlondeTallAverageYesnone AlexBrownTallAverageYesNone AnnieBlondeShortAverageNoSunburn.

Decision Trees IDHairHeightWeightLotionResult SarahBlondeAverageLightNoSunburn DanaBlondeTallAverageYesnone AlexBrownTallAverageYesNone AnnieBlondeShortAverageNoSunburn.
MACHINE LEARNING. 2 What is learning? A computer program learns if it improves its performance at some task through experience (T. Mitchell, 1997) A computer.

MACHINE LEARNING. 2 What is learning? A computer program learns if it improves its performance at some task through experience (T. Mitchell, 1997) A computer.

Similar presentations

Ads by Google