Concept Learning Machine Learning by T. Mitchell (McGraw-Hill) Chp. 2

Much of learning involves acquiring general concepts from specific training examples e.g. what is a bird? what is a chair? Concept learning: Inferring a boolean-valued function from training examples of its input and output.

A Concept Learning Task Example Target Concept “Days on which Aldo enjoys his favorite water sport” (you may find it more intuitive to think of “Days on which the beach will be crowded” concept) Task Learn to predict the value of EnjoySport/Crowded for an arbitrary day Training Examples for the Target Concept 6 attributes (Nominal-valued (symbolic) attributes): Sky (SUNNY, RAİNY, CLOUDY), Temp (WARM,COLD), Humidity (NORMAL, HIGH), Wind (STRONG, WEAK), Water (WARM, COOL), Forecast (SAME, CHANGE)

A Learning Problem x1 Unknown Function x2 y = f (x1, x2, x3, x4 ) x3 x4 Hypothesis Space (H): Set of all possible hypotheses that the learner may consider during learning the target concept. How many?

Hypothesis Space: Unrestricted Case | A  B | = | B | | A | |H4  H | = | {0,1}  {0,1}  {0,1}  {0,1}  {0,1} | = 224 = 65536 function values After 7 examples, still have 29 = 512 possibilities (out of 65536) for f Is learning possible without any assumptions?

A Concept Learning Task In this notation, a conjunction such as <warm,?,stong> indicaties A1=Warm AND A3=strong Hypothesis h: Conjunction of Constraints on Attributes Constraint Values: Specific value (e.g., Water = Warm) All values allowed for that attribute (e.g., “Water = ?”) No value allowed for that attribute (e.g., “Water = Ø”) Hypothesis Representation – Example Hypothesis for EnjoySport: ‘’Aldo enjoys his favorite sport only on sunny days with Strong wind’’: Sky AirTemp Humidity Wind Water Forecast <Sunny ? ? Strong ? ? > The most general hypothesis – every day is a positive example of this concept <?, ?, ?, ?, ?, ?> A very specific hypothesis – no day is a positive ex. of this concept < Sunny, Warm, Normal, Strong, Warm, Same> The most specific possible hypothesis – no day is a positive ex. of this concept <  ,, , , , > Is this hypothesis consistent with the training examples? What are some hypotheses that are consistent with the examples?

A Concept Learning Task(2) The instance space, X (book uses ‘’set of instances’’) all possible days represented by attributes Sky, AirTemp,... Target concept, c : Any boolean-valued function defined over the instance space X c : X  {0, 1} (I.e. if EnjoySport = Yes, then c(x) = 1 ) Training Examples (denoted by D): ordered pair <x, c(x)> Positive example: member of the target concept, c(x) = 1 Negative example: nonmember of the target concept, c(x) = 0 Assumption: no missing X values No noise in values of c (contradictory labels). Hypotheses Space, H: Often picked by the designer, for instance only conjunctions (e.g. A1  A5) H is the set of Boolean valued functions defined over X Or you may narrow it down to conjunction of constraints on attributes The goal of the learner Hypothesize or estimate c: Find a hypothesis h such that h(x) = c(x) for all x in X

A Concept Learning Task(3) Although the learning task is to determine a hypothesis h identical to c, over the entire set of instances X, the only information available about c is its value over the training instances D. Inductive Learning Hypothesis Any hypothesis found to approximate the target function well over a sufficiently large set of training examples will also approximate the target function well over other unobserved examples.

Concept Learning As Search Concept learning can be viewed as the task of searching through a large space of hypotheses implicitly defined by the hypothesis representation. The goal of this search is to find the hypothesis that (best) fits the training examples. Sky AirTemp Humidity Wind Water Forecast <Sunny/Rainy/Cloudy Warm/Cold Normal/High Weak/Strong Warm/Cold Change/Same > EnjoySport Learning Task Size of the instance space X 3  2  2  2  2  2 = 96 Syntactically distinct hypotheses (including ?, Ø) 5  4  4  4  4  4 = 5120 Semantically distinct hypotheses (+1 for no positive example) 1 + (4  3  3  3  3  3) = 973 Often much larger, sometimes infinite hypotheses spaces

Inductive Bias A biased Hypothesis space EnjoySport example Restriction : only conjunctions of attribute values. No representation for a disjunctive target concept (Sky = Sunny) or (Wind = Weak) Potential problem We biased the learner (inductive bias) to consider only conjunctive hypotheses But the concept may require a more expressive hypothesis space

UnBiased Learner An Unbiased Learner Obvious solution: Provide a hypothesis space capable of representing every teachable concept = every possible subset of the instance space X The set of all subsets of a set X is called the power set of X EnjoySport Example |Instance space| : 96 |Power set of X| : 296 = 79228162514264337593543950336 |Conjunctive hypothesis space| : 973 Very biased hypothesis space indeed!!

Need for Inductive Bias An Unbiased Learner Reformulate the EnjoySport learning task in an unbiased way Defining a new hypothesis space H’ that can represent every subset of X Allow arbitrary disjunctions, conjunctions, and negations Example: “Sky = Sunny or Wind = Weak” <Sunny, ?, ?, ?, ?, ?> OR <?, ?, ?, Weak, ?, ?> New problem: We will be completely unable to generalize beyond the observed samples! Explanation: If we allow full flexibility for the concept (conjunctions, disjunctions,…) , then even if one unseen sample is left, then there are still 2 hypotheses left that are consistent with the data (i.e. You still havent precisely found the right concept, eliminating all those that are inconsistent with the data you have seen!).

Need For Inductive Bias Fundamental property of inductive inference: A learner that makes no a priori assumptions regarding the identity of the target concept has no rational basis for classifying any unseen instances