Introduction to Machine Learning course fall 2007 Lecturer: Amnon Shashua Teaching Assistant: Yevgeny Seldin School of Computer Science and Engineering Hebrew University

What is Machine Learning? Inference engine (computer program) that when given sufficient data (examples) computes a function that matches as close as possible the process generating the data. Make accurate prediction based on observed data Algorithms to optimize a performance criterion based on observed data Learning to do better in the future based on what was experienced in the past Programming by examples: instead of writing a program to solve a task directly, machine learning seeks methods by which the computer will come up with its own program based on training examples.

Why Machine Learning? Data-driven algorithms are able examine large amounts of data. A human expert on the other hand is likely to be guided by subjective impressions or by examining a relatively small number of examples. Humans often have trouble expressing what they know but have no difficulty in labeling data Machine learning is effective in domains where declarative (rule based) knowledge is difficult to obtain yet generating training data is easy

Typical Examples Visual recognition (say, detect faces in an image): the amount of variability in appearance introduce challenges that are beyond the capacity of direct programming Spam filtering: data-driven programming can adapt to changing tactics by spammers Extract topics from documents: categorize news articles whether they are about politics, sports, science, etc. Natural language understanding: from spoken words to text; categorize the meaning of spoken sentences Optical character recognition (OCR) Medical diagnosis: from symptoms to diagnosis Credit card transaction fraud detection Wealth prediction

Fundamental Issues Over-fitting: doing well on a training set does not guarantee accuracy on new examples What is the resource we wish to optimize? For a given accuracy, use the smallest size training set Examples are drawn from some (fixed) distribution D over X x Y (instance space x output space). Does the learner actually need to recover D during the learning process? How does the learning process depend on the complexity of the family of learning functions (concept class C)? How does one define complexity of C? When the goal is to learn the joint distribution D then the problem is computationally unwieldy because the joint distribution table is exponentially large. What assumptions can be made to simplify the task?

Multiclass classification. K=2 is normally of most interest. Supervised vs. Un-supervised Supervised Learning Models: where X is the instance (data) space and Y is the output space Regression. Predict the price of a used car given brand, year, mileage.. Kinematics of a robot arm; navigate by determining steering angle from image input.. Un-supervised Learning Models: Find regularities in the input data assuming there is some structure in the input space Density estimation Clustering (non-parametric density estimation): divide customers to groups which have similar attributes.. Latent class models: extract topics from documents Compression: represent the input space with fewer parameters; projection to lower-dimensional spaces

Notations X is the instance space: space from which observations are drawn. Examples, input instance, a single observation. Examples, Y is the output space: set of possible outcomes that can be associated with a measurement. Examples, An example is an instance-label pair (x,y). If |Y|=2 one typically uses {0,1} or {-1,1}. We say that an example (x,y) is positive if y=1 and otherwise we call it a negative example A training set Z consists of m instance-label pairs: In some cases we refer to the training set without labels:

Separating hyperplanes: a concept h(x) is specified by a vector and a scalar b such that: Conjunction learning: a conjunction is a special case of a Boolean formula. A literal Is a variable or its negation and a term is a conjunction of literals, i.e. A target function is a term which consists of a subset of literals. In this case and Each is called a concept or hypothesis or classifier. Example, if Notations A concept (hypothesis) class C is a set (not necessarily finite) of functions of the form: Other examples: then C might be: Decision trees: when then any boolean function can be described by a binary tree. Thus, C consists of decision trees ( )

The Formal Learning Model Probably Approximate Correct (PAC) Distribution invariant: Learner does not need to estimate the joint distribution D over X x Y. Assumptions are that examples arrive i.i.d. and that D exists and is fixed. The training sample complexity (size of the training set Z) depends only the desired accuracy and confidence parameters - does not depend on D. Not all concept classes D are PAC-learnable. But some interesting classes are.

Unrealizable case: when and the training set is and D is over XxY Realizable case: when a target concept is known to lie inside C. In this case, the training set is sampled randomly and independently (i.i.d) according to some (unknown) Distribution D, i.e., S is distributed according to the product distribution Given a concept function is the probability that an instance x sampled according to D will be labeled incorrectly by h(x) PAC Model Definitions

given to the learner specifies desired accuracy, i.e. Note: in realizable case because given to the learner specifies desired confidence, i.e. The learner is allowed to deviate occasionally from the desired accuracy but only rarely so.. PAC Model Definitions

We will say that an algorithm L learns C if for every and for every D over XxY, L generates a concept function such that the probability that is at least PAC Model Definitions

from the set of all training examples to C with the following property: given any there is an integer such that if then, for any probability distribution D on XxY, if Z is a training set of length m drawn randomly according to, then with probability of at least then hypothesis is such that Formal Definition of PAC Learning A learning algorithm L is a function: We say that C is learnable (or PAC-learnable) if there is a learning algorithm for C

Formal Definition of PAC Learning does not depend on D, i.e., PAC model is distribution invariant The class C determines the sample complexity. For “simple” classes would be small compared to more “complex” classes. Notes:

Course Syllabus 3 x PAC: 2 x Separating Hyperplanes: Support Vector Machine, Kernels, Linear Discriminant Analysis 3 x Unsupervised Learning: Dimensionality Reduction (PCA), Density Estimation, Non-parametric Clustering (spectral methods) 5 x Statistical Inference: Maximum Likelihood, Conditional Independence, Latent Class Models, Expectation-Maximization Algorithm, Graphical Models