A discriminant function for 2-class problem can be defined as the ratio of class likelihoods g(x) = p(x|C1)/p(x|C2) Derive formula for g(x) when class likelihoods are Gaussian

A discriminant function for 2-class problem is defined as the Bayesian odds ratio g(x) = log(P(C1|x)/P(C2|x)) Derive a formula for g(x) when class likelihoods are Gaussian

Discriminant functions for 2-class problem are defined as gi(x) = log(P(Ci|x)) i=1,2 Derive a formula for gi(x) when class likelihoods are Gaussian Derive a formula for Bayesian discriminant points by setting g1(x) = g2(x) and solving for x

Duda and Hart model applied to 2-class problem R(a1|x) = l11 P(C1|x) + l12 P(C2|x) R(a2|x) = l21 P(C1|x) + l22 P(C2|x) l11 = l22 = 0 No cost for correct decisions l12 = 10, and l21 = 1 Cost incorrect assignment to C1 is ten times greater than cost of incorrect assignment to C2 Posteriors are normalized Derive the classification rule in terms of P(C1|x)

Example: Bernoulli distribution Two states, failure & success, x is {0,1} P (x) = pox (1 – po ) (1 – x) po = probability of success L (po|X) = log( ∏t poxt (1 – po ) (1 – xt) ) Show that po = ∑t xt / N = successes/trial Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) 5

A simple example constrained optimization using Lagrange multipliers find the stationary point of f(x1, x2) = 1 - x12 – x22 subject to the constraint g(x1, x2) = x1 + x2 - 1 = 0

X is a training set for a classification problem with K>2 classes xt is a scalar, rt is Boolean vector Class likelihoods are assumed to be Gaussian Write formulas for MLEs of priors, means and variance in terms of ni, the number of examples in class i

For a 1D, 2-class problem with Gaussian class likelihoods, derive the functional form of P(C1|x) when the following are true: (1) variances and priors are equal, (2) posteriors are normalized Hint: start with the ratio of posteriors to eliminate priors and evidence posterior likelihood prior evidence

Maximum likelihood estimation of g(x,q) Log Likelihood simplify this log-likelihood function as much as possible Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) 9

Independent Attributes If xi are independent, off-diagonals of ∑ are 0, p(x) reduces to a product of probabilities for each component Simpliy this expression when d=2 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) 10

2 attributes and 2 classes What can we say from this graph about the value of r in the covariance matrices of the 2 classes? Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) 11

2 attributes and 2 classes: What is the value of r in each example?

2 attributes 2 classes Same mean different Covariance One contour shown for each class likelihood Discriminant is dark blue Describe the covariance matrices in each case