Chapter 8 Discriminant Analysis

8.1 Introduction  Classification is an important issue in multivariate analysis and data mining.  Classification: classifies data (constructs a model) based on the training set and the values (class labels) in a classifying attribute and uses it in classifying new data, i.e., predicts unknown or missing values

Classification — A Two-Step Process  Model construction: describing a set of predetermined classes Each tuple/sample is assumed to belong to a predefined class, as determined by the class label attribute The set of tuples used for model construction is training set The model is represented as classification rules, decision trees, or mathematical formulae  Prediction: for classifying future or unknown objects Estimate accuracy of the model The known label of test sample is compared with the classified result from the model Accuracy rate is the percentage of test set samples that are correctly classified by the model Test set is independent of training set, otherwise over-fitting will occur If the accuracy is acceptable, use the model to classify data tuples whose class labels are not known

Classification Process : Model Construction Training Data Classification Algorithms IF rank = ‘professor’ OR years > 6 THEN tenured = ‘yes’ Classifier (Model)

Classification Process: Use the Model in Prediction Classifier Testing Data Unseen Data (Jeff, Professor, 4) Tenured?

Supervised vs. Unsupervised Learning  Supervised learning (classification) Supervision: The training data (observations, measurements, etc.) are accompanied by labels indicating the class of the observations New data is classified based on the training set  Unsupervised learning (clustering) The class labels of training data is unknown Given a set of measurements, observations, etc. with the aim of establishing the existence of classes or clusters in the data

Discrimination— Introduction Discrimination is a technique concerned with allocating new observations to previously defined groups. There are k samples from k distinct populations: One wants to find the so-called discriminant function and related rule to identify the new observations.

Example 11.3 Bivariate case

Discriminant function and rule

Example 11.1: Riding mowers Consider two groups in city: riding-mower owners and those without riding mowers. In order to identify the best sales prospects for an intensive sales campaign, a riding-mower manufacturer is interested in classifying families as prospective owners or non- owners on the basis of income and lot size.

Example 11.1: Riding mowers

8.2 Discriminant by Distance Assume k=2 for simplicity

Consider the Mahalanobis distance 8.2 Discriminant by Distance

Let 8.2 Discriminant by Distance

Example Univariate Case with equal variance a

a* Example Univariate Case with equal variance

8.3 Fisher’s Discriminant Function Idea: projection, ANOVA

Training samples 8.3 Fisher’s Discriminant Function

Projection the data on a direction, the F-statistics where 8.3 Fisher’s Discriminant Function

To find such that The solution of is the eigenvector associated with the largest eigenvalue of. Discriminant function: 8.3 Fisher’s Discriminant Function

(B) Two Populations Note We haveand There is only one non-zero eigenvalue of as

The associated eigenvector is where (B) Two Populations

Whenis replaced by where (B) Two Populations

Example Inset Classification Note:data x1 and x2 are the characteristics of insect (Hoel,1947) n.g.means natural group (species), c.g.the classified group, y the value of the discriminant function

The eigenvalue of is and the associated eigenvector is Example Inset Classification

The discriminant function is and the associated value of each observation is given in the table. The cutting point is Classification is If we use, we have the same classification. Example Inset Classification

8.4 Bayes’ Discriminant Analysis A.Idea There are k populations G 1, …, G k in R p. A partition of R p, R 1, …, R k, is determined based on a training sample. Rule:if falls into R i Loss: is from G i, but falls into R j The Probability of this misclassification whereis the density of.

Expected cost of misclassification is where q 1, …, q k are prior probabilities. We want to minimize ECM(R 1, …, R k ) w.r.t. R 1, …, R k. 8.4 Bayes’ Discriminant Analysis

Theorem Let Then the optimal R t ’s are B. Method

Take if and 0 if. Then Proof: Corollary 1

In the case of k=2 we have Corollary 2

In the case of k=2 and Corollary 3

Then

C. Example 11.3: Detection of hemophilia A carriers For the detection of hemophilia A carriers, to construct a procedure for detecting potential hemophilia A carriers, blood samples were assayed for two groups of women and measurements on the two variables. The first group of 30 women were selected from a population of women who did not carry the hemophilia gene. This group was called the normal group. The second group of 22 women was selected from known hemophilia A carriers. This group was called the obligatory carriers.

Variables: log 10 (AHF activity) log 10 (AHF-like antigen) Populations:population of women who did not carry the hemophilia gene (n 1 =30) population of women who are known hemophilia A carriers (n 2 =45) hemophilia A carriers (n 2 =45) C. Example 11.3: Detection of hemophilia a carriers

C. Example 11.3: Detection of hemophilia a carriers

Data set normal log10(AHF activity) log10(AHF-like antigen) Obligatorycarrier log10(AHF activity) log10(AHF-like antigen) C. Example 11.3: Detection of hemophilia a carriers

SAS output C. Example 11.3: Detection of hemophilia a carriers

C. Example 11.3: Detection of hemophilia a carriers

C. Example 11.3: Detection of hemophilia a carriers

C. Example 11.3: Detection of hemophilia a carriers