1. More Bayes, Decision trees, and cross-validation
Peter Fox and Greg Hughes
Data Analytics – ITWS-4600/ITWS-6600
Group 2 Module 7, February 21, 2017

2. Contents

3. Numeric v. non-numeric

4. In R – data frame and types
Almost always at input R sees categorical data as “strings” or “character”
You can test for membership (as a type) using is.<x> (x=number, factor, etc.)
You can “coerce” it (i.e. change the type) using as.<x> (same x)
To tell R you have categorical types (also called enumerated types), R calls them “factor”…. Thus – as.factor()

5. In R
factor(x = character(), levels, labels = levels, exclude = NA, ordered = is.ordered(x), nmax = NA), ordered(x, ...), is.factor(x), is.ordered(x), as.factor(x), as.ordered(x), addNA(x, ifany = FALSE) levels - values that x might have taken. labels - either an optional vector of labels for the levels (in the same order as levels after removing those in exclude), or a character string of length 1. Exclude - Ordered - Nmax - upper bound on the number of levels; Ifany -

6. Relate to the datasets…
Abalone - Sex = {F, I, M}
Eye color?
EPI - GeoRegions?
Sample v. population – levels and names IN the dataset versus all possible levels/names

7. Naïve Bayes – what is it?
Example: testing for a specific item of knowledge that 1% of the population has been informed of (don’t ask how).
An imperfect test:
99% of knowledgeable people test positive
99% of ignorant people test negative
If a person tests positive – what is the probability that they know the fact?

8. Naïve approach… We have 10,000 representative people
100 know the fact/item, 9,900 do not
We test them all:
Get 99 knowing people testing knowing
Get 99 not knowing people testing not knowing
But 99 not knowing people testing as knowing
Testing positive (knowing) – equally likely to know or not = 50%

9. Tree diagram 10000 ppl 1% know (100ppl) 99% test to know (99ppl)
1% test not to know (1per)
99% do not know (9900ppl)
1% test to know (99ppl)
99% test not to know (9801ppl)

10. Relation between probabilities
For outcomes x and y there are probabilities of p(x) and p (y) that either happened
If there’s a connection, then the joint probability = that both happen = p(x,y)
Or x happens given y happens = p(x|y) or vice versa then:
p(x|y)*p(y)=p(x,y)=p(y|x)*p(x)
So p(y|x)=p(x|y)*p(y)/p(x) (Bayes’ Law)
E.g. p(know|+ve)=p(+ve|know)*p(know)/p(+ve)= (.99*.01)/(.99* *.99) = 0.5

11. How do you use it?
If the population contains x what is the chance that y is true?
p(SPAM|word)=p(word|SPAM)*p(SPAM)/p(word)
Base this on data:
p(spam) counts proportion of spam versus not
p(word|spam) counts prevalence of spam containing the ‘word’
p(word|!spam) counts prevalence of non-spam containing the ‘word’

12. Or..
What is the probability that you are in one class (i) over another class (j) given another factor (X)?
Invoke Bayes:
Maximize p(X|Ci)p(Ci)/p(X) (p(X)~constant and p(Ci) are equal if not known)
So: conditional indep -

13. P(xk | Ci) is estimated from the training samples
Categorical: Estimate P(xk | Ci) as percentage of samples of class i with value xk
Training involves counting percentage of occurrence of each possible value for each class
Numeric: Actual form of density function is generally not known, so “normal” density (i.e. distribution) is often assumed

14. Digging into iris
classifier<-naiveBayes(iris[,1:4], iris[,5]) table(predict(classifier, iris[,-5]), iris[,5], dnn=list('predicted','actual')) classifier$apriori classifier$tables$Petal.Length plot(function(x) dnorm(x, 1.462, ), 0, 8, col="red", main="Petal length distribution for the 3 different species") curve(dnorm(x, 4.260, ), add=TRUE, col="blue") curve(dnorm(x, 5.552, ), add=TRUE, col = "green")

16. Bayes
> cl <- kmeans(iris[,1:4], 3) > table(cl$cluster, iris[,5]) setosa versicolor virginica # > m <- naiveBayes(iris[,1:4], iris[,5]) > table(predict(m, iris[,1:4]), iris[,5]) setosa versicolor virginica
pairs(iris[1:4],main="Iris Data (red=setosa,green=versicolor,blue=virginica)", pch=21, bg=c("red","green3","blue")[unclass(iris$Species)])

17. And use a contingency table
> data(Titanic) > mdl <- naiveBayes(Survived ~ ., data = Titanic) > mdl
Naive Bayes Classifier for Discrete Predictors
Call: naiveBayes.formula(formula = Survived ~ ., data = Titanic)
A-priori probabilities:
Survived
No Yes
Conditional probabilities:
Class
Survived st nd rd Crew No
Yes
Sex
Survived Male Female No
Yes
Age
Survived Child Adult No
Yes
group2/lab2_nbayes1.R

18. Using a contingency table
> predict(mdl, as.data.frame(Titanic)[,1:3]) [1] Yes No No No Yes Yes Yes Yes No No No No Yes Yes Yes Yes Yes No No No Yes Yes Yes Yes No [26] No No No Yes Yes Yes Yes Levels: No Yes

19. http://www. ugrad. stat. ubc. ca/R/library/mlbench/html/HouseVotes84
require(mlbench) data(HouseVotes84) model <- naiveBayes(Class ~ ., data = HouseVotes84) predict(model, HouseVotes84[1:10,-1]) predict(model, HouseVotes84[1:10,-1], type = "raw") pred <- predict(model, HouseVotes84[,-1]) table(pred, HouseVotes84$Class)

20. nbayes1
> table(pred, HouseVotes84$Class) pred democrat republican democrat republican

21. > predict(model, HouseVotes84[1:10,-1], type = "raw") democrat republican [1,] e e-01 [2,] e e-01 [3,] e e-01 [4,] e e-03 [5,] e e-02 [6,] e e-01 [7,] e e-01 [8,] e e-01 [9,] e e-01 [10,] e e-11

22. At this point…
You may realize the inter-relation among classifications and clustering methods, at an absolute and relative level (i.e. hierarchical -> trees…) is COMPLEX…
Trees are interesting from a decision perspective: if this or that, then this…. More in the next module
Beyond just distance measures: clustering (kmeans) to probabilities (Bayesian)
And, so many ways to visualize them…

23. General idea behind trees
Although the basic philosophy of all the classifiers based on decision trees is identical, there are many possibilities for its construction.
Among all the key points in the selection of an algorithm to build decision trees some of them should be highlighted for their importance:
Criteria for the choice of feature to be used in each node
How to calculate the partition of the set of examples
When you decide that a node is a leaf
What is the criterion to select the class to assign to each leaf

24. Some important advantages can be pointed to the decision trees, including:
Can be applied to any type of data
The final structure of the classifier is quite simple and can be stored and handled in a graceful manner
Handles very efficiently conditional information, subdividing the space into sub-spaces that are handled individually
Reveal normally robust and insensitive to misclassification in the training set
The resulting trees are usually quite understandable and can be easily used to obtain a better understanding of the phenomenon in question. This is perhaps the most important of all the advantages listed

25. Stopping – leaves on the tree
A number of stopping conditions can be used to stop the recursive process. The algorithm stops when any one of the conditions is true:
All the samples belong to the same class, i.e. have the same label since the sample is already "pure"
Stop if most of the points are already of the same class. This is a generalization of the first approach, with some error threshold
There are no remaining attributes on which the samples may be further partitioned
There are no samples for the branch test attribute

26. Recursive partitioning
Recursive partitioning is a fundamental tool in data mining. It helps us explore the structure of a set of data, while developing easy to visualize decision rules for predicting a categorical (classification tree) or continuous (regression tree) outcome.
The rpart programs build classification or regression models of a very general structure using a two stage procedure; the resulting models can be represented as binary trees.

27. Recursive partitioning
The tree is built by the following process:
first the single variable is found which best splits the data into two groups ('best' will be defined later). The data is separated, and then this process is applied separately to each sub-group, and so on recursively until the subgroups either reach a minimum size or until no improvement can be made.
second stage of the procedure consists of using cross-validation to trim back the full tree.

28. Why are we careful doing this?
Because we will USE these trees, i.e. apply them to make decisions about what things are and what to do with them!

29. Ionosphere…
> printcp(rpart.model) Classification tree: rpart(formula = Type ~ ., data = trainset) Variables actually used in tree construction: [1] Al Ba Mg RI Root node error: 92/143 = n= 143 CP nsplit rel error xerror xstd

30. plotcp(rpart.model)

31. > rsq.rpart(rpart.model) Classification tree: rpart(formula = Type ~ ., data = trainset) Variables actually used in tree construction: [1] Al Ba Mg RI Root node error: 92/143 = n= 143 CP nsplit rel error xerror xstd Warning message: In rsq.rpart(rpart.model) : may not be applicable for this method

32. rsq.rpart

33. > print(rpart. model) n= 143 node), split, n, loss, yval, (yprob)
> print(rpart.model) n= 143 node), split, n, loss, yval, (yprob) * denotes terminal node 1) root ( ) 2) Ba< ( ) 4) Al< ( ) 8) RI>= ( ) 16) RI< ( ) * 17) RI>= ( ) 34) RI>= ( ) 68) Mg>= ( ) * 69) Mg< ( ) * 35) RI< ( ) * 9) RI< ( ) * 5) Al>= ( ) 10) Mg>= ( ) * 11) Mg< ( ) * 3) Ba>= ( ) *

34. Tree plot > plot(rpart.model,compress=TRUE)
> text(rpart.model, use.n=TRUE)
plot(object, uniform=FALSE, branch=1, compress=FALSE, nspace, margin=0, minbranch=.3, args)

35. And if you are brave
summary(rpart.model) … pages….

36. Remember to LOOK at the data
> names(Glass) [1] "RI" "Na" "Mg" "Al" "Si" "K" "Ca" "Ba" "Fe" "Type" > head(Glass) RI Na Mg Al Si K Ca Ba Fe Type

37. rpart.pred
> rpart.pred Levels:

38. plot(rpart.pred)

39. New dataset to work with trees
fitK <- rpart(Kyphosis ~ Age + Number + Start, method="class", data=kyphosis) printcp(fitK) # display the results plotcp(fitK) # visualize cross-validation results summary(fitK) # detailed summary of splits # plot tree plot(fitK, uniform=TRUE, main="Classification Tree for Kyphosis") text(fitK, use.n=TRUE, all=TRUE, cex=.8) # create attractive postscript plot of tree post(fitK, file = “kyphosistree.ps", title = "Classification Tree for Kyphosis") # might need to convert to PDF (distill)

41. > pfitK<- prune(fitK, cp= fitK$cptable[which
> pfitK<- prune(fitK, cp= fitK$cptable[which.min(fitK$cptable[,"xerror"]),"CP"])
> plot(pfitK, uniform=TRUE, main="Pruned Classification Tree for Kyphosis")
> text(pfitK, use.n=TRUE, all=TRUE, cex=.8)
> post(pfitK, file = “ptree.ps", title = "Pruned Classification Tree for Kyphosis”)

42. > fitK <- ctree(Kyphosis ~ Age + Number + Start, data=kyphosis)
> plot(fitK, main="Conditional Inference Tree for Kyphosis”)

43. > plot(fitK, main="Conditional Inference Tree for Kyphosis",type="simple")

44. randomForest
> require(randomForest) > fitKF <- randomForest(Kyphosis ~ Age + Number + Start, data=kyphosis) > print(fitKF) # view results Call: randomForest(formula = Kyphosis ~ Age + Number + Start, data = kyphosis) Type of random forest: classification Number of trees: 500 No. of variables tried at each split: 1 OOB estimate of error rate: 20.99% Confusion matrix: absent present class.error absent present > importance(fitKF) # importance of each predictor MeanDecreaseGini Age Number Start
Random forests improve predictive accuracy by generating a large number of bootstrapped trees (based on random samples of variables), classifying a case using each tree in this new "forest", and deciding a final predicted outcome by combining the results across all of the trees (an average in regression, a majority vote in classification).

45. Trees for the Titanic
data(Titanic) rpart, ctree, hclust, etc. for Survived ~ .

46. More on another dataset.
# Regression Tree Example
library(rpart)
# build the tree
fitM <- rpart(Mileage~Price + Country + Reliability + Type, method="anova", data=cu.summary)
printcp(fitM) # display the results ….
Root node error: /60 =
n=60 (57 observations deleted due to missingness)
CP nsplit rel error xerror xstd

47. Mileage…
plotcp(fitM) # visualize cross-validation results summary(fitM) # detailed summary of splits <we will look more at this in a future lab>

48. par(mfrow=c(1,2))
rsq.rpart(fitM) # visualize cross-validation results

49. # plot tree plot(fitM, uniform=TRUE, main="Regression Tree for Mileage ") text(fitM, use.n=TRUE, all=TRUE, cex=.8) # prune the tree pfitM<- prune(fitM, cp= ) # from cptable # plot the pruned tree plot(pfitM, uniform=TRUE, main="Pruned Regression Tree for Mileage") text(pfitM, use.n=TRUE, all=TRUE, cex=.8) post(pfitM, file = ”ptree2.ps", title = "Pruned Regression Tree for Mileage”)

51. # Conditional Inference Tree for Mileage fit2M <- ctree(Mileage~Price + Country + Reliability + Type, data=na.omit(cu.summary))

52. Enough of trees!

53. Cross-validation
Cross-validation is a model validation technique for assessing how the results of a statistical analysis will generalize to an independent data set.
It is mainly used in settings where the goal is prediction, and one wants to estimate how accurately a predictive model will perform in practice.
I.e. predictive and prescriptive analytics…

54. Cross-validation
In a prediction problem, a model is usually given a dataset of known data on which training is run (training dataset), and a dataset of unknown data (or first seen data) against which the model is tested (testing dataset). Sound familiar?

55. Cross-validation
The goal of cross validation is to define a dataset to "test" the model in the training phase (i.e., the validation dataset), in order to limit problems like overfitting And, give an insight on how the model will generalize to an independent data set (i.e., an unknown dataset, for instance from a real problem), etc.

56. Common type of x-validation
K-fold
2-fold (do you know this one?)
Rep-random-subsample
Leave out-subsample
Lab in a few weeks … to try these out

57. K-fold
Original sample is randomly partitioned into k equal size subsamples.
Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data.
Repeat cross-validation process k times (folds), with each of the k subsamples used exactly once as the validation data.
The k results from the folds can then be averaged (usually) to produce a single estimation.

58. Leave out subsample
As the name suggests, leave-one-out cross-validation (LOOCV) involves using a single observation from the original sample as the validation data, and the remaining observations as the training data.
i.e. K=n-fold cross-validation
Leave out > 1 = bootstraping and jackknifing

59. boot(strapping)
Generate replicates of a statistic applied to data (parametric and nonparametric).
nonparametric bootstrap, possible methods: ordinary bootstrap, the balanced bootstrap, antithetic resampling, and permutation.
For nonparametric multi-sample problems stratified resampling is used:
this is specified by including a vector of strata in the call to boot.
importance resampling weights may be specified.

60. Jackknifing
Systematically recompute the statistic estimate, leaving out one or more observations at a time from the sample set
From this new set of replicates of the statistic, an estimate for the bias and an estimate for the variance of the statistic can be calculated.
Often use log(variance) [instead of variance] especially for non-normal distributions

61. Repeat-random-subsample
Random split of the dataset into training and validation data.
For each such split, the model is fit to the training data, and predictive accuracy is assessed using the validation data.
Results are then averaged over the splits.
Note: for this method can the results will vary if the analysis is repeated with different random splits.

62. Advantage?
The advantage of K-fold over repeated random sub-sampling is that all observations are used for both training and validation, and each observation is used for validation exactly once.
10-fold cross-validation is commonly used
The advantage of rep-random over k-fold cross validation is that the proportion of the training/validation split is not dependent on the number of iterations (folds).

63. Disadvantage
The disadvantage of rep-random is that some observations may never be selected in the validation subsample, whereas others may be selected more than once.
i.e., validation subsets may overlap.

64. Assignment 6 Introduction (2%) % may change…
Your term projects should fall within the scope of a data analytics problem of the type you have worked with in class/ labs, or know of yourself – the bigger the data the better. This means that the work must go beyond just making lots of figures. You should develop the project to indicate you are thinking of and exploring the relationships and distributions within your data. Start with a hypothesis, think of a way to model and use the hypothesis, find or collect the necessary data, and do both preliminary analysis, detailed modeling and summary (interpretation). Grad students must develop two types of models.
Note: You do not have to come up with a positive result, i.e. disproving the hypothesis is just as good.
Introduction (2%) % may change…
Data Description (3%)
Analysis (5%)
Model Development (12%)
Conclusions and Discussion (3%)
Oral presentation (5%) (~5 mins)