1. Interpreting: regression, weighted kNN, clustering, trees and Bayesian methods
Peter Fox and Greg Hughes
Data Analytics – ITWS-4600/ITWS-6600
Group 2 Module 6, February 13, 2017

2. Contents

3. K Nearest Neighbors (classification)
Script – group2/lab1_nyt.R
> nyt1<-read.csv(“nyt1.csv")
… from week 3b slides or script
> classif<-knn(train,test,cg,k=5) #
> head(true.labels)
[1]
> head(classif)
[1]
Levels: 0 1
> ncorrect<-true.labels==classif
> table(ncorrect)["TRUE"] # or > length(which(ncorrect))
> What do you conclude?

4. Bronx 1 = Regression You were reminded that log(0) is … not fun
> plot(log(bronx$GROSS.SQUARE.FEET), log(bronx$SALE.PRICE) )
> m1<-lm(log(bronx$SALE.PRICE)~log(bronx$GROSS.SQUARE.FEET),data=bronx)
You were reminded that log(0) is … not fun
 THINK through what you are doing…
Filtering is somewhat inevitable:
> bronx<-bronx[which(bronx$GROSS.SQUARE.FEET>0 & bronx$LAND.SQUARE.FEET>0 & bronx$SALE.PRICE>0),]
Lab5b_bronx1_2016.R

5. Interpreting this!
Call: lm(formula = log(SALE.PRICE) ~ log(GROSS.SQUARE.FEET), data = bronx) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) <2e-16 *** log(GROSS.SQUARE.FEET) <2e-16 *** --- Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 1.95 on 2435 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 1 and 2435 DF, p-value: < 2.2e-16

6. Plots – tell me what they tell you!

7. Solution model 2
> m2<-lm(log(bronx$SALE.PRICE)~log(bronx$GROSS.SQUARE.FEET)+log(bronx$LAND.SQUARE.FEET)+factor(bronx$NEIGHBORHOOD),data=bronx) > summary(m2) > plot(resid(m2)) # > m2a<-lm(log(bronx$SALE.PRICE)~0+log(bronx$GROSS.SQUARE.FEET)+log(bronx$LAND.SQUARE.FEET)+factor(bronx$NEIGHBORHOOD),data=bronx) > summary(m2a) > plot(resid(m2a))

8. How do you interpret this residual plot?

9. Solution model 3 and 4
> m3<-lm(log(bronx$SALE.PRICE)~0+log(bronx$GROSS.SQUARE.FEET)+log(bronx$LAND.SQUARE.FEET)+factor(bronx$NEIGHBORHOOD)+factor(bronx$BUILDING.CLASS.CATEGORY),data=bronx) > summary(m3) > plot(resid(m3)) # > m4<-lm(log(bronx$SALE.PRICE)~0+log(bronx$GROSS.SQUARE.FEET)+log(bronx$LAND.SQUARE.FEET)+factor(bronx$NEIGHBORHOOD)*factor(bronx$BUILDING.CLASS.CATEGORY),data=bronx) > summary(m4) > plot(resid(m4))

10. And this one?

11. Bronx 2 = complex example
See lab1_bronx2.R Manipulation Mapping knn kmeans

12. Did you get to create the neighborhood map?
table(mapcoord$NEIGHBORHOOD) mapcoord$NEIGHBORHOOD <- as.factor(mapcoord$NEIGHBORHOOD) The MAP!!

14. mapmeans<-cbind(adduse,as
mapmeans<-cbind(adduse,as.numeric(mapcoord$NEIGHBORHOOD)) colnames(mapmeans)[26] <- "NEIGHBORHOOD" #This is the right way of renaming. keeps <- c("ZIP.CODE","NEIGHBORHOOD","TOTAL.UNITS","LAND.SQUARE.FEET","GROSS.SQUARE.FEET","SALE.PRICE","Latitude","Longitude") mapmeans<-mapmeans[keeps]#Dropping others mapmeans$NEIGHBORHOOD<-as.numeric(mapcoord$NEIGHBORHOOD) for(i in 1:8){ mapmeans[,i]=as.numeric(mapmeans[,i]) }#Now done for conversion to numeric

15. #Classification mapcoord$class<as
#Classification mapcoord$class<as.numeric(mapcoord$NEIGHBORHOOD) nclass<-dim(mapcoord)[1] split<-0.8 trainid<-sample.int(nclass,floor(split*nclass)) testid<-(1:nclass)[-trainid]

16. KNN!
Did you loop over k? { knnpred<-knn(mapcoord[trainid,3:4],mapcoord[testid,3:4],cl=mapcoord[trainid,2],k=5) knntesterr<-sum(knnpred!=mappred$class)/length(testid) } knntesterr [1] What do you think?

17. Try these on mapmeans, etc.

18. K-Means!
> mapmeans<-data.frame(adduse$ZIP.CODE, as.numeric(mapcoord$NEIGHBORHOOD), adduse$TOTAL.UNITS, adduse$"LAND.SQUARE.FEET", adduse$GROSS.SQUARE.FEET, adduse$SALE.PRICE, adduse$'querylist$latitude', adduse$'querylist$longitude') > mapobj<-kmeans(mapmeans,5, iter.max=10, nstart=5, algorithm = c("Hartigan-Wong", "Lloyd", "Forgy", "MacQueen")) > fitted(mapobj,method=c("centers","classes"))

19. > mapobj$centers
adduse.ZIP.CODE as.numeric.mapcoord.NEIGHBORHOOD. adduse.TOTAL.UNITS adduse.LAND.SQUARE.FEET
adduse.GROSS.SQUARE.FEET adduse.SALE.PRICE adduse..querylist.latitude. adduse..querylist.longitude.

20. > plot(mapmeans,mapobj$cluster)
> mapobj$size
[1]
ZIP.CODE, NEIGHBORHOOD, TOTAL.UNITS, LAND.SF, GROSS.SF, SALE.PRICE, lat, long
ZIP.CODE, NEIGHBORHOOD, TOTAL.UNITS, LAND.SQUARE.FEET, GROSS.SQUARE.FEET, SALE.PRICE, latitude, longitude'

21. Return object
cluster A vector of integers (from 1:k) indicating the cluster to which each point is allocated. centers A matrix of cluster centres. totss The total sum of squares. withinss Vector of within-cluster sum of squares, one component per cluster. tot.withinss Total within-cluster sum of squares, i.e., sum(withinss). betweenss The between-cluster sum of squares, i.e. totss-tot.withinss. size The number of points in each cluster.

22. Plotting clusters
library(cluster) clusplot(mapmeans, mapobj$cluster, color=TRUE, shade=TRUE, labels=2, lines=0) # Centroid Plot against 1st 2 discriminant functions library(fpc) plotcluster(mapmeans, mapobj$cluster)

23. Plotting clusters
require(cluster) clusplot(mapmeans, mapobj$cluster, color=TRUE, shade=TRUE, labels=2, lines=0)

24. Plot

25. Clusplot (k=17)

26. Dendogram for this = tree of the clusters:
Highly supported by data?
Okay, this is a little complex – perhaps something simpler?

27. What else could you cluster/classify?
SALE.PRICE?
If so, how would you measure error?
# I added SALE.PRICE as 5th column in adduse…
> pcolor<- color.scale(log(mapcoord[,5]),c(0,1,1),c(1,1,0),0)
> geoPlot(mapcoord,zoom=12,color=pcolor)
TAX.CLASS.AT.PRESENT?
TAX.CLASS.AT.TIME.OF.SALE?
measure error?

28. Regression Exercises
Using the EPI dataset find the single most important factor in increasing the EPI in a given region
Examine distributions down to the leaf nodes and build up an EPI “model”

29. Linear and least-squares
> EPI_data<- read.csv(”EPI_data.csv") > attach(EPI_data) > boxplot(ENVHEALTH,DALY,AIR_H,WATER_H) > lmENVH<-lm(ENVHEALTH~DALY+AIR_H+WATER_H) > lmENVH … (what should you get?) > summary(lmENVH) … > cENVH<-coef(lmENVH)

30. Linear and least-squares
> lmENVH<-lm(ENVHEALTH~DALY+AIR_H+WATER_H) > lmENVH Call: lm(formula = ENVHEALTH ~ DALY + AIR_H + WATER_H) Coefficients: (Intercept) DALY AIR_H WATER_H e e e e-01 > summary(lmENVH) … > cENVH<-coef(lmENVH)

31. Read the documentation!

32. Linear and least-squares
> summary(lmENVH) Call: lm(formula = ENVHEALTH ~ DALY + AIR_H + WATER_H) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) e e DALY 5.000e e <2e-16 *** AIR_H 2.500e e <2e-16 *** WATER_H 2.500e e <2e-16 *** ---
p < 0.01 : very strong presumption against null hypothesis vs. this fit
0.01 < p < : strong presumption against null hypothesis
0.05 < p < 0.1 : low presumption against null hypothesis
p > 0.1 : no presumption against the null hypothesis

33. Linear and least-squares
Continued: --- Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: on 178 degrees of freedom (49 observations deleted due to missingness) Multiple R-squared: 1, Adjusted R-squared: 1 F-statistic: 3.983e+09 on 3 and 178 DF, p-value: < 2.2e-16 > names(lmENVH) [1] "coefficients" "residuals" "effects" "rank" "fitted.values" "assign" [7] "qr" "df.residual" "na.action" "xlevels" "call" "terms" [13] "model"

34. Plot original versus fitted
> plot(ENVHEALTH,col="red")
> points(lmENVH$fitted.values,col="blue")
> Huh?

35. Try again!
> plot(ENVHEALTH[!is.na(ENVHEALTH)], col="red") > points(lmENVH$fitted.values,col="blue")

36. Predict
> cENVH<-coef(lmENVH) > DALYNEW<-c(seq(5,95,5)) #2 > AIR_HNEW<-c(seq(5,95,5)) #3 > WATER_HNEW<-c(seq(5,95,5)) #4

37. Predict
> NEW<-data.frame(DALYNEW,AIR_HNEW,WATER_HNEW) > pENV<- predict(lmENVH,NEW,interval=“prediction”) > cENV<- predict(lmENVH,NEW,interval=“confidence”) # look up what this does

38. Predict object returns
predict.lm produces a vector of predictions or a matrix of predictions and bounds with column names fit, lwr, and upr if interval is set. Access via [,1] etc. If se.fit is TRUE, a list with the following components is returned: fit vector or matrix as above se.fit standard error of predicted means residual.scale residual standard deviations df degrees of freedom for residual

39. Output from predict
> head(pENV) fit lwr upr 1 NA NA NA NA NA NA …

40. > tail(pENV) fit lwr upr 226 NA NA NA 227 NA NA NA 228 34. 95256 34

41. Read the documentation!

42. Ionosphere: group2/lab1_kknn2.R
require(kknn) data(ionosphere) ionosphere.learn <- ionosphere[1:200,] ionosphere.valid <- ionosphere[-c(1:200),] fit.kknn <- kknn(class ~ ., ionosphere.learn, ionosphere.valid) table(ionosphere.valid$class, fit.kknn$fit) # vary kernel (fit.train1 <- train.kknn(class ~ ., ionosphere.learn, kmax = 15, kernel = c("triangular", "rectangular", "epanechnikov", "optimal"), distance = 1)) table(predict(fit.train1, ionosphere.valid), ionosphere.valid$class) #alter distance (fit.train2 <- train.kknn(class ~ ., ionosphere.learn, kmax = 15, kernel = c("triangular", "rectangular", "epanechnikov", "optimal"), distance = 2)) table(predict(fit.train2, ionosphere.valid), ionosphere.valid$class)

43. Results
ionosphere.learn <- ionosphere[1:200,] # convenience samping!!!! ionosphere.valid <- ionosphere[-c(1:200),] fit.kknn <- kknn(class ~ ., ionosphere.learn, ionosphere.valid) table(ionosphere.valid$class, fit.kknn$fit) b g b 19 8 g 2 122

44. (fit. train1 <- train. kknn(class ~. , ionosphere
(fit.train1 <- train.kknn(class ~ ., ionosphere.learn, kmax = 15, + kernel = c("triangular", "rectangular", "epanechnikov", "optimal"), distance = 1)) Call: train.kknn(formula = class ~ ., data = ionosphere.learn, kmax = 15, distance = 1, kernel = c("triangular", "rectangular", "epanechnikov", "optimal")) Type of response variable: nominal Minimal misclassification: 0.12 Best kernel: rectangular Best k: 2 table(predict(fit.train1, ionosphere.valid), ionosphere.valid$class) b g b 25 4 g 2 120

45. (fit. train2 <- train. kknn(class ~. , ionosphere
(fit.train2 <- train.kknn(class ~ ., ionosphere.learn, kmax = 15, + kernel = c("triangular", "rectangular", "epanechnikov", "optimal"), distance = 2)) Call: train.kknn(formula = class ~ ., data = ionosphere.learn, kmax = 15, distance = 2, kernel = c("triangular", "rectangular", "epanechnikov", "optimal")) Type of response variable: nominal Minimal misclassification: 0.12 Best kernel: rectangular Best k: 2 table(predict(fit.train2, ionosphere.valid), ionosphere.valid$class) b g b 20 5 g 7 119

46. However… there is more

47. Naïve Bayes – what is it?
Example: testing for a specific item of knowledge that 1% of the population has been informed of (don’t ask how).
An imperfect test:
99% of knowledgeable people test positive
99% of ignorant people test negative
If a person tests positive – what is the probability that they know the fact?

48. Naïve approach… We have 10,000 representative people
100 know the fact/item, 9,900 do not
We test them all:
Get 99 knowing people testing knowing
Get 99 not knowing people testing not knowing
But 99 not knowing people testing as knowing
Testing positive (knowing) – equally likely to know or not = 50%

49. Tree diagram 10000 ppl 1% know (100ppl) 99% test to know (99ppl)
1% test not to know (1per)
99% do not know (9900ppl)
1% test to know (99ppl)
99% test not to know (9801ppl)

50. Relation between probabilities
For outcomes x and y there are probabilities of p(x) and p (y) that either happened
If there’s a connection, then the joint probability = that both happen = p(x,y)
Or x happens given y happens = p(x|y) or vice versa then:
p(x|y)*p(y)=p(x,y)=p(y|x)*p(x)
So p(y|x)=p(x|y)*p(y)/p(x) (Bayes’ Law)
E.g. p(know|+ve)=p(+ve|know)*p(know)/p(+ve)= (.99*.01)/(.99* *.99) = 0.5

51. How do you use it?
If the population contains x what is the chance that y is true?
p(SPAM|word)=p(word|SPAM)*p(SPAM)/p(word)
Base this on data:
p(spam) counts proportion of spam versus not
p(word|spam) counts prevalence of spam containing the ‘word’
p(word|!spam) counts prevalence of non-spam containing the ‘word’

52. Or..
What is the probability that you are in one class (i) over another class (j) given another factor (X)?
Invoke Bayes:
Maximize p(X|Ci)p(Ci)/p(X) (p(X)~constant and p(Ci) are equal if not known)
So: conditional indep -

53. P(xk | Ci) is estimated from the training samples
Categorical: Estimate P(xk | Ci) as percentage of samples of class i with value xk
Training involves counting percentage of occurrence of each possible value for each class
Numeric: Actual form of density function is generally not known, so “normal” density (i.e. distribution) is often assumed

54. Digging into iris
classifier<-naiveBayes(iris[,1:4], iris[,5]) table(predict(classifier, iris[,-5]), iris[,5], dnn=list('predicted','actual')) classifier$apriori classifier$tables$Petal.Length plot(function(x) dnorm(x, 1.462, ), 0, 8, col="red", main="Petal length distribution for the 3 different species") curve(dnorm(x, 4.260, ), add=TRUE, col="blue") curve(dnorm(x, 5.552, ), add=TRUE, col = "green")

56. Bayes
> cl <- kmeans(iris[,1:4], 3) > table(cl$cluster, iris[,5]) setosa versicolor virginica # > m <- naiveBayes(iris[,1:4], iris[,5]) > table(predict(m, iris[,1:4]), iris[,5]) setosa versicolor virginica
pairs(iris[1:4],main="Iris Data (red=setosa,green=versicolor,blue=virginica)", pch=21, bg=c("red","green3","blue")[unclass(iris$Species)])

57. And use a contingency table
> data(Titanic) > mdl <- naiveBayes(Survived ~ ., data = Titanic) > mdl
Naive Bayes Classifier for Discrete Predictors
Call: naiveBayes.formula(formula = Survived ~ ., data = Titanic)
A-priori probabilities:
Survived
No Yes
Conditional probabilities:
Class
Survived st nd rd Crew No
Yes
Sex
Survived Male Female No
Yes
Age
Survived Child Adult No
Yes
Try Lab5b_nbayes1_2016.R

58. Using a contingency table
> predict(mdl, as.data.frame(Titanic)[,1:3]) [1] Yes No No No Yes Yes Yes Yes No No No No Yes Yes Yes Yes Yes No No No Yes Yes Yes Yes No [26] No No No Yes Yes Yes Yes Levels: No Yes

59. http://www. ugrad. stat. ubc. ca/R/library/mlbench/html/HouseVotes84
require(mlbench) data(HouseVotes84) model <- naiveBayes(Class ~ ., data = HouseVotes84) predict(model, HouseVotes84[1:10,-1]) predict(model, HouseVotes84[1:10,-1], type = "raw") pred <- predict(model, HouseVotes84[,-1]) table(pred, HouseVotes84$Class)

60. Exercise for you
> data(HairEyeColor) > mosaicplot(HairEyeColor) > margin.table(HairEyeColor,3) Sex Male Female > margin.table(HairEyeColor,c(1,3)) Hair Male Female Black Brown Red Blond How would you construct a naïve Bayes classifier and test it?

61. And use a contingency table
> data(Titanic) > mdl <- naiveBayes(Survived ~ ., data = Titanic) > mdl
Naive Bayes Classifier for Discrete Predictors
Call: naiveBayes.formula(formula = Survived ~ ., data = Titanic)
A-priori probabilities:
Survived
No Yes
Conditional probabilities:
Class
Survived st nd rd Crew No
Yes
Sex
Survived Male Female No
Yes
Age
Survived Child Adult No
Yes
Try group2/lab2_nbayes1.R

62. http://www. ugrad. stat. ubc. ca/R/library/mlbench/html/HouseVotes84
require(mlbench) data(HouseVotes84) model <- naiveBayes(Class ~ ., data = HouseVotes84) predict(model, HouseVotes84[1:10,-1]) predict(model, HouseVotes84[1:10,-1], type = "raw") pred <- predict(model, HouseVotes84[,-1]) table(pred, HouseVotes84$Class)

63. nbayes1
> table(pred, HouseVotes84$Class) pred democrat republican democrat republican

64. > predict(model, HouseVotes84[1:10,-1], type = "raw") democrat republican [1,] e e-01 [2,] e e-01 [3,] e e-01 [4,] e e-03 [5,] e e-02 [6,] e e-01 [7,] e e-01 [8,] e e-01 [9,] e e-01 [10,] e e-11

65. Ex: Classification Bayes
Retrieve the abalone.csv dataset
Predicting the age of abalone from physical measurements.
Perform naivebayes classification to get predictors for Age (Rings). Interpret.
Discuss in next lab.

66. Exercise
> data(HairEyeColor) > mosaicplot(HairEyeColor) > margin.table(HairEyeColor,3) Sex Male Female > margin.table(HairEyeColor,c(1,3)) Hair Male Female Black Brown Red Blond How would you construct a naïve Bayes classifier and test it?

67. At this point…
You may realize the inter-relation among classifications and clustering methods, at an absolute and relative level (i.e. hierarchical -> trees…) is COMPLEX…
Trees are interesting from a decision perspective: if this or that, then this…. More in the next module
Beyond just distance measures: clustering (kmeans) to probabilities (Bayesian)
And, so many ways to visualize them…