1. Group 1 Lab 2 exercises /assignment 2
Peter Fox and Greg Hughes
Data Analytics – ITWS-4600/ITWS-6600
Group 1, Lab 2, February 2, 2017

2. Labs 2a. Regression 2b. kNN 2c. Kmeans I.e. 1 each for Assignment 2
New multivariate dataset
2b. kNN
New Abalone dataset
2c. Kmeans
(Sort of) New Iris dataset
I.e. 1 each for Assignment 2
And then general exercises

3. The Dataset(s) http://aquarius.tw.rpi.edu/html/DA
Some new ones; dataset_multipleRegression.csv, abalone.csv
Code fragments, i.e. they will not run as-is, on the following slides as group1/lab2_knn1.R, etc.

4. Remember a few useful cmds
head(<object>) tail(<object>) summary(<object>)

5. How does this work?
Following slides have 3 lab assignments for you to complete.
These should be completed individually
Once you complete (one or all), please raise your hand or approach one of us to review what you obtained (together these = 10% of your grade)
There is nothing to hand in
If you do not complete part/all in the lab session == that is okay but you will need to schedule a time to show your results

6. Refer to lecture slides and
Script fragments on website..

7. Regression (1) Retrieve this dataset: dataset_multipleRegression.csv
Using the unemployment rate (UNEM) and number of spring high school graduates (HGRAD), predict the fall enrollment (ROLL) for this year by knowing that UNEM=9% and HGRAD=100,000.
Repeat and add per capita income (INC) to the model. Predict ROLL if INC=$30,000
Summarize and compare the two models.
Comment on significance

8. Classification (2) Retrieve the abalone.csv dataset
Predicting the age of abalone from physical measurements.
The age of abalone is determined by cutting the shell through the cone, staining it, and counting the number of rings through a microscope: a boring and time-consuming task.
Other measurements, which are easier to obtain, are used to predict the age.
Perform knn classification to get predictors for Age (Rings). Interpretation not required.

9. Clustering (3) The Iris dataset (in R use data(“iris”) to load it)
The 5th column is the species and you want to find how many clusters without using that information
Create a new data frame and remove the fifth column
Apply kmeans (you choose k) with 1000 iterations
Use table(iris[,5],<your clustering>) to assess results

10. End of Lab assignment 2

11. Regression Exercises
Using the EPI dataset find the single most important factor in increasing the EPI in a given region
Examine distributions down to the leaf nodes and build up an EPI “model”

12. boxplot(ENVHEALTH,ECOSYSTEM)

13. qqplot(ENVHEALTH,ECOSYSTEM)

14. ENVHEALTH/ ECOSYSTEM
> shapiro.test(ENVHEALTH) Shapiro-Wilk normality test data: ENVHEALTH W = , p-value = 1.083e  Reject. > shapiro.test(ECOSYSTEM) data: ECOSYSTEM W = , p-value =  ~reject

15. Kolmogorov- Smirnov - KS test -
> ks.test(ENVHEALTH,ECOSYSTEM) Two-sample Kolmogorov-Smirnov test data: ENVHEALTH and ECOSYSTEM D = , p-value = 5.413e-07 alternative hypothesis: two-sided Warning message: In ks.test(ENVHEALTH, ECOSYSTEM) : p-value will be approximate in the presence of ties

16. Linear and least-squares
> EPI_data <- read.csv(”EPI_data.csv") > attach(EPI_data); > boxplot(ENVHEALTH,DALY,AIR_H,WATER_H) > lmENVH<-lm(ENVHEALTH~DALY+AIR_H+WATER_H) > lmENVH … (what should you get?) > summary(lmENVH) … > cENVH<-coef(lmENVH)

17. Predict
> DALYNEW<-c(seq(5,95,5)) > AIR_HNEW<-c(seq(5,95,5)) > WATER_HNEW<-c(seq(5,95,5)) > NEW<-data.frame(DALYNEW,AIR_HNEW,WATER_HNEW) > pENV<- predict(lmENVH,NEW,interval=“prediction”) > cENV<- predict(lmENVH,NEW,interval=“confidence”)

18. Repeat for
AIR_E
CLIMATE

19. Classification Exercises (group1/lab2_knn1.R)
> nyt1<-read.csv(“nyt1.csv") > nyt1<-nyt1[which(nyt1$Impressions>0 & nyt1$Clicks>0 & nyt1$Age>0),] > nnyt1<-dim(nyt1)[1] # shrink it down! > sampling.rate=0.9 > num.test.set.labels=nnyt1*(1.-sampling.rate) > training <-sample(1:nnyt1,sampling.rate*nnyt1, replace=FALSE) > train<-subset(nyt1[training,],select=c(Age,Impressions)) > testing<-setdiff(1:nnyt1,training) > test<-subset(nyt1[testing,],select=c(Age,Impressions)) > cg<-nyt1$Gender[training] > true.labels<-nyt1$Gender[testing] > classif<-knn(train,test,cg,k=5) # > classif > attributes(.Last.value) # interpretation to come!

20. Classification Exercises (group1/lab2_knn2.R)
2 examples in the script

21. Clustering Exercises group1/lab2_kmeans1.R
group1/lab2_kmeans2.R – plotting up results from the iris clustering