1 Peter Fox Data Analytics – ITWS-4963/ITWS-6965 Week 2b, February 6, 2015 Lab exercises: beginning to work with data: filtering, distributions, populations, testing and models.

The DATUM project This class is one included in the DATUM project for assessment and evaluation Here’s what that means… 2

Assignment 1 – discuss. Review/ critique (business case, area of application, approach/ methods, tools used, results, actions, benefits). What did you choose/ why and what are your review comments? 3

Today - lab Exploring data and their distributions Fitting distributions, joint and conditional Constructing models and testing their fitness 4

Objectives for today Get familiar with: –R –Distributions –Populations –Fitting –Filtering –Testing 5

Table: Matlab/R/scipy-numpy 6

Files EPI_data.xls – with missing values changed to suit your application (EPI2010_all countries or EPI2010_onlyEPIcountries tabs) Get the data read in as you did last week (in e.g. I will use “EPI_data” for the object (in R)) > EPI_data /2010EPI_data.csv") > View(EPI_data) 7

Tips (in R) > attach(EPI_data) # sets the ‘default’ object > fix(EPI_data) # launches a simple data editor > EPI # prints out values EPI_data$EPI [1] NA NA 36.3 NA 71.4 NA NA NA [18] NA NA NA [35] NA 76.8 NA NA NA [52] NA NA NA 78.2 [69] NA NA NA 44.4 NA NA NA 54.0 NA [86] NA 59.2 NA NA [103] NA NA NA 50.1 NA NA 63.7 [120] NA NA 65.6 NA NA [137] NA NA NA NA 59.3 NA 37.6 NA NA 66.4 [154] NA NA NA NA [171] NA NA 48.9 NA NA NA [188] NA NA NA NA 64.6 NA NA [205] 38.4 NA NA NA NA NA NA 62.9 [222] NA NA 59.0 NA NA NA > tf <- is.na(EPI) # records True values if the value is NA > E <- EPI[!tf] # filters out NA values, new array 8

Exercise 1: exploring the distribution > summary(EPI) # stats Min. 1st Qu. Median Mean 3rd Qu. Max. NA's > fivenum(EPI,na.rm=TRUE) [1] > stem(EPI) # stem and leaf plot > hist(EPI) > hist(EPI, seq(30., 95., 1.0), prob=TRUE) > lines(density(EPI,na.rm=TRUE,bw=1.)) # or try bw=“SJ” > rug(EPI) > Use help( ), e.g. > help(stem) 9

10 Save your plots, name them. Save the commands you used to generate them.

Exercise 1: fitting a distribution beyond histograms Cumulative density function? > plot(ecdf(EPI), do.points=FALSE, verticals=TRUE) Quantile-Quantile? > par(pty="s") > qqnorm(EPI); qqline(EPI) Simulated data from t-distribution: > x <- rt(250, df = 5) > qqnorm(x); qqline(x) Make a Q-Q plot against the generating distribution by: x<-seq(30,95,1) > qqplot(qt(ppoints(250), df = 5), x, xlab = "Q-Q plot for t dsn") > qqline(x) 11

Exercise 1: fitting a distribution Your exercise: do the same exploration and fitting for another 2 variables in the EPI_data, i.e. primary variables (DALY, WATER_H, …) Try fitting other distributions – i.e. as ecdf or qq- 12

Comparing distributions > boxplot(EPI,DALY) > t.test(EPI,DALY) Welch Two Sample t-test data: EPI and DALY t = , df = , p-value = alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: sample estimates: mean of x mean of y

qqplot(EPI,DALY) 14

But there is more Your exercise – intercompare: EPI, ENVHEALTH, ECOSYSTEM, DALY, AIR_H, WATER_H, AIR_EWATER_E, BIODIVERSITY ** (subject to possible filtering…) 15

Exercise 2: filtering (populations) Conditional filtering: > EPILand<-EPI[!Landlock] > Eland <- EPILand[!is.na(EPILand)] > hist(ELand) > hist(ELand, seq(30., 95., 1.0), prob=TRUE) Repeat exercise 1… Also look at: No_surface_water, Desert and High_Population_Density Your exercise: how to filter on EPI_regions or GEO_subregion? E.g. EPI_South_Asia ] 16

Exercise 3: testing the fits shapiro.test(EPI) ks.test(EPI,seq(30.,95.,1.0)) You should be thinking about how to interpret (we covered some of this early this week and will also in the next few classes)? 17

Exercise 4: joint distributions > t.test(EPI,DALY) Welch Two Sample t-test data: EPI and DALY t = , df = , p-value = alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: sample estimates: mean of x mean of y > var.test(EPI,DALY) F test to compare two variances data: EPI and DALY F = , num df = 162, denom df = 191, p-value < 2.2e-16 alternative hypothesis: true ratio of variances is not equal to 1 95 percent confidence interval: sample estimates: ratio of variances

Kolmogorov- Smirnov - KS test - > ks.test(EPI,DALY) Two-sample Kolmogorov-Smirnov test data: EPI and DALY D = , p-value = alternative hypothesis: two-sided Warning message: In ks.test(EPI, DALY) : p-value will be approximate in the presence of ties 19

Objectives for today Get familiar with: –Distributions –Populations –Fitting –Filtering –Testing 20