1. Evan Girvetz girvetz@u.washington.edu 206-543-5772 209 Winkenwerder Intro to R Programming: Lecture 2 © R Foundation, from http://www.r-project.org

2. Overview Reading data into R (e.g. from Excel) Working with and manipulating data frames Writing data to text files to use in Excel

3. Finding the Right Command There are many ways to find the correct command to read data files 1.Use help.search > help.search(“read data table”) 2. Look in reference material (e.g. Ref Card) 3. Use Google Google “Read data table R”

4. Finding the Right Command read.table() read.csv() scan() Look at these using R help ?read.table help(read.table)

5. Commands for Reading Data Tables Commands are the same except for defaults: read.table : general table reading command (tab delimited default) read.csv : comma separated values scan : good for reading odd formatting I use read.csv the most

6. Reading Data into R It is best to prepare your data in Excel (or other spreadsheet program. –That is what they are made for

7. Reading Data into R Only have at most one row of column headings –OK to have no headings, but not two rows Same for row names Columns and rows should be the same length –Or have NA to signify no data (NA) Column and row names should not have spaces in them –Replace spaces with a period (.) or underscore (_)

8. Hands-On Exercise Open the file “chinook_adult_return_data.xls”

9. Reading Data into R Try to “clean-up” data as best as possible in Spreadsheet program What are some of the problems with the excel file?

10. Reading Data into R Only have at most one row of column headings –OK to have no headings, but not two rows Same for row names Columns and rows should be the same length –Or have NA to signify no data (NA) Column and row names should not have spaces in them –Replace spaces with a period (.) or underscore (_)

11. Reading Data into R Save data from Excel in text format –e.g..csv or.txt (I like.csv) Make sure you know what are the delimiters that separate the data –This a comma for.csv, likely a tab or white space for.txt

12. Hands-On Exercise Clean up the Adult Chinook Return Data in Excel Export to two text files (.csv and.txt) Open the exported files in a text editor –e.g. Notepad, Wordpad, Tinn-R

13. read.table Look at the help for read.table and read.csv > ?read.table

14. read.table() read.table(file, header = FALSE, sep = "", quote = "\"'", dec = ".", row.names, col.names, as.is = !stringsAsFactors, na.strings = "NA", colClasses = NA, nrows = -1, skip = 0, check.names = TRUE, fill = !blank.lines.skip, strip.white = FALSE, blank.lines.skip = TRUE, comment.char = "#", allowEscapes = FALSE, flush = FALSE, stringsAsFactors = default.stringsAsFactors(), encoding = "unknown")

15. read.table() : file the name of the file which the data are to be read from. Each row of the table appears as one line of the file. If it does not contain an absolute path, the file name is relative to the current working directory, getwd(). Tilde- expansion is performed where supported. Alternatively, file can be a readable text-mode connection (which will be opened for reading if necessary, and if so closed (and hence destroyed) at the end of the function call). (If stdin() is used, the prompts for lines may be somewhat confusing. Terminate input with a blank line or an EOF signal, Ctrl-D on Unix and Ctrl-Z on Windows. Any pushback on stdin() will be cleared before return.) file can also be a complete URL.

16. read.table() : header a logical value indicating whether the file contains the names of the variables as its first line. If missing, the value is determined from the file format: header is set to TRUE if and only if the first row contains one fewer field than the number of columns.

17. read.table() : sep the field separator character. Values on each line of the file are separated by this character. If sep = "" (the default for read.table) the separator is ‘white space’, that is one or more spaces, tabs, newlines or carriage returns.

18. read.table() : quote the set of quoting characters. To disable quoting altogether, use quote = "". See scan for the behaviour on quotes embedded in quotes. Quoting is only considered for columns read as character, which is all of them unless colClasses is specified.

19. read.table() : rownames a vector of row names. This can be a vector giving the actual row names, or a single number giving the column of the table which contains the row names, or character string giving the name of the table column containing the row names. If there is a header and the first row contains one fewer field than the number of columns, the first column in the input is used for the row names. Otherwise if row.names is missing, the rows are numbered. Using row.names = NULL forces row numbering. Missing or NULL row.names generate row names that are considered to be ‘automatic’ (and not preserved by as.matrix).

20. read.table() : colnames a vector of optional names for the variables. The default is to use "V" followed by the column number

21. Commands for Reading Data Tables Commands are the same except for defaults: read.table : general table reading command (tab delimited default) read.csv : comma separated values read.delim : tab delimited values

22. Reading Tables: Defaults Separator Decimal Symbol Header Default read.table read.csv read.delim

23. Hands-On Exercise Read both exported data files (the.txt and the.csv) into R objects called: adultReturn (for the.csv) and adultReturnTxt (for the.txt)

24. Checking Data Use head() to show first 6 lines of data frame, tail() shows the last 6 lines > head(adultReturn) > head(adultReturn, 10) > head(adultReturnTxt) > tail(adultReturn, 10) What is the difference between these two data tables?

25. read.table Read.table needs the argument header=T to read in the column headings > adultReturnTxt <- + read.table(“chinook_adult_return_data.txt”, + header = T)

26. Read data using scan Look at help for scan > ?scan

27. Read data using scan > adultReturnScan <-scan + ("chinook_adult_return_data.txt") > adultReturnScan > ?scan This does not look very good…

28. Read data using scan > adultReturnScan <-scan + ("chinook_adult_return_data.txt“, + what = list(“”,””,””,””,””,””)) > adultReturnScan > ?scan This looks better, but the column headers are with the data

29. Read data using scan > adultReturnScan <-scan + ("chinook_adult_return_data.txt“, + what = list(“”,””,””,””,””,””), + skip = 1) > adultReturnScan We can work with this.

30. Quick side note: lists Lists are a collection of many objects of any type You know it is a list because the elements are indexed by double brackets –[[1]], [[2]], etc. Index lists using double brackets

31. Indexing Lists Get the fifth element of the list: > adultReturnScan[[5]] First 10 entries of the fifth element > firstElement <- adultReturnScan[[5]][1:10] > firstElement[1:10] Do this shorter: > adultReturnScan[[5]][1:10]

32. Hands-on Exercise Get the 4 th element of adultReturnTxt Get the 4 th element of adultReturnScan Are these the same?

33. Making a data frame from list > adultReturnScan.df <- data.frame( + dam= adultReturnScan[[1]], + End.Date= adultReturnScan[[2]], + year= adultReturnScan[[3]], + Run=adultReturnScan[[4]], + Adult= as.numeric(adultReturnScan[[5]]), + Jack = as.numeric(adultReturnScan[[6]])) > adultReturnScan.df

34. Subsetting Data Frames > adultReturn[1:6,] Same as: > head(adultReturn)

35. Subsetting Data Frames Look at column names: > names(adultReturn) [1] "Dam" "End.Date" "year" "Run" "Adult" "Jack“ Look at all of Adult column > adultReturn$Adult

36. Subsetting Data Frames > tail(adultReturn$Adult) Same as: > lenData <- length(adultReturn$Adult) > lenData1 <- dim(adultReturn)[1] > lenData == lenData1 > adultReturn$Adult[(lenData-6): + lenData]

37. Subsetting Data Frames Select only observations at Bonneville Dam: > adultReturnBon <- + adultReturn[ adultReturn$Dam == + “BON”,]

38. Hands-on Exercises Make a new data frame called adultReturn2007 that only has observations from 2007, and adultReturn2008 that only has observations from 2008. Then do this again, but remove the column End.date when you do it. Plot Adult versus Juvenile for 2008 (use plot ) Plot Adult in 2008 versus Adult in 2007

39. Add a row to a data frame: rbind > adultReturn1 <- rbind(adultReturn, + c(“FSH”,”31-Oct”,2007,”summer”, + 12345,9876)) Warning message: In `[<-.factor`(`*tmp*`, ri, value = "FSH") : invalid factor level, NAs generated

40. Factors What is a factor: –Categorical data (Nominal) –Ordered data (Ordinal) Factors have many uses: –ANOVA and other categorical analyses –Creating groups for graphs

41. Factors > adultReturn$Run Show the levels of a factor > levels(adultReturn$Run) [1] "spring" "summer"

42. Factors Change the order of levels in gender > adultReturn$Run <- + factor(adultReturn$Run, + levels= c("summer", "spring")) > adultReturn$Run

43. Factors Make a column not a factor > adultReturn$Run <- + as.character(adultReturn$Run) > adultReturn$Run Note that there are now no levels

44. Checking for factors To find out if something is a factor, ask the question: > is.factor(adultReturn$Run) [1] FALSE > is.factor(adultReturn$Dam) [1] TRUE

45. Analyzing All Columns at Once Use sapply to analyze all columns each separately > ?sapply > sapply(adultReturn, FUN = is.factor) Dam End.Date year Run Adult Jack TRUE TRUE FALSE FALSE FALSE FALSE

46. Factors But, I prefer to do my data manipulations without factors, then make data into factors at the time of analysis or graphing

47. Reading Data into R Factorless Look at help again: > ?read.table stringsAsFactors logical: should character vectors be converted to factors? Note that this is overridden by as.is and colClasses, both of which allow finer control. > adultReturn <- + read.csv(“chinook_adult_return_data.csv”, + header = T, stringsAsFactors = F)

48. Making Factors > runFact <- + as.factor(adultReturn$Run) as.factor makes characters or numbers into factors

49. Hands-on Exercise Make a new object called yearFact that is the column year as a factor

50. Graphing with Factors Load the lattice library: > library(lattice) > histogram(~adultReturn$Adult) > histogram(~adultReturn$Adult|runFact) > histogram(~adultReturn$Adult| + runFact*yearFact) Graphs groups of groups Graphs groups The power of factors

51. Calculating Column Means > mean(adultReturn$Adult) [1] NA Look up help for mean > ?mean

52. mean() mean(x, trim = 0, na.rm = FALSE,...) Arguments na.rm: a logical value indicating whether NA values should be stripped before the computation proceeds.

53. Calculating Column Means > mean(adultReturn$Adult, na.rm = T) [1] 33489.71

54. Hands-on Exercise Calculate the mean for the Jack column

55. Check for NA values Testing is something is NA: is.na() > is.na(adultReturn$Adult)

56. Check for NA values Ask if any of the values are NA: > any(is.na(adultReturn$Adult)) Ask if all of the values are NA: > all(is.na(adultReturn$Adult))

57. Omitting NA from Calculations Omitting NA from calculations: For many functions you can use na.rm = TRUE to get rid of NA values This does not work for all functions –And often does not work for the more “obscure” contributed package functions that you want to use for ecological analysis

58. Subsetting Data Frames What if we want only to take the mean of Adult for year 2008? > Adult2007 <- adultReturn$Adult + [adultReturn$year==2007] > mean(Adult2007)

59. Subsetting Data Frames What if we want only to take the mean of Adult for year 2008 and spring run? > AdultSpring2007 <- adultReturn$Adult + [(adultReturn$year==2007) & + (adultReturn$Run==“spring”)] > mean(AdultSpring2007)

60. Hands-on Exercise Make a new data frame called adultMeans with the means for Adults in spring and summer runs in both 2007 and 2008. It should look like this: YearRunMean 2007spring 2007summer 2008spring 2008summer

61. Hands-on Exercise Find the command to write a table to a.csv file. Now, write the data frame you just created ( adultMeans ) to a file called adultMeans.csv Open this file up in Excel

62. Advanced Topics Aggregate Stack/unstack Time series objects

63. Aggregate by factors > ?aggregate > aggregate(adultReturn$Adult, by = list(yearFact, runFact), FUN = mean, na.rm=T)

64. Aggregation and Stacking stack(dataFrame) – turns an n by m data frame into an n*m by 2 data frame unstack(dataFrame, values~ind) turns an n*m by 2 data frame into an n by m data frame

65. Time Series Create a time series numjobs > numjobs <- c(982,981,984,982,981,983, 983,983,983,979,973,979, 974,981,985,987,986,980, 983,983,988,994,990,999)

66. Time Series Make numjobs a time series object: > numjobs <- ts(numjobs, start=1995, frequency = 12) Plot the time series: > plot(numjobs)