File I/O 1. Goals of this chapter 2. General Concept, lots of examples 3. High-level I/O 1. Built-in functions that do everything in one step 4. Low-level I/O 1. Built-in functions that are specific to one task 1

1. Goals of this chapter Understand why external files are greatly used in programming, such as.txt,.xls,.dat… Recognize when to use a file vs. hardcoding data Understand how Matlab sees any file Define and recognize delimiting-characters Depending on which type of file, choose an appropriate built-in function Remember the overall idea behind each built-in function Remember the logical order of reading/writing a file 2

2. General Concept What is an “input-file”? It is file that contains the givens needed to solve the entire problem, or maybe just a part of the problem. LOTS of data (as little as 5 lines to billions of lines…) What is an “output-file”? It is a file which contains the results (once the problem has been solved). Most common example: I _ V O _ C _ 3 Some examples of input files…

Property Sales in Flagler (North of Daytona Beach) 4 INPUT FILE

Property Listing in Ormond Beach (North of Daytona Beach) 5 Online data dated March 21 st 2010.

Hurricane Data 6 Online data dated March 21 st 2010, nhc.noaa.gov INPUT FILE

Hurricane Data... What? 7 Same as previous file… A text file doesn’t always look pretty. Try different softwares to open it until one shows clearly the columns and rows! -Notepad -Wordpad -Internet explorer -Firefox -word INPUT FILE

Coefficient of Friction 8.xls, Excel format.txt, text format(INPUT FILES)

File Formats Files are stored in essentially two different formats: ASCII text, and binary. ASCII text consists entirely of characters which have meaning within the ASCII table. The only special codes are already present in the ASCII table, such as newlines and tabs. Notepad, for example, works only with ASCII text files.

File Formats, cont. Binary files may contain some ASCII text, but will also contain data which has meaning to the program which was intended to use the file. MS Word, for example, embeds codes for bold, underline, HTML, etc in the file. These codes only have meaning to Word – not to programs like Notepad. Word can work with Notepad files, but Notepad cannot work with Word files.

General Concept, cont. When reading data from a file, it continually requests values from the file data stream until the end of the file is reached. As the data is received, the program must identify the delimiting characters (tabs, spaces, enter key, special characters..) and reformat the data to reconstruct the organization of the data as represented in a file. When writing data to a file: to preserve the organization of the data, the appropriate delimiting characters must be inserted into the stream of information. 11 FileProgram An I nput Stream of information ProgramFile An O uput Stream of information

General Concept, cont. Using the data contained inside input-files Analyze, min, max, average Estimate trends (stock market) Use it as a database for one or two variables (coefficient of friction) Use it to give user possibilities to pick from Plot Using output-files, once populated with data Print them them Publish in a report Reuse as an-input file for another software 12 Examples…

Analyzing Flagler Sales Price of properties function of square footage Linear Fitting of all data points (red) Standard deviation (green) 13

Attendance Sheet 14 OUTPUT FILE

Analyze grades 15 INPUT FILE OUTPUT FILE

Another Hurricane Data This one shows all coordinates (latitude/longitude) of the storm at different times during the day, as well as the wind speed. 16 INPUT FILE

Plot Hurricane Data This is a final project in JAVA (CS225) in Fall 2007 that analyzed the hurricane input file. 17

General Concept, cont. Although there are only a few ADVANTAGES to using files, the ones that do exist are sufficient in themselves!!! LOTS of data, usually contained in very little variables (possibly only 1 cell-array!) The software is developed knowing that the file cannot change format, but definitely knowing that the file can grow in size! Although the hurricane data obviously gets updated yearly, the software to analyze the data does not change a bit! User does not have to enter the data: less prone to errors. Long term saving/backup of data. 18

General Concept, cont. Only certain formats can be analyzed.txt.xls.csv.dat.jpg.png.gif.trk The following is not possible under MATLAB.doc 19

NOW THAT YOU HAVE AN IDEA.. How is all this done? 20

General Concept, cont. There are two types of built-in functions High-level functions: one (and 1 only) command opens the file, read or writes data, and closes the file. All is stored immediately in variable(s). Low-level functions: 1. one command needs to open the file, 2. another command (maybe more than one) needs to read or write the data, 3. and finally another command closes the file. Learn both. High-level functions can only be used with some very specific files. 21

3. HIGH LEVEL FUNCTIONS In one command-line, the function grabs all the data 22

High Level, cont. Load data from external sources (INPUT) Save data back to a file (OUTPUT) When reading data, it is a good habit to explore the data in the file first. What type of file is it? (excel, text, jpg..) Find the organization overall (data in rows, or data in columns) Recognize the delimiters (space, tabs, new lines, dash, colon, or any specific symbol. What makes it obvious it is a new column? What makes it obvious it is a new row?) Recognize the data types (all numerical? all strings? Or combination of both) 23

High Level, cont. NUMERICAL AND CHARACTERS Excel worksheet xlsread() (sometimes does not work depending on which system you have) NUMERICAL DATA ONLY (of course, not in an excel file…) General-delimited dlmread() Tab-separated dlmread() (same function as tab) Comma-separated csvread() Most can be interchanged. These are powerful functions. Remember xlsread() and dlmread() mostly! 24

Whatever you do, the files (text,.m,.mat) should all be in the same directory for function calls to work! 25

3.1 Using xlsread() The function recognizes the data types of each cell, and groups similar data- type together >>[v1 v2 v3] = xlsread('grades.xlsx') First variable (v1) will have all the numerical data Second variable (v2) will have all the character data Third variable (v3) groups ALL the data together in one cell-array. Not column by column >> [names age grade] = xlsread('grades.xlsx') 26

xlsread(), cont nd return value: characters 3 rd return value: The data all together 1 st return value: numerical

xlsread(), cont. txt = nbs = 'Name' 'Age' 'Grade' 'Fred' '' '' 'joe' '' '' 'SaLLy' '' '' 'CharliE' '' '' 'mary' '' '' 'Ann' '' '' raw = 'Name' 'Age' 'Grade' 'Fred' [ 19] [ 78] 'joe' [ 22] [ 83] 'SaLLy' [ 98] [ 99] 'CharliE' [ 21] [ 56] 'mary' [ 23] [ 89] 'Ann' [ 19] [ 51] >> [txt nbs raw] = xlsread('grades.xlsx‘)   Variable names are up to the programmer. If placed in wrong order by mistake… BAD. The order of the return values is important.

xlsread(), cont. Simply omit the 2 nd and 3 rd return value to collect only numerical values. values = textread(‘grades.xlsx’); If a project needs all the data together, collect the 2 nd and 3 rd return values into a dummy variable. [trash trash data] = textread(‘grades.xlsx’); If there happens to be ‘holes’ in the spreadsheet, Matlab fills it with a NaN value (not a number). The function isnan() can help determine where those ‘holes’ are. (This function will not work under Citrix, since Excel is not installed on Citrix. Come to the lab to do labs and homeworks.) 29

Writing to an excel sheet? After manipulating, analyzing, or creating an array, excel spreadsheets can also be written-to using: xlswrite(,,, ) The and arguments are optional. 30 clc clear %create phony data table = rand(5,3)*100; %print to excel xlswrite('testingTesting.xls',table) Then use excel to plot and analyze the data!

3.2 Using delimited files – Numbers only Columns are delimited by tabs, and/or spaces. Rows are delimited by the new line character (enter key, invisible to the human eye). 31 Matlab automatically filled in with zeros where data was missing.

Same function used for Comma-delimited numbers Added feasibility to ‘skip’ columns 32

High-Level, conclusion The variables are automatically created by MATLAB. Sometimes a regular array [ ] Other times, a cell array{ } Use knowledge from previous chapter to analyze the data in the arrays. FOR loops, A(position), A(row,col), A{row,col}, max(arrayName), mean(arrayName), sort(..)