1. COSC 1306—COMPUTER SCIENCE AND PROGRAMMING PYTHON FUNCTIONS Jehan-François Pâris jfparis@uh.edu

2. Module Overview We will learn how to read, create and modify files –Pay special attention to pickled files They are very easy to use!

3. The file system Provides long term storage of information. Will store data in stable storage (disk) Cannot be RAM because: – Dynamic RAM loses its contents when powered off – Static RAM is too expensive –System crashes can corrupt contents of the main memory

4. Overall organization Data managed by the file system are grouped in user-defined data sets called files The file system must provide a mechanism for naming these data –Each file system has its own set of conventions –All modern operating systems use a hierarchical directory structure

5. Windows solution Each device and each disk partition is identified by a letter –A: and B: were used by the floppy drives –C: is the first disk partition o f the hard drive –If hard drive has no other disk partition, D: denotes the DVD drive Each device and each disk partition has its own hierarchy of folders

6. Windows solution C: Windows Users Second disk D: Program Files Flash drive F:

7. UNIX/LINUX organization Each device and disk partition has its own directory tree –Disk partitions are glued together through the operation to form a single tree Typical user does not know where her files are stored

8. UNIX/LINUX organization Root partition bin usr / Other partition The magic mount Second partition can be accessed as /usr

9. Mac OS organization Similar to Windows –Disk partitions are not merged –Represented by separate icons on the desktop

10. Accessing a file (I) Your Python programs are stored in a folder AKA directory –On my home PC it is C:\Users\Jehan-Francois Paris\Documents\ Courses\1306\Python All files in that directory can be directly accessed through their names – "myfile.txt"

11. Accessing a file (II) Files in subdirectories can be accessed by specifying first the subdirectory – Windows style: "test\\sample.txt" – Note the double backslash – Linux/Unix/Mac OS X style: "test/sample.txt" – Generally works for Windows

12. Why the double backslash? The backslash is an escape character in Python –Combines with its successor to represent non-printable characters ‘\n’ represents a newline ‘\t’ represents a tab –Must use ‘ \\ ’ to represent a plain backslash

13. Accessing a file (III) For other files, must use full pathname – Windows Style: "C:\\Users\\Jehan-Francois Paris\\ Documents\\Courses\\1306\\Python\\ myfile.txt"

14. Accessing file contents Two step process: –First we open the file –Then we access its contents Read Write When we are done, we close the file.

15. What happens at open() time? The system verifies –That you are an authorized user –That you have the right permission Read permission Write permission Execute permission exists but doesn’t apply and returns a file handle / file descriptor

16. The file handle Gives the user –Direct access to the file No directory lookups –Authority to execute the file operations whose permissions have been requested

17. Python open() open(name, mode = ‘r’, buffering = -1) where – name is name of file – mode is permission requested Default is ‘ r ’ for read only – buffering specifies the buffer size Use system default value (code -1)

18. The modes Can request – ‘r’ for read-only – ‘w’ for write-only Always overwrites the file –‘a’ for append Writes at the end – ‘r+’ or ‘a+’ for updating (read + write/append)

19. Examples f1 = open("myfile.txt") same as f1 = open("myfile.txt", "r") f2 = open("test\\sample.txt", "r") f3 = open("test/sample.txt", "r") f4 = open("C:\\Users\\Jehan-Francois Paris\\ Documents\\Courses\\1306\\Python\\myfile.txt")

20. Reading a file Three ways: –Global reads –Line by line –Pickled files

21. Global reads fh.read() –Returns whole contents of file specified by file handle fh –File contents are stored in a single string that might be very large

22. Example f2 = open("test\\sample.txt", "r") bigstring = f2.read() print(bigstring) f2.close() # not required

23. Output of example To be or not to be that is the question Now is the winter of our discontent –Exact contents of file ‘test\sample.txt’

24. Line-by-line reads for line in fh : # do not forget the column #anything you want fh.close() # not required

25. Example f3 = open("test/sample.txt", "r") for line in f3 : # do not forget the column print(line) f3.close() # not required

26. Output To be or not to be that is the question Now is the winter of our discontent –With one or more extra blank lines

27. Why? Each line ends with an end-of-line marker print(…) adds an extra end-of-line

28. Trying to remove blank lines print('----------------------------------------------------') f5 = open("test/sample.txt", "r") for line in f5 : # do not forget the column print(line[:-1]) # remove last char f5.close() # not required print('-----------------------------------------------------')

29. The output ---------------------------------------------------- To be or not to be that is the question Now is the winter of our disconten ----------------------------------------------------- The last line did not end with an EOL!

30. A smarter solution (I) Only remove the last character if it is an EOL – if line[-1] == ‘\n’ : print(line[:-1] else print line

31. A smarter solution (II) print('----------------------------------------------------') fh = open("test/sample.txt", "r") for line in fh : # do not forget the column if line[-1] == '\n' : print(line[:-1]) # remove last char else : print(line) print('-----------------------------------------------------') fh.close() # not required

32. It works! ---------------------------------------------------- To be or not to be that is the question Now is the winter of our discontent -----------------------------------------------------

33. Making sense of file contents Most files contain more than one data item per line –COSC713-743-3350 UHPD 713-743-3333 Must split lines – mystring.split(sepchar) where sepchar is a separation character returns a list of items

34. Splitting strings >>> text = "Four score and seven years ago" >>> text.split() ['Four', 'score', 'and', 'seven', 'years', 'ago'] >>>record ="1,'Baker, Andy', 83, 89, 85" >>> record.split(',') [' 1', "'Baker", " Andy'", ' 83', ' 89', ' 85'] Not what we wanted!

35. Example # how2split.py print('----------------------------------------------------') f5 = open("test/sample.txt", "r") for line in f5 : words = line.split() for xxx in words : print(xxx) f5.close() # not required print('-----------------------------------------------------')

36. Output ---------------------------------------------------- To be … of our discontent -----------------------------------------------------

37. Other separators (I) Commas –CSV Excel format Values are separated by commas Strings are stored without quotes –Unless they contain a comma “Doe, Jane”, freshman, 90, 90 –Quotes within strings are doubled

38. Other separators (II) Tabs( ‘\t’) – Advantages: Your fields will appear nicely aligned Spaces, commas, … are not an issue – Disadvantage: You do not see them –They look like spaces

39. Why it is important When you must pick your file format, you should decide how the data inside the file will be used: –People will read them –Other programs will use them –Will be used by people and machines

40. An exercise Converting our output to CSV format –Replacing tabs by commas Easy –Will use string replace function

41. First attempt fh_in = open('grades.txt', 'r') # the 'r' is optional buffer = fh_in.read() newbuffer = buffer.replace('\t', ',') fh_out = open('grades0.csv', 'w') fh_out.write(newbuffer) fh_in.close() fh_out.close() print('Done!')

42. The output Alice9090909090 Bob8585858585 Carol7575757575 becomes Alice,90,90,90,90,90 Bob,85,85,85,85,85 Carol,75,75,75,75,75

43. Dealing with commas (I) Work line by line For each line –split input into fields using TAB as separator –store fields into a list Alice9090909090 becomes [‘Alice’, ’90’, ’90’, ’90’, ’90’, ’90’]

44. Dealing with commas (II) –Put within double quotes any entry containing one or more commas –Output list entries separated by commas ['"Baker, Alice"', 90, 90, 90, 90, 90] becomes "Baker, Alice",90,90,90,90,90

45. Dealing with commas (III) Our troubles are not over: –Must store somewhere all lines until we are done –Store them in a list

46. Dealing with double quotes Before wrapping items with commas with double quotes replace –All double quotes by pairs of double quotes – 'Aguirre, "Lalo" Eduardo' becomes 'Aguirre, ""Lalo"" Eduardo' then '"Aguirre, ""Lalo"" Eduardo"'

47. General organization (I) linelist = [ ] for line in file –itemlist = line.split(…) –linestring = '' # empty string –for each item in itemlist remove any trailing newline double all double quotes if item contains comma, wrap add to linestring

48. General organization (II)

49. General organization (III)

50. The program (I) # betterconvert2csv.py """ Convert tab-separated file to csv """ fh = open('grades.txt','r') #input file linelist = [ ] # global data structure for line in fh : # outer loop itemlist = line.split('\t') # print(str(itemlist)) # just for debugging linestring = '' # start afresh

51. The program (II) for item in itemlist : #inner loop item = item.replace('"','""') # for quotes if item[-1] == '\n' : # remove it item = item[:-1] if ',' in item : # wrap item linestring += '"' + item +'"' + ',' else : # just append linestring += item +',' # end of inside for loop

52. The program (III) # must replace last comma by newline linestring = linestring[:-1] + '\n' linelist.append(linestring) # end of outside for loop fh.close() fhh = open('great.csv', 'w') for line in linelist : fhh.write(line) fhh.close()

53. Notes Most print statements used for debugging were removed –Space considerations Observe that the inner loop adds a comma after each item –Wanted to remove the last one Must also add a newline at end of each line

54. The input file Alice9090909090 Bob8585858585 Carol 7575757575 Doe, Jane9090 90 80 70 Fulano, Eduardo "Lalo"90909090

55. The output file Alice,90,90,90,90,90 Bob,85,85,85,85,85 Carol,75,75,75,75,75 "Doe, Jane",90,90,90,80,75 "Fulano, Eduardo ""Lalo""",90,90,90,90

56. Mistakes being made (I) Mixing lists and strings: –Earlier draft of program declared linestring = [ ] and did linestring.append(item) – Outcome was ['Alice,', '90,'. … ] instead of 'Alice,90, …'

57. Mistakes being made (II) Forgetting to add a newline –Output was a single line Doing the append inside the inner loop: –Output was Alice,90 Alice,90,90 Alice,90,90,90 …

58. Mistakes being made Forgetting that strings are immutable: –Trying to do linestring[-1] = '\n' instead of linestring = linestring[:-1] + '\n' – Bigger issue: Do we have to remove the last comma?

59. Could we have done better? (I) Make the program more readable by decomposing it into functions –A function to process each line of input do_line(line) –Input is a string ending with newline –Output is a string in CSV format –Should call a function processing individual items

60. Could we have done better? (II) –A function to process individual items do_item(item) –Input is a string –Returns a string With double quotes "doubled" Without a newline Within quotes if it contains a comma

61. The new program (I) def do_item(item) : item = item.replace('"','""') if item[-1] == '\n' : item = item[:-1] if ',' in item : item ='"' + item +'"' return item

62. The new program (II) def do_line(line) : itemlist = line.split('\t') linestring = '' # start afresh for item in itemlist : linestring += do_item(item) +',' linestring += '\n' return linestring

63. The new program (III) fh = open('grades.txt','r') linelist = [ ] for line in fh : linelist.append(do_line(line)) fh.close()

64. The new program (IV) fhh = open('great.csv', 'w') for line in linelist : fhh.write(line) fhh.close()

65. Why it is better Program is decomposed into small modules that are much easier to understand –Each fits on a PowerPoint slide

66. The break statement Makes the program exit the loop it is in In next example, we are looking for first instance of a string in a file –Can exit as soon it is found

67. Example (I) searchstring= input('Enter search string:') found = False fh = open('grades.txt') for line in fh : if searchstring in line : print(line) found = True break

68. Example (II) if found == True : print("String %s was found" % searchstring) else : print("String %s NOT found " % searchstring)

69. Flags A variable like found –That can either be True or False –That is used in a condition for an if or a while is often referred to as a flag

70. A dumb mistake Unlike C and its family of languages, Python does not let you write – if found = True for – if found == True There are still cases where we can do mistakes!

71. Example >>> b = 5 >>> c = 8 >>> a = b = c >>> a 8 >>> a = b == c >>> a True

72. HANDLING EXCEPTIONS

73. When a wrong value is entered When user is prompted for – number = int(input("Enter a number: ") and enters –a non-numerical string a ValueError exception is raised and the program terminates Python a programs catch errors

74. The try… except pair (I) try: except Exception as ex: Observe –the colons –the indentation

75. The try… except pair (II) try: except Exception as ex: If an exception occurs while the program executes the statements between the try and the except, control is immediately transferred to the statements after the except

76. A better example done = False while not done : filename= input("Enter a file name: ") try : fh = open(filename) done = True except Exception as ex: print ('File %s does not exist' % filename) print(fh.read())

77. An Example (I) done = False while not done : try : number = int(input('Enter a number:')) done = True except Exception as ex: print ('You did not enter a number') print ("You entered %.2f." % number) input("Hit enter when done with program.")

78. A simpler solution done = False while not done myinput = (input('Enter a number:')) if myinput.isdigit() : number = int(myinput) done = True else : print ('You did not enter a number') print ("You entered %.2f." % number) input("Hit enter when done with program.")

79. PICKLED FILES

80. Pickled files import pickle –Provides a way to save complex data structures in a file –Sometimes said to provide a serialized representation of Python objects

81. Basic primitives (I) dump(object,fh) –appends a sequential representation of object into file with file handle fh – object is virtually any Python object – fh is the handle of a file that must have been opened in 'wb' mode b is a special option allowing to write or read binary data

82. Basic primitives (II) target = load( filehandle) –assigns to target next pickled object stored in file filehandle – target is virtually any Python object – filehandle id filehandle of a file that was opened in rb mode

83. Example (I) >>> mylist = [ 2, 'Apples', 5, 'Oranges'] >>> mylist [2, 'Apples', 5, 'Oranges'] >>> fh = open('testfile', 'wb') # b is for BINARY >>> import pickle >>> pickle.dump(mylist, fh) >>> fh.close()

84. Example (II) >>> fhh = open('testfile', 'rb') # b is for BINARY >>> theirlist = pickle.load(fhh) >>> theirlist [2, 'Apples', 5, 'Oranges'] >>> theirlist == mylist True

85. What was stored in testfile? Some binary data containing the strings 'Apples' and 'Oranges'

86. Using ASCII format Can require a pickled representation of objects that only contains printable characters –Must specify protocol = 0 Advantage: –Easier to debug Disadvantage: –Takes more space

87. Example import pickle mydict = {'Alice': 22, 'Bob' : 27} fh = open('asciifile.txt', 'wb') # MUST be 'wb' pickle.dump(mydict, fh, protocol = 0) fh.close() fhh = open('asciifile.txt', 'rb') theirdict = pickle.load(fhh) print(mydict) print(theirdict)

88. The output {'Bob': 27, 'Alice': 22} {'Bob': 27, 'Alice': 22}

89. What is inside asciifile.txt? (dp0VBobp1L27Ls V Alicep2L22Ls.

90. Dumping multiple objects (I) import pickle fh = open('asciifile.txt', 'wb') for k in range(3, 6) : mylist = [i for i in range(1,k)] print(mylist) pickle.dump(mylist, fh, protocol = 0) fh.close()

91. Dumping multiple objects (II) fhh = open('asciifile.txt', 'rb') lists = [ ] # initializing list of lists while 1 : # means forever try: lists.append(pickle.load(fhh)) except EOFError : break fhh.close() print(lists)

92. Dumping multiple objects (III) Note the way we test for end-of-file ( EOF ) – while 1 : # means forever try: lists.append(pickle.load(fhh)) except EOFError : break

93. The output [1, 2] [1, 2, 3] [1, 2, 3, 4] [[1, 2], [1, 2, 3], [1, 2, 3, 4]]

94. What is inside asciifile.txt? (lp0L1LaL2La.(lp0L1LaL2LaL3La.(lp0L1LaL2L aL3LaL4La.

95. Practical considerations You rarely pick the format of your input files – May have to do format conversion You often have to use specific formats for you output files – Often dictated by program that will use them Otherwise stick with pickled files !