INTRO2CS Tirgul 10 - Files and Networks 1

What will we see today?  Handling files  Network  OOP - Top down design 2

Files - why are they needed? ❏ The containers we know (list\tuple etc.) are limited in the amount of information they can hold (~536M items on a 32bit system [ref] ) - files can maintain a greater capacity.ref ❏ The output format we are familiar with (print to screen) is lost with every system restart and cannot be shared between different programs. Files’ information is maintained across system shut downs\reboots, and could be accessed by different programs. 3

Files - What are they?  A file is an ordered sequence of bytes.  Upon reading a file, a suitable program interprets the content of a file (playing music, displaying images and text).  To create files, write to files and read from files Python defines a custom file handling API ( Application Programming Interface). 4

Creating a file object f = open(fileName)  open is python built-in function which returns a file object, with which we can work with files.  fileName is a string which indicates the file location within the file system (absolute or relative to the folder from which the program is running). 5

File ‘open’ing mode f = open(fileName,[mode])  When opening a file we should decalre our intentional use of this file.  Protects us against undesired consequences.  Two basic types of modes :  Read (default)  Write 6

File ‘open’ing mode (2) 7 ●The multiple modes “defend” us so we would not do what we don’t mean to (e.g unwanted changes in the file) ●We have to declare in advance what are our intentions toward the open file (do we wish to manipulate it? read only?)

File ‘open’ing mode f = open(fileName,[mode])  mode =  ‘r’ - read only (default)  ‘w’ - write (an existing file with the same name will be erased)  ‘a’ - append (data written to the file is added to its end).  ‘r+’ - reading and writing 8

open in read mode (defualt) f = open(fileName,'r')  If fileName does not exist : Traceback (most recent call last): File " ", line 1, in f = open('filename') FileNotFoundError: [Errno 2] No such file or directory: 'filename' 9

open in write mode f = open(fileName,'w')  If fileName does not exist:  python will create such.  If fileName does exist:  python will overide (delete) the existing version and replace it with a new (the current) file. – Beware! 10

open in append mode f = open(fileName,'a')  If fileName does not exist:  python will create such.  If fileName does exist:  python will add any writen data to the end of the file. 11

12 n = f.write(s)  Write the string s to the file f.  n = number of written characters. f = open("file.txt") n = f.write('Happy New Year') write to file

write to file (2) f = open("file.txt", 'a') n = f.write('Happy New Year')  write adds content to the end of the file but does not intsert line breaks. E.g : f.write('Don') f.write('key')  f content : Happy New YearDonkey 13

write multiple lines to file f.writelines(seq)  Write the strings contained in seq to the file one by one.  For example: my_strings = ['Don', ‘key'] f.writelines(my_strings)  f content : Donkey  writelines expects a list of strings, while write expects a single string. 14

closeing a file f.close()  After completing the usage of the file, it should be closed using the close method. Free the file resource for usage of other processes. Some environments will not allow read-only and write modes opening simultaneously 15

closeing a file f.close() Flush the content of the file. Whenever the write is called, it is not guaranteed the content would be placed in the file right away, usually it is ‘buffered’ and written in larger chunks. close ing the file guarantees that all its content would be written promptly. Could also be achieved by f.flush() 16

the open with statement  Things don’t always go as smoothly as we plan, and sometimes causes programs to crash.  E.g. trying a 0 division.  If the program crashes, we don’t get a chance to close (and free the resource of) the open file.  To verify the appropriate handling of files, even in case of program crash we can make use of the with statement. 17

the with statement with open(file_path, 'w') as our_open_file: #we can work here with the open file and be sure it is properly closed in every scenario  The with statement guarantees that even if the program crashed inside the with block, it will still be properly closed. 18

19 the with statement  so to open a file, process its contents, and to make sure to close it, you can simply do: with open("x.txt") as f: data = f.read() do something with data  More info: 1. mts.html#withhttps://docs.python.org/3/reference/compound_st mts.html#with 2.

20  In Python, the file object has been equipped with __enter__ and __exit__ methods; the former simply returns the file object itself, and the latter closes the file: >>> f = open("x.txt") >>> f <open file 'x.txt', mode 'r' at 0x00AE82F0> >>> f.__enter__() <open file 'x.txt', mode 'r' at 0x00AE82F0> >>> f.read(1) 'X' >>> f.__exit__(None, None, None) >>> f.read(1) Traceback (most recent call last): File " ", line 1, in ValueError: I/O operation on closed file the with statement

You can’t write to a closed file f.close() f.writelines(['hi','bi','pi']) Traceback (most recent call last): File " ", line 1, in f.writelines(['hi','bi','pi']) ValueError: I/O operation on closed file. 21

reading from file f.read(n)  Read at most n characters from f (default = all file) f.readline()  Read one line. f.readlines()  Read all lines in the file - returns a list of strings ( list of the file’s lines). 22

reading a 'w' mode file f = open(fileName,'w') f.read() Traceback (most recent call last): File " ", line 1, in f.read() io.UnsupportedOperation: not readable  Use ‘r+’ mode for reading and writing 23

Iterating over a file When we read a file, Python defines a pointer (an iterator) which advances with every consecutive reading. f.readline() # 1st time >> Hi :) f.readline()# 2nd time >> Bi :( f content Hi :) Bi :( Mitz Paz 24

Iterating over a file - tell The tell() method returns the current position within the file. f.readline() # 1st time >> Hi :) f.tell() >> 5 f content Hi :) Bi :( Mitz Paz 25

Access (e.g. print) all the lines in a file f = open(fileName) for line in f: print(line)  When do the iteration stop? When reaching end-of-file (EOF)  Its an iterator! If the iterator already reached EOF, it will print nothing. 26

Network  Client-Server model  Ports and sockets  Python’s socket module  Listen  Bind  Close  Send  Recieve 27

Client-server model  Distributed application structure  Definition of two entities :  Clients – Requester of a particular service  Servers – Providers of a particular service 28

Distributed system (example)  The task – easy access, backup, and update option to multiple files.  The solution – Dropbox!  The files are stored at Dropbox’s servers  We (the clients) update the files over the internet (inter-network) 29

Distributed system (example)  Where is the implementation?  It is not at Dropbox – they only store.  It is not on our computers – we only update.  The task solution is distributed via the network. 30

Networks - motivation  Why not simply copy all the servers’ abilities to the clients?  Awful waste of resources.  Agreed “meeting point“ – if we wanted to implement a chat server :  Client based solution – send each message to all clients and check locally if this is the right recipients – waste of resources, hard to implement.  Client-server based solution – have all clients messages directed to a single entity (the server) which efficiently manages them. 31

Who can be a client? Server?  We are used to talk about servers (“ooh, Google servers are so strong!”) as a synonym for a powerful computer.  But, being a client or a server is just a role a specific program takes on itself.  We can run several programs on the same machine :  Some would be clients that will reach other servers for service.  Some would be servers providing services to other clients, or even to a client on the current machine. 32

Famous client-server apllications   Network printing  World wide web 33

Communication over network  Network communication is defined on some hardware (e.g. – optic fibers, metal wires)  To transfer signals (e.g. changes in light intensity or electrical voltage) the two communicating sides should agree on a protocol  What is the address of each?  How to start a conversation?  What if someone does not answer?  Many many more. 34

Ports and sockets  (leaving a lot of details that will be revealed in a dedicated networks course) TCP-IP is one such communication protocol.  An IP defines the address of a computer  in the form of 4, dot separated 8bit numbers, e.g. :  A port defines to which application (or process) the communication in a specific address is targeted. E.g. :  Port 80 = HTTP, 443 = HTTPS  Port 20 = FTP  Port 6881 = Bittorent TCP-IP Illustrated Volume 1 The Protocols, W. Richard Stevens, 1994 Addison Wesley 35

Ports and sockets  The combination of an IP and a port defines a socket.  Sockets let us create a dedicated communication channel between two network entities. To\from this channel we can (similar to files syntax and terminology):  Write – to transfer information to the other side.  Read – to receive any information written to the channel by the other side. TCP-IP Illustrated Volume 1 The Protocols, W. Richard Stevens, 1994 Addison Wesley 36

Networks with Python  The socket module lets us create and manage network communication via Python.  Creating a new socket object: import socket my_socket = socket.socket()  Python’s socket object implement the general notion of network sockets and abstracts the “lower levels” of communication.  For a full API see :  For a nice tutorial :

connect ing a socket - client  Connecting a socket requires the definition of:  Target address ( host ) : IP\host name.  Target port : should be identical between both sides. client_socket = socket.socket() host = ' ' port = 1024 #identical to server’s address = (host, port) client_socket.connect(address) 38

bind ing a socket - server  Binding a socket = Declaring it is dedicated for a communication via a specific address. server_socket = socket.socket() my_name = socket.gethostname() port = 1024#identical to client’s address = (my_name, port) server_socket.bind(address) 39

listen to a socket - server  listen : the dedicated communication channel (= the socket) will wait for client requests for connections.  If a connection is in progress and another request arrives, we put it in a queue. The queue size should be defined when we start to listen server_socket.listen(5) ■the argument to listen tells the socket library that we want it to queue up as many as 5 connect requests (the normal max) before refusing outside connections. 40

listen to a socket - server  Each socket object can maintain only a single communication channel simultaneously.  If additional request for connections arrive while communication is in progress we (can) queue them. server_socket.bind(address) QUED_CONN = 0 server_socket.listen(QUED_CONN) 41

accept from a socket - server  Accepting communication could only be from a bind ed socket.  When the socket receives a request it returns:  a new socket for sending and receiving data.  the address of the other side. server_socket.listen(QUED_CONN) conn, addr = server_socket.accept() 42

String  Bytes  String  Q : What information can we write to a socket?  A : Bytes!  Any string we want to write to the socket (=send over the communication channel) we should first encode as bytes. Any information read from the port should be decoded from bytes to string.  Encoding : The process of translating characters to bytes\numbers.  Famous encoding methods : UTF8, ASCII 43

 bytes : when creating a bytes object from string we should declare what encoding to use (example coding = utf8) : >> s = bytes('hi!','utf8') >> b‘hi!' # b stands for bytes >> s + 'bi' Traceback (most recent call last): File " ", line 1, in s + 'bi' TypeError: can't concat bytes to str Encoding String  Bytes 44

 We decode using the same encoding method >> s = bytes('hi!','utf8') >> b'hi!' >> s.decode('utf8') >> 'hi!' Dcoding Bytes  String 45

Socket write\read  Send messages using the sendall method : my_socket.connect(address) my_socket.sendall(string_as_bytes)  Receive using the recv method.  Maximum amount of data to be received per single message is defined by BUFFER_SIZE. BUFFER_SIZE = 1024 msg = my_socket.recv(BUFFER_SIZE) 46

close ing a socket  Same as files, when communication is terminated, the socket should be closed: my_socket.close() 47

s = socket.socket() s.bind((HOST, PORT)) s.listen(0) #Waiting for client connections conn, addr = s.accept() # Connected! While True: encoded_data = conn.recv(BUFF_SIZE) if not encoded_data: break #else, do something with data (decode) reply_msg = bytes('Bi!', 'utf-8') conn.sendall(reply_msg) conn.close() Client-server communication example Client Server s = socket.socket() s.connect((HOST, PORT)) msg_to_srvr = 'hi!' encoded_msg = bytes(msg_to_srvr,'utf-8') s.sendall(encoded_msg) data = s.recv(BUFF_SIZE) s.close() print('Received', data) 48

OOP - revisited  Motivation – why using OOP?  How to OOP?  Top-down design  Good practices 49

Why OOP  OOP lets us group information pieces that are conceptually related into a single entity – a class.  We do not need to ‘keep in mind’ the multiple parameter and their inter-relation.  Rather, all related parameters are accessed via a single class instance. 50

Why OOP  OOP lets us abstract complex notions and describe them in an intuitive every-day language.  Consider Python’s list object. List is an intuitive notion which is actually not trivial to implement:  How is the order between elements defined?  How each element knows who is the next element?  How do we insert elements in the middle of the list?  How do we know where the list starts and ends? 51

Why OOP  Have you ever wondered what happens “under the surface” when we increase the size of an, until now, fixed size list?  No! because we have the list class which abstract all of this for us. 52

Top-down design  Last tirgul We have seen “bottom-up” approach, where we have defined the building blocks of our structure and then assembled them together.  This time we go the other way around:  We define the high level requirements, describe the API which meets them and work our way to the finer details. 53

The task – represent the notion of family  We’ll start designing and every time we think of a new required element, we will write down an API with good documentation and leave the actual implementation for later.  We count on our, yet to be implemented, API to do what we state it would do. To do list 54

The task – represent the notion of family  We want to represent and work with the general concept of family : lets define the Family class. Implementation class Family: ''‘ Represent a family''' pass 55

Using the family class a_family = Family() # Initialization, a_family is an instance of the Family class 56

The __doc__ attribute >> Family.__doc__ 'Represent a family'  The __doc__ attribute of a class\class instance returns the comment which directly follows the class declaration.  Useful for letting the user know what the class is all about and how it works.  Try it with some of the classes you already know – list, tuple, etc. 57

Attributes\Data members  What are the family features that we want to include in our abstraction?  Depends on the problems we would like to solve. Let’s arbitrarily decide that what’s important to us is family members.  Could have been : generations, geographical location, number of cell phones etc. class Family: ''‘ Represent a family''' members = [] Implementation 58

Expanding the Family class  What do we want to know about our family? Let's arbitrarily decide we want to buy clothes for the whole family, so it important for us to know the favorite color of each family member and the total number of shoes required. class Family: ''‘ Represent a family''' members = [] Implementation 59

Design question  Total number of shoes – is this a feature of the family? Yes! class Family: ''‘ Represent a family''' members = [] Implementation 60

Defining a class function  We write down(and document) the signature of the method which returns the number of shoes, and remember to implement it later. class Family: ''‘ Represent a family''' members = [] def total_shoes(self): '‘’ Returns the total number of family shoes ''' pass Implementation 61

Members favorite colors  We want to know all the colors family members like.  Now that we are concerned for the first time with family members we should ask what do we know about them, is it color only?  Let’s arbitrarily assume we are also interested in members’ age and name. class Family: ''‘ Represent a family''' members = [] def total_shoes(self): '‘’ Returns the total number of family hoes ''' pass Implementation 62

Family members  Easy! Let’s define a  list for age,  list for name,  and list for color. such that the i'th element at each list represent the age, name and color of the i‘th family member. class Family: ''‘ Represent a family''' members = [] age = [] color = [] name = [] def total_shoes(self): '‘’ Returns the total number of family shoes ''' pass Implementation 63

Design question class Family: ''‘ Represent a family''' members = [] age = [] color = [] name = [] def total_shoes(self): '‘’ Returns the total number of family shoes ''' pass Implementation  Favorite color, age and name of each member – is this a feature of the family?  No, it is a feature of each member!  What if we suddenly wanted another features of family members (e.g. list of favorite music), should we add another list of these to Family? and then another? and then another? 64

OOP way of doing things  When we talk about the features we talk about favorite color of the family member, the member age and member name.  OOP guides us to encapsulate all this information in a Person class of its own. class Person: ''‘ Represent a person’'‘ pass class Family: ''‘ Represent a family''' members = [] def total_shoes(self): '‘’ Returns the total number of family shoes ''' pass Implementation 65

Person class  Let’s define a method which returns all the favorite colors of the family.  To do this we’ll add a Person.favorite_color() method to the Person class.  How is it implemented?  We can extract from Facebook account  Ask the Person  Give a random value  But we don’t worry about it now and just ‘count on it’ to be implemented later 66

67 class Family: ''‘ Represent a family''' members = [] def total_shoes(self): '‘’ Returns the total number of family shoes ''' pass def favrotie_colors(self): '‘’ Returns all the colors the family like’’’ colors = [] for member in self.members: colors.append(memeber.favorite_color()) class Person: ''‘ Represent a person’'‘ def favorite_color(self): ‘ ‘’ Return the person’s favorite color‘’’ pass Person and Family classes

OOP good practices  Let’s try to print an instance of the family class : >> a_family = Family() >> print(a_family)  Not very informative…  We can custom the printed information of an instance in printed by implementing the __str__ method. 68

__str__ method  The __str__ method should return a string  Very useful for debugging and getting instant insights into our objects’ states. >> a_family = Family() >> ron = Person('Ron') >> a_family.members.append(ron) >> print(a_family) Family members are : 69

70 class Family: ''' Represent a family''' members = [] def __str__(self): ''' String representation of the family''' family_as_string = 'Family members are: ' for member in self.members: family_as_string += str(member) return family_as_string class Person: name = '' def __init__(self,name): ''' Person’s constructor''' self.name = name Person and Family classes

What did we forget?  Implementing Person’s __str__ method as well! Implementation class Person: name = '' def __init__(self,name): ''' Person’s constructor''' self.name = name def __str__(self): ''' String representation of a person''' return "my name is " + self.name >> a_family = Family() >> ron = Person('Ron') >> a_family.members.append(ron) >> print(a_family) Family members are : my name is Ron 71