1. Distributed File Systems: RPC, NFS, and AFS

2. Announements Homework 6 available later tonight
Due next Tuesday, December 2nd
See me after class to pick up prelim
Upcoming Agenda
No class on Thursday—Happy Thanksgiving!
Next week last week of classes—December 2nd and 4th
Final—Thursday, December 18th at 2pm
Room 131 Warren Hall
Length is 2hrs

3. Goals for Today Distributed file systems (DFS)
Network file system (NFS)
Remote Procedure Calls (RPC)
Andrew file system (AFS)

4. Distributed File Systems (DFS)

5. Distributed File Systems
Goal: view a distributed system as a file system
Storage is distributed
Web tries to make world a collection of hyperlinked documents
Issues not common to usual file systems
Naming transparency
Load balancing
Scalability
Location and network transparency
Fault tolerance
We will look at some of these today

6. Transfer Model Upload/download Model: Remote Access Model:
Client downloads file, works on it, and writes it back on server
Simple and good performance
Remote Access Model:
File only on server; client sends commands to get work done

7. Naming transparency
Naming is a mapping from logical to physical objects
Ideally client interface should be transparent
Not distinguish between remote and local files
/machine/path or mounting remote FS in local hierarchy are not transparent
A transparent DFS hides the location of files in system
2 forms of transparency:
Location transparency: path gives no hint of file location
/server1/dir1/dir2/x tells x is on server1, but not where server1 is
Location independence: move files without changing names
Separate naming hierarchy from storage devices hierarchy

8. File Sharing Semantics
Sequential consistency: reads see previous writes
Ordering on all system calls seen by all processors
Maintained in single processor systems
Can be achieved in DFS with one file server and no caching

9. Caching Keep repeatedly accessed blocks in cache How it works:
Improves performance of further accesses
How it works:
If needed block not in cache, it is fetched and cached
Accesses performed on local copy
One master file copy on server, other copies distributed in DFS
Cache consistency problem: how to keep cached copy consistent with master file copy
Where to cache?
Disk: Pros: more reliable, data present locally on recovery
Memory: Pros: diskless workstations, quicker data access,
Servers maintain cache in memory

10. File Sharing Semantics
Other approaches:
Write through caches:
immediately propagate changes in cache files to server
Reliable but poor performance
Delayed write:
Writes are not propagated immediately, probably on file close
Session semantics (AFS): write file back on close
Alternative (NFS): scan cache periodically and flush modified blocks
Better performance but poor reliability
File Locking:
The upload/download model locks a downloaded file
Other processes wait for file lock to be released

11. Network File System (NFS)

12. Network File System (NFS)
Developed by Sun Microsystems in 1984
Used to join FSes on multiple computers as one logical whole
Used commonly today with UNIX systems
Assumptions
Allows arbitrary collection of users to share a file system
Clients and servers might be on different LANs
Machines can be clients and servers at the same time
Architecture:
A server exports one or more of its directories to remote clients
Clients access exported directories by mounting them
The contents are then accessed as if they were local

13. Example

14. NFS Mount Protocol
Client sends path name to server with request to mount
Not required to specify where to mount
If path is legal and exported, server returns file handle
Contains FS type, disk, i-node number of directory, security info
Subsequent accesses from client use file handle
Mount can be either at boot or automount
Using automount, directories are not mounted during boot
OS sends a message to servers on first remote file access
Automount is helpful since remote dir might not be used at all
Mount only affects the client view!

15. NFS Protocol
Supports directory and file access via remote procedure calls (RPCs)
All UNIX system calls supported other than open & close
Open and close are intentionally not supported
For a read, client sends lookup message to server
Server looks up file and returns handle
Unlike open, lookup does not copy info in internal system tables
Subsequently, read contains file handle, offset and num bytes
Each message is self-contained
Pros: server is stateless, i.e. no state about open files
Cons: Locking is difficult, no concurrency control

16. NFS Implementation Three main layers: System call layer:
Handles calls like open, read and close
Virtual File System Layer:
Maintains table with one entry (v-node) for each open file
v-nodes indicate if file is local or remote
If remote it has enough info to access them
For local files, FS and i-node are recorded
NFS Service Layer:
This lowest layer implements the NFS protocol

17. NFS Layer Structure

18. How NFS works? Mount: Open:
Sys ad calls mount program with remote dir, local dir
Mount program parses for name of NFS server
Contacts server asking for file handle for remote dir
If directory exists for remote mounting, server returns handle
Client kernel constructs v-node for remote dir
Asks NFS client code to construct r-node for file handle
Open:
Kernel realizes that file is on remotely mounted directory
Finds r-node in v-node for the directory
NFS client code then opens file, enters r-node for file in VFS, and returns file descriptor for remote node

19. Cache coherency Clients cache file attributes and data Solutions:
If two clients cache the same data, cache coherency is lost
Solutions:
Each cache block has a timer (3 sec for data, 30 sec for dir)
Entry is discarded when timer expires
On open of cached file, its last modify time on server is checked
If cached copy is old, it is discarded
Every 30 sec, cache time expires
All dirty blocks are written back to the server

20. Remote Procedure Call (RPC)

21. Procedure Call
More natural way is to communicate using procedure calls:
every language supports it
semantics are well defined and understood
natural for programmers to use
Basic idea: define server as a module that exports a set of procedures callable by client programs.
To use the server, the client just does a procedure call, as if it were linked with the server
call
Client
Server
return

22. (Remote) Procedure Call
So, we would like to use procedure call as a model for distributed communication.
Lots of issues:
how do we make this invisible to the programmer?
what are the semantics of parameter passing?
how is binding done (locating the server)?
how do we support heterogeneity (OS, arch., language)
etc.

23. Remote Procedure Call
The basic model for Remote Procedure Call (RPC) was described by Birrell and Nelson in 1980, based on work done at Xerox PARC.
Goal to make RPC as much like local PC as possible.
Used computer/language support.
There are 3 components on each side:
a user program (client or server)
a set of stub procedures
RPC runtime support

24. RPC Basic process for building a server:
Server program defines the server’s interface using an interface definition language (IDL)
The IDL specifies the names, parameters, and types for all client-callable server procedures
A stub compiler reads the IDL and produces two stub procedures for each server procedure: a client-side stub and a server-side stub
The server writer writes the server and links it with the server-side stubs; the client writes her program and links it with the client-side stubs.
The stubs are responsible for managing all details of the remote communication between client and server.

25. RPC Stubs
Client-side stub is a procedure that looks to the client as if it were a callable server procedure.
Server-side stub looks like a calling client to the server
The client program thinks it is calling the server;
in fact, it’s calling the client stub.
The server program thinks it’s called by the client;
in fact, it’s called by the server stub.
The stubs send messages to each other to make RPC happen.

26. RPC Call Structure client program client makes local call to
stub proc.
server is
called by
its stub
proc foo(a,b)
begin foo...
end foo
server
program
call foo(x,y)
call foo
call foo
stub unpacks
params and
makes call
client
stub
proc foo(a,b)
stub builds msg
packet, inserts
params
call foo(x,y)
server
stub
send msg
msg received
runtime sends
msg to remote
node
runtime
receives msg
and calls stub
RPC
runtime
RPC
runtime
Call

27. RPC Return Structure client program proc foo(a,b) begin foo... end foo
server proc
returns
server
program
call foo(x,y)
client continues
return
return
stub builds
result msg
with output
args
client
stub
proc foo(a,b)
stub unpacks
msg, returns
to caller
call foo(x,y)
server
stub
msg received
send msg
runtime
responds
to original
msg
runtime
receives msg,
calls stub
RPC
runtime
RPC
runtime
return

28. RPC Information Flow Client Stub marshal args call send Client
(caller)
RPC
Runtime
return
receive
unmarshal
ret vals
mbox2
Network
Network
Machine A
Machine B
marshal
return values
mbox1
return
Server
(callee)
Server
Stub
unmarshal
args
send
RPC
Runtime
call
receive

29. RPC Binding Binding is the process of connecting the client and server
The server, when it starts up, exports its interface,
identifying itself to a network name server and
telling the local runtime its dispatcher address.
The client, before issuing any calls, imports the server,
which causes the RPC runtime to lookup the server through the name service and
contact the requested server to setup a connection.
The import and export are explicit calls in the code.

30. RPC Marshalling
Marshalling is packing of procedure params into message packet.
RPC stubs call type-specific procedures to marshall (or unmarshall) all of the parameters to the call.
On client side, client stub marshalls parameters into call packet;
On the server side the server stub unmarshalls the parameters to call the server’s procedure.
On return, server stub marshalls return parameters into return packet;
Client stub unmarshalls return params and returns to the client.

31. Problems with RPC Non-Atomic failures Performance
Different failure modes in distributed system than on a single machine
Consider many different types of failures
User-level bug causes address space to crash
Machine failure, kernel bug causes all processes on same machine to fail
Some machine is compromised by malicious party
Before RPC: whole system would crash/die
After RPC: One machine crashes/compromised while others keep working
Can easily result in inconsistent view of the world
Did my cached data get written back or not?
Did server do what I requested or not?
Answer? Distributed transactions/Byzantine Commit
Performance
Cost of Procedure call « same-machine RPC « network RPC
Means programmers must be aware that RPC is not free
Caching can help, but may make failure handling complex

32. Cross-Domain Comm./Location Transparency
How do address spaces communicate with one another?
Shared Memory with Semaphores, monitors, etc…
File System
Pipes (1-way communication)
“Remote” procedure call (2-way communication)
RPC’s can be used to communicate between address spaces on different machines or the same machine
Services can be run wherever it’s most appropriate
Access to local and remote services looks the same
Examples of modern RPC systems:
CORBA (Common Object Request Broker Architecture)
DCOM (Distributed COM)
RMI (Java Remote Method Invocation)

33. Microkernel operating systems
Example: split kernel into application-level servers.
File system looks remote, even though on same machine
Why split the OS into separate domains?
Fault isolation: bugs are more isolated (build a firewall)
Enforces modularity: allows incremental upgrades of pieces of software (client or server)
Location transparent: service can be local or remote
For example in the X windowing system: Each X client can be on a separate machine from X server; Neither has to run on the machine with the frame buffer.
App
file system
Windowing
Networking VM
Threads
Monolithic Structure
File
sys
windows
RPC
address
spaces
threads
Microkernel Structure

34. Andrew File System (AFS)

35. Andrew File System (AFS)
Named after Andrew Carnegie and Andrew Mellon
Transarc Corp. and then IBM took development of AFS
In 2000 IBM made OpenAFS available as open source
Features:
Uniform name space
Location independent file sharing
Client side caching with cache consistency
Secure authentication via Kerberos
Server-side caching in form of replicas
High availability through automatic switchover of replicas
Scalability to span 5000 workstations

36. AFS Overview Based on the upload/download model
Clients download and cache files
Server keeps track of clients that cache the file
Clients upload files at end of session
Whole file caching is central idea behind AFS
Later amended to block operations
Simple, effective
AFS servers are stateful
Keep track of clients that have cached files
Recall files that have been modified

37. AFS Details Has dedicated server machines
Clients have partitioned name space:
Local name space and shared name space
Cluster of dedicated servers (Vice) present shared name space
Clients run Virtue protocol to communicate with Vice
Clients and servers are grouped into clusters
Clusters connected through the WAN
Other issues:
Scalability, client mobility, security, protection, heterogeneity

38. AFS: Shared Name Space AFS’s storage is arranged in volumes
Usually associated with files of a particular client
AFS dir entry maps vice files/dirs to a 96-bit fid
Volume number
Vnode number: index into i-node array of a volume
Uniquifier: allows reuse of vnode numbers
Fids are location transparent
File movements do not invalidate fids
Location information kept in volume-location database
Volumes migrated to balance available disk space, utilization
Volume movement is atomic; operation aborted on server crash

39. AFS: Operations and Consistency
AFS caches entire files from servers
Client interacts with servers only during open and close
OS on client intercepts calls, and passes it to Venus
Venus is a client process that caches files from servers
Venus contacts Vice only on open and close
Does not contact if file is already in the cache, and not invalidated
Reads and writes bypass Venus
Works due to callback:
Server updates state to record caching
Server notifies client before allowing another client to modify
Clients lose their callback when someone writes the file
Venus caches dirs and symbolic links for path translation

40. AFS Implementation Client cache is a local directory on UNIX FS
Venus and server processes access file directly by UNIX i-node
Venus has 2 caches, one for status & one for data
Uses LRU to keep them bounded in size

41. Summary RPC NFS: AFS: Call procedure on remote machine
Provides same interface as procedure
Automatic packing and unpacking of arguments without user programming (in stub)
NFS:
Simple distributed file system protocol. No open/close
Stateless server
Has problems with cache consistency, locking protocol
AFS:
More complicated distributed file system protocol
Stateful server
session semantics: consistency on close

42. Prelim II Prelims graded Re-grade policy
Mean 73 (Median 76), Stddev 13.3, High 98 out of 100!
Good job!
Re-grade policy
Submit written re-grade request to Nazrul.
Entire prelim will be re-graded.
We were generous the first time…
If still unhappy, submit another re-grade request.
Nazrul will re-grade herself
If still unhappy, submit a third re-grade request.
I will re-grade. Final grade is law.

43. Grade distribution

44. Question #3 Hardlinks Softlinks Need a count in inode/fileheader
Remove decrements count and removes file if count is 0
New syscall: Link(char *src, char *dst)
Softlinks
Need to file type indicated in inode/fileheader
Also, need path to target file
Open needs to change to perform recursive lookup
New syscall: SymLink(char *src, char *dst)

45. Question #4 Concurrent writers
RAID 0, 1+0, 5, 6
Not 1 or 4 because cannot perform independent writes
Concurrent readers (but not concurrent writers)
RAID 1 and 4
2*k-1 disks and want concurrent readers
Unavailable: RAID 1 — requires 2*k disks
Undesirable: RAID 4, 5, and 6 — requires complex controllers

46. Happy Thanksgiving!!!