Lecture 20 Remote Procedure Call eXternal Data Representation CPE 401 / 601 Computer Network Systems slides are modified from Dave Hollinger

Distributed Program Design  Communication-Oriented Design  Design protocol first  Build programs that adhere to the protocol  Application-Oriented Design  Build application(s)  Divide programs up and add communication protocols RPC Overview 2 TypicalSocketsApproach RPC

RPC: Remote Procedure Call  Call a procedure (subroutine) that is running on another machine.  Issues:  identifying and accessing the remote procedure  parameters  return value RPC Overview 3

Remote Subroutine RPC Overview 4 blah, blah, blah bar = foo(a,b); blah, blah, blah bar = foo(a,b); blah, blah, blah int foo(int x, int y ) { if (x>100) return(y-2); else if (x>10) return(y-x); else return(x+y); } int foo(int x, int y ) { if (x>100) return(y-2); else if (x>10) return(y-x); else return(x+y); } Client Server protocol

Sun RPC  There are a number of popular RPC specifications.  Sun RPC (ONC RPC) is widely used.  NFS (Network File System) is RPC based.  Rich set of support tools. RPC Overview 5

Sun RPC Organization RPC Overview 6 Procedure 1 Procedure 2 Procedure 3 Shared Global Data Remote Program

Procedure Arguments  To reduce the complexity of the interface specification,  Sun RPC includes support for a single argument to a remote procedure Typically the single argument is a structure that contains a number of values. Newer versions can handle multiple args RPC Overview 7

Procedure Identification  Each procedure is identified by:  Hostname (IP Address)  Program identifier (32 bit integer)  Procedure identifier (32 bit integer) usually start at 1 and are numbered sequentially  Program version identifies for testing and migration typically start at 1 and are numbered sequentially RPC Overview 8

Program Identifiers  Each remote program has a unique ID.  Sun divided up the IDs: 0x x1fffffff 0x x3fffffff 0x x5fffffff 0x xffffffff RPC Overview 9 SunSysAdminTransientReserved

Iterative Server  Sun RPC specifies that at most one remote procedure within a program can be invoked at any given time.  If a 2 nd procedure is called, the call blocks until the 1 st procedure has completed.  Having an iterative server is useful for applications that may share data among procedures.  Eg: database to avoid insert/delete/modify collisions RPC Overview 10

Call Semantics  What does it mean to call a local procedure?  the procedure is run exactly one time.  What does it mean to call a remote procedure?  It might not mean "run exactly once"! RPC Overview 11

Remote Call Semantics  To act like a local procedure  exactly one invocation per call a reliable transport (TCP) is necessary.  Sun RPC does not support reliable call semantics !  "At Least Once" Semantics  if we get a response (a return value)  "Zero or More" Semantics  if we don't hear back from the remote subroutine RPC Overview 12

Remote Procedure deposit() deposit(DavesAccount,$100)  Always remember that you don't know how many times the remote procedure was run!  The net can duplicate the request (UDP). RPC Overview 13

Network Communication  The actual network communication is nothing new  it's just TCP/IP.  Many RPC implementations are built upon the sockets library.  the RPC library does all the work!  We are just using a different API,  the underlying stuff is the same! RPC Overview 14

Dynamic Port Mapping  Servers typically do not use well known protocol ports!  Clients know the Program ID  and host IP address  RPC includes support for looking up the port number of a remote program.  A port lookup service runs on each host  RPC servers register  "I'm program 17 and I'm looking for requests on port 1736” RPC Overview 15

The portmapper  Each system which will support RPC servers runs a port mapper server  provides a central registry for RPC services  Servers tell the port mapper what services they offer  Clients ask a remote port mapper for the port number corresponding to Remote Program ID  The portmapper is itself an RPC server!  is available on a well-known port (111) Netprog: RPC Overview 16

RPC Programming  RPC library is a collection of tools for automating the creation of clients and servers  clients are processes that call remote procedures  servers are processes that include procedure(s) that can be called by clients  RPC library  XDR routines  RPC run time library call rpc service register with portmapper dispatch incoming request to correct procedure  Program Generator RPC Overview 17

RPC Run-time Library  High- and Low-level functions that can be used by clients and servers  High-level functions provide simple access to RPC services  High-Level RPC library calls support UDP only must use lower-level RPC library functions to use TCP  High-Level library calls do not support any kind of authentication RPC Overview 18

Low-level RPC Library  Full control over all Inter-Process Communication options  TCP & UDP  Timeout values  Asynchronous procedure calls  Multi-tasking Servers  Broadcasting RPC Overview 19

RPCGEN  There is a tool for automating the creation of RPC clients and servers.  The program rpcgen does most of the work for you  The input to rpcgen is a protocol definition  in the form of a list of remote procedures and parameter types RPC Overview 20

RPCGEN RPC Overview 21 Input File rpcgen Client Stubs XDR filtersheader file Server skeleton C Source Code ProtocolDescription

rpcgen Output Files > rpcgen –C foo.x foo_clnt.c (client stubs) foo_svc.c (server main) foo_xdr.c (xdr filters) foo.h (shared header file) RPC Overview 22

Client Creation > gcc -o fooclient foomain.c foo_clnt.c foo_xdr.c -lnsl  foomain.c is the client main() (and possibly other functions) that call rpc services via the client stub functions in foo_clnt.c  The client stubs use the xdr functions RPC Overview 23

Server Creation gcc -o fooserver fooservices.c foo_svc.c foo_xdr.c –lrpcsvc -lnsl  fooservices.c contains the definitions of the actual remote procedures. RPC Overview 24

Example Protocol Definition struct twonums { int a; int b; }; program UIDPROG { version UIDVERS { int RGETUID(string ) = 1; string RGETLOGIN( int ) = 2; int RADD(twonums) = 3; } = 1; } = 0x ; RPC Overview 25

XDR: External Data Representation XDR 27 Process A XDR Encode/Decode Transport Process A XDR Encode/Decode Transport

XDR  Powerful paradigm for creation and transfer of complex data structures  XDR provides a service associated with the OSI Presentation Layer.  Common data representation  Library not part of the O.S.  Easy to port to new architectures  Independence from transport layer XDR 28

Data Conversion  Asymmetric Data Conversion  client always converts to the server’s data representation.  Symmetric Data Conversion  both client and server convert to/from some standard representation. XDR is Symmetric Data Conversion XDR 29

Implicit vs. Explicit Typing  Explicit typing  each piece of data includes information about the type  Implicit typing  the sender and receiver must agree on the order and type of all data XDR uses Implicit Typing XDR 30

XDR Data Types  boolean  char  short  int  long  float  double XDR 31  enumeration  structure  string  fixed length array (1-D)  variable length array (1-D)  union  opaque data

XDR Programming  XDR libraries are based on a stream paradigm  The process of converting local data to XDR also puts the data in the XDR stream  When extracting an item from a stream, conversion is done to the local representation  Streams can be attached to a file, pipe, socket or memory  Individual data items are added to (removed from) the stream one at a time XDR 32

Conversion Terminology  Converting from local representation to XDR representation is called Encoding.  Converting from XDR representation to local representation is called Decoding. XDR 33 Sender Receiver XDR ENCODE DECODE

XDR Stream Creation  There are a number of stream creation functions in the XDR library  xdrmem_create destination for encoding -or- source for decoding is a chunk of memory  xdrstdio_create destination for encoding -or- source for decoding is a file descriptor. XDR 34

XDR filters  The XDR library includes an extensive set of filters that perform encoding/decoding operations  Each XDR stream includes an attribute that determines the specific operation to be performed by a filter  encoding or decoding XDR 35

XDR Filters  The filter to encode/decode an integer is called xdr_int : bool_t xdr_int(XDR *xdrs, int *ip);  the return value indicates success.  xdrs is a pointer to an XDR stream  ip is a pointer to an integer. XDR 36

xdr_int() encode vs. decode XDR 37 int count; xdr_int(xstream,&count); int count; xdr_int(xstream,&count); Source Encodes Destination decodes

Complex Data Filters  The XDR library includes a set of filters designed to  translate complex C data structures to and from XDR representation  Many of these filters use simpler filters to convert individual components  xdr_array()  for encoding/decoding a variable length array  xdr_string()  string conversion filter has a maximum size XDR 38

xdr_array() (cont.) XDR 39 Source Array Dest. Array elproc()

XDR and RPC  RPC uses XDR, but the XDR filters are built for you automatically!  You define a complex data type  a code generator writes an XDR filter that can encode/decode your data type  Sometimes you need to understand XDR anyway... XDR 40

RPC Programming with rpcgen Issues:  Protocol Definition File  Client Programming Creating an "RPC Handle" to a server Calling client stubs  Server Programming Writing Remote Procedures RPC Programming 42

Protocol Definition File  Description of the interface of the remote procedures.  Almost function prototypes  Definition of any data structures used in the calls (argument types & return types)  Can also include shared C code (shared by client and server). Netprog: RPC Programming 43

XDR the language  XDR data types are not C data types!  There is a mapping from XDR types to C types that's most of what rpcgen does.  Most of the XDR syntax is just like C  Arrays, strings are different. RPC Programming 44

XDR Arrays  Fixed Length arrays look just like C code: int foo[100]  Variable Length arrays look like this: int foo<> or int foo RPC Programming 45 Implicit maximum size is

What gets sent on the network int x[n] RPC Programming 46 x0x0 x1x1 int y int y x n-1 x2x2... y0y0 y1y1 k k is actual array size k  m y2y2 ykyk

XDR String Type  Look like variable length arrays: string s  What is sent: length followed by sequence of ASCII chars: RPC Programming n s0s0 s1s1 s2s2 s3s3 S n-1 n is actual string length (sent as int)

Linked Lists! struct foo { int x; foo *next; }  The generated XDR filter uses xdr_pointer() to encode/decode the stuff pointed to by a pointer RPC Programming 48 rpcgen recognizes this as a linked list

Declaring The Program program SIMP_PROG { version SIMP_VERSION { type1 PROC1(operands1) = 1; type2 PROC2(operands2) = 2; } = 1; } = ; RPC Programming 49 KeywordsGenerated Symbolic Constants Used to generate stub and procedure names Color Code:

Procedure Numbers  Procedure #0 is created for you automatically.  Start at procedure #1!  Procedure #0 is a dummy procedure that can help debug things  sort of an RPC ping server RPC Programming 50

Procedure Names  Rpcgen converts to lower case and prepends underscore and version number: rtype PROCNAME(arg)  Client stub: rtype *proc_1(arg *, CLIENT *);  Server procedure: rtype *proc_1_svc(arg *, struct svc_req *); RPC Programming 51

Client Programming  You can find the porgram numbers currently used with "rpcinfo –p hostname“  Create RPC handle.  Establishes the address of the server.  RPC handle is passed to client stubs (generated by rpcgen).  Type is CLIENT * RPC Programming 52

clnt_create CLIENT *clnt_create( char *host, u_long prog, u_long vers, char *proto); RPC Programming 53 Hostname of server Program number Version number Can be "tcp" or "udp"

Calling Client Stubs  Remember:  Return value is a pointer to what you expect.  Argument is passed as a pointer.  If you are passing a string, you must pass a char**  When in doubt – look at the ".h" file generated by rpcgen RPC Programming 54

Server Procedures  Rpcgen writes most of the server.  You need to provide the actual remote procedures.  Look in the ".h" file for prototypes.  Run " rpcgen –C –Ss " to generate (empty) remote procedures! RPC Programming 55

Running rpcgen  Command line options vary from one OS to another.  Sun/BSD/Linux – you need to use "-C" to get ANSI C code!  Rpcgen can help write the files you need to write:  To generate sample server code: "-Ss"  To generate sample client code: "-Sc" RPC Programming 56

RPC without rpcgen  Can do asynchronous RPC  Callbacks  Single process is both client and server.  Write your own dispatcher  and provide concurrency  Can establish control over many network parameters  protocols, timeouts, resends, etc. Netprog: RPC Programming 57

rpcinfo  rpcinfo –p host prints a list of all registered programs on host.  rpcinfo –[ut] host program# makes a call to procedure #0 of the specified RPC program (RPC ping) RPC Programming 58 u : UDP t : TCP

Sample Code  Simple  integer add and subtract  ulookup  look up username and uid  varray  variable length array example  linkedlist  arg is linked list RPC Programming 59

Example simp  Standalone program simp.c  Takes 2 integers from command line and prints out the sum and difference.  Functions: int add( int x, int y ); int subtract( int x, int y ); RPC Programming 60

Splitting simp.c  Move the functions add() and subtract() to the server.  Change simp.c to be an RPC client  Calls stubs add_1(), subtract_1()  Create server that serves up 2 remote procedures  add_1_svc() and subtract_1_svc() RPC Programming 61

Protocol Definition: simp.x struct operands { int x; int y; }; program SIMP_PROG { version SIMP_VERSION { int ADD(operands) = 1; int SUB(operands) = 2; } = VERSION_NUMBER; } = ; RPC Programming 62

rpcgen –C simp.x RPC Programming 63 simp.x rpcgen simp_clnt.c simp_xdr.c simp.h simp_svc.c Client Stubs XDR filters header file Server skeleton

xdr_operands XDR filter bool_t xdr_operands( XDR *xdrs, operands *objp){ if (!xdr_int(xdrs, &objp->x)) return (FALSE); if (!xdr_int(xdrs, &objp->y)) return (FALSE); return (TRUE); } RPC Programming 64

simpclient.c  This was the main program – is now the client.  Reads 2 ints from the command line.  Creates a RPC handle.  Calls the remote add and subtract procedures.  Prints the results. RPC Programming 65

simpservice.c  The server main is in simp_svc.c.  simpservice.c is what we write – it holds the add and subtract procedures that simp_svc will call when it gets RPC requests.  The only thing you need to do is to match the name/parameters that simp_svc expects (check simp.h !). RPC Programming 66

Userlookup program  Provide access to passwd database via remote procedures:  getpwnam BYNAME  getpwuid BYNUM RPC Programming 67 Unix library functions Remote Procedures

userlookup.x typedef string username ; program ULKUP_PROG { version ULKUP_VERSION { int byname(username) = 1; username bynum(int) = 2; } = 1; } = ; RPC Programming 68

Problem with userlookup  It's hard to tell if there are errors:  What if there is no user with the name passed to byname() ?  What if the username passed is not a valid username? RPC Programming 69

Better userlookup.h %#define NOTFOUND 0 %#define FOUND 1 typedef string username ; struct uname_retval { int found; username name; }; RPC Programming 70

Better userlookup.h (cont.) struct uid_retval { int found; int uid; }; program ULKUP_PROG { version ULKUP_VERSION { uid_retval byname(username) = 1; uname_retval bynum(int) = 2; } = 1; } = ; RPC Programming 71

Varray example  Variable length array (determined at run time).  Remote procedure that returns the sum of the elements in an array of int. RPC Programming 72

varray.x typedef int iarray<>; program VADD_PROG { version VADD_VERSION { int VADD(iarray) = 1; } = 1; } = ; RPC Programming 73

iarray typedef struct { u_int iarray_len; int *iarray_val; } iarray; RPC Programming 74 typedef int iarray<>; rpcgen

vadd_1_svc() int * vadd_1_svc(iarray *argp, struct svc_req *rqstp) { static int result; int i; result=0; for (i=0;i iarray_len;i++) result += argp->iarray_val[i]; return (&result); } RPC Programming 75

linkedlist  Linked list of int.  Remote procedure computes sum of the integers in the list. RPC Programming 76

ll.x struct foo { int x; foo *next; }; program LL_PROG { version LL_VERSION { int SUM(foo) = 1; } = VERSION_NUMBER; } = ; RPC Programming 77