Module 2 Programming with Processes

Processes Process -- a program in execution –May share code segments –Typically do not share data, stack, heap OS data structures (file handles, coroutine, context block) kept in struct named User One User struct per process Process -- a program in execution –May share code segments –Typically do not share data, stack, heap OS data structures (file handles, coroutine, context block) kept in struct named User One User struct per process

Login Authenticates via name/password Name converted to number(s) –User Id –Group Id : used for sharing Macintosh-2:~ shawne$ ls -l total 48 drwx shawne staff 3026 Mar 1 14:03 Desktop drwx shawne staff 238 Feb 13 15:50 Documents drwx shawne staff 170 Feb 19 11:07 Downloads Authenticates via name/password Name converted to number(s) –User Id –Group Id : used for sharing Macintosh-2:~ shawne$ ls -l total 48 drwx shawne staff 3026 Mar 1 14:03 Desktop drwx shawne staff 238 Feb 13 15:50 Documents drwx shawne staff 170 Feb 19 11:07 Downloads User id group id

Process Creation and Destruction int execle (const char *filepath, const char *arg,..., (char *)0, char *const envp[] ); /* replaces caller's memory image with "program" */ void exit (int status); /* terminates execution; returns status to parent */ pid_t wait (int *stat_loc); pid_t waitpid (pid_t pid, int *stat_loc, int options); /* blocks caller until a child terminates. pid_t fork (void); /* duplicates the memory image of the caller */ int execle (const char *filepath, const char *arg,..., (char *)0, char *const envp[] ); /* replaces caller's memory image with "program" */ void exit (int status); /* terminates execution; returns status to parent */ pid_t wait (int *stat_loc); pid_t waitpid (pid_t pid, int *stat_loc, int options); /* blocks caller until a child terminates. pid_t fork (void); /* duplicates the memory image of the caller */

#include int main() { int pid, out; char *envp[]={"ME=BOB",0}; pid=fork(); printf("I am %d\n",pid); if (pid!=0) { wait(&out); printf("child returned %d\n",out); execle("/bin/echo","dog","cat",0,envp); } exit(55); } #include int main() { int pid, out; char *envp[]={"ME=BOB",0}; pid=fork(); printf("I am %d\n",pid); if (pid!=0) { wait(&out); printf("child returned %d\n",out); execle("/bin/echo","dog","cat",0,envp); } exit(55); }

C program uses OS APIs to write commands Shell programs Window programs Browser Browser programs C program uses OS APIs to write commands Shell programs Window programs Browser Browser programs

UNIX Shell

Boot-up to Command

Remote Execution ssh command Automatic login is desirable –The scheme is based on public-key cryptography, using cryptosystems where encryption and decryption are done using separate keys, and it is infeasible to derive the decryption key from the encryption key. –The idea is that each user creates a public/private key pair for authentication purposes. The server knows the public key, and only the user knows the private key. –Private key encrypts login info; public key decrypts Automatic login is desirable –The scheme is based on public-key cryptography, using cryptosystems where encryption and decryption are done using separate keys, and it is infeasible to derive the decryption key from the encryption key. –The idea is that each user creates a public/private key pair for authentication purposes. The server knows the public key, and only the user knows the private key. –Private key encrypts login info; public key decrypts

Data Parallelism Why: speed-up How (greatly simplified): –split work into independent similar pieces –execute pieces in parallel –collect results Simple example –Count number of identifiers in a set of files Practical Parallel and Concurrent Programming DRAFT: comments to 106/16/2010

Count ids in several files./test foo.txt moo.txt boo.txt #include int main(int argc, char *argv[]) { int count=0, i; for (i=1; i<argc; i++) { if (fork()) continue; execl("./cnt","./cnt",argv[i],0); } //for for (i=1; i<argc; i++) { int cnt; wait(&cnt); count+=WEXITSTATUS(cnt); } printf("count = %d ¥ n", count); exit(0); } #include int main(int argc, char *argv[]) { int count=0, i; for (i=1; i<argc; i++) { if (fork()) continue; execl("./cnt","./cnt",argv[i],0); } //for for (i=1; i<argc; i++) { int cnt; wait(&cnt); count+=WEXITSTATUS(cnt); } printf("count = %d ¥ n", count); exit(0); }

Shell procedure to count ids declare -i x=0 for i in $* ; do x=x+`./cnt $i` ; echo $x ; Done Command Line./foo.sh /Users/bobcook/Desktop/*.txt Output declare -i x=0 for i in $* ; do x=x+`./cnt $i` ; echo $x ; Done Command Line./foo.sh /Users/bobcook/Desktop/*.txt Output

Amdahl’s hypothesis Assume that sorting takes 70% of the execution time of a sequential program You replace the sorting algorithm with one that scales perfectly on multi-core hardware How many cores do you need to get a 4x speed-up of the program? 6/16/2010 Practical Parallel and Concurrent Programming DRAFT: comments to 14

6/16/2010 Practical Parallel and Concurrent Programming DRAFT: comments to 15 = the parallel portion of execution = the sequential portion of execution = number of cores used Amdahl’s Formula

6/16/2010 Practical Parallel and Concurrent Programming DRAFT: comments to 16 Speedup achieved with perfect scaling Amdahl’s law limit, just 1.11x f = 10%

6/16/2010 Practical Parallel and Concurrent Programming DRAFT: comments to 17 Desired 4x speedup Speedup achieved (perfect scaling on 70%) Limit as c→∞ = 1/(1-f) = 3.33 f = 70%

6/16/2010 Practical Parallel and Concurrent Programming DRAFT: comments to 18 f = 98%

Lesson Speedup is limited by sequential code Even a small percentage of sequential code can greatly limit potential speedup 6/16/2010 Practical Parallel and Concurrent Programming DRAFT: comments to 19

Reasons for IPC Information sharing Program speedup Fault tolerance Resource limitations per node Heterogeneity Modularity Convenience Privilege separation -- different levels of protection Inter-Process Communication (IPC)

Pipes Message queues Shared memory Shared files Signals Sockets (TCP/IP protocols) Message passing (MPI, discussed later) Semaphores (discussed later) Remote variable access (discussed later) Remote procedure call (discussed later) Remote process invocation (ssh commands) Common IPC APIs

#include int main() { //pipe input to sort command FILE *write; char *str[]={"hi", "by", "so", "mo"}; int i; write = popen("sort", "w"); if (write !=NULL) { for (i=0; i<4; i++) fprintf(write, "%s\n", str[i]); pclose(write); exit(EXIT_SUCCESS); } exit(EXIT_FAILURE); } OUTPUT by hi mo so Pipe Command Example (invoke commands from C programs)

#include int pipe(int fildes[2] /* out parameter*/); fildes[0] - read end of pipe fildes[1] - write end of pipe Be aware that write() can write less than you request and read() can read less. File handles are inherited by fork()/exec() processes. Pipe System Call API

FIFO File Object System API #include int mkfifo(const char *path, mode_t mode); Like a file, a FIFO can be opened for reading or writing or both intermittently.

Message Queue API

Message Queue -record oriented #include /* For O_* constants */ #include /* For mode constants */ #include struct mq_attr { long mq_flags; /* Flags: 0|O_NONBLOCK */ long mq_maxmsg; /* Max. # of msgs on Q*/ long mq_msgsize; /* Max. msg size (bytes) */ long mq_curmsgs; /* # msgs currently in Q*/ }; mqd_t mq_open(const char name[], int oflag); //to open only mqd_t mq_open(const char name[], int oflag, mode_t access_mode, struct mq_attr *attr); int mq_close(mqd_t mqdes); int mq_unlink(const char name[]); int mq_send(mqd_t mqdes, const char msg_ptr[], size_t msg_len, unsigned msg_priority); ssize_t mq_receive(mqd_t mqdes, char msg_ptr[], size_t msg_len, unsigned *msg_priority);

Processing Multiple Files in Parallel One thread per Chunk

Shared Memory Among Multiple Processes

//returns shmid for new segment of size bytes int shmget(key_t key, size_t size, int mode_t); //control ops such as stat, unlink, lock, unlock int shmctl(int shmid, int cmd, struct shmid_ds *buf); //attach segment to process at shmaddr void *shmat(int shmid, const void *shmaddr, int shmflag); //detach segment at shmaddr from this process int shmdt(const void *shmaddr); Shared Memory API

Memory-Mapped Files for Increased Efficiency

#include void *mmap(void *start_addr, size_t length, int protection, int flags, int filedescriptor, off_t offset); int munmap(void *start_addr, size_t length); Memory-Mapped File API