1. Interprocess Communication and Synchronization using System V IPC
Message Queues
Shared Memory
Semaphores

2. System V IPC
System V IPC was first introduced in SVR2, but is available now in most versions of unix
Message Queues represent linked lists of messages, which can be written to and read from
Shared memory allows two or more processes to share a region of memory, so that they may each read from and write to that memory region
Semaphores synchronize access to shared resources by providing synchronized access among multiple processes trying to access those critical resources.

3. Message Queues
A Message Queue is a linked list of message structures stored inside the kernel’s memory space and accessible by multiple processes
Synchronization is provided automatically by the kernel
New messages are added at the end of the queue
Each message structure has a long message type
Messages may be obtained from the queue either in a FIFO manner (default) or by requesting a specific type of message (based on message type)

4. Message Queues
Message Queues give you another way to send information from one task to another. There is overlap in Message Queue capabilities and Named Pipes. However, there are some substantial differences:
A Named pipe is byte oriented, which means that reading a Named Pipe from 2 different processes is problematic. However, Message Queues are packet oriented. Reading a Message Queue from 2 different processes is an excellent usage of Message Queues. In effect, you can have N identical worker tasks that pull request from a Message Queue.
Message Queues allow you to take packets out of order in some cases. Each message has a message type associated with it. A Message Queue reader can specify which type of message that it will read. Or it can say that it will read all messages in order.
It is quite possible to have any number of Msg Queue readers, or writers. In fact the same process can be both a writer and a reader.
One real bad characteristic: Msg Queues are based on a system buffer resource. It is possible for one process to let its messages pile up. This can result in all processes on a given system to be hung up because the system is out of resources. This is bad news when it happens.
There is a limit to the size of each packet and there is a limit to the total number of bytes that can show up in any given Message Queue.

5. Message Structs
Each message structure must start with a long message type: struct mymsg { long msg_type; char mytext[512]; /* rest of message */ int somethingelse; float dollarval; };

6. Message Queue Limits
Each message queue is limited in terms of both the maximum number of messages it can contain and the maximum number of bytes it may contain
New messages cannot be added if either limit is hit (new writes will normally block)
On linux, these limits are defined as (in /usr/include/linux/msg.h):
MSGMAX /*total number of messages */
MSBMNB /* max bytes in a queue */

7. Obtaining a Message Queue
#include <sys/types.h> #include <sys/ipc.h> #include <sys/msg.h> int msgget(key_t key, int msgflg);
The key parameter is either a non-zero identifier for the queue to be created or the value IPC_PRIVATE, which guarantees that a new queue is created.
The msgflg parameter is the read-write permissions for the queue OR’d with one of two flags:
IPC_CREAT will create a new queue or return an existing one
IPC_EXCL added will force the creation of a new queue, or return an error

8. Writing to a Message Queue
int msgsnd(int msqid, const void * msg_ptr, size_t msg_size, int msgflags);
msgqid is the id returned from the msgget call
msg_ptr is a pointer to the message structure
msg_size is the size of that structure
msgflags defines what happens when no message of the appropriate type is waiting, and can be set to the following:
IPC_NOWAIT (non-blocking, return –1 immediately if queue is empty)_

9. Reading from a Message Queue
int msgrcv(int msqid, const void * msg_ptr, size_t msg_size, long msgtype, int msgflags);
msgqid is the id returned from the msgget call
msg_ptr is a pointer to the message structure
msg_size is the size of that structure
msgtype is set to:
= 0 first message available in FIFO stack
> 0 first message on queue whose type equals type
< 0 first message on queue whose type is the lowest value less than or equal to the absolute value of msgtype
msgflags defines what happens when no message of the appropriate type is waiting, and can be set to the following:
IPC_NOWAIT (non-blocking, return –1 immediately if queue is empty)
example: ~mark/pub/51081/message.queues/potato.*.c

10. If the flags are zero, then the receiver will block if there is nothing to receive in the message queue. All other options are OR'ed in as masking flags:
IPC_NOWAIT indicates that this operation should return an error if there is no packet to receive. In this case we should get a ENOMSG error return code.
MSG_NOERROR indicates that the message should be truncated if necessary to fit the receiving buffer. The error E2BIG is returned when this option is used and the message cannot fit into the buffer.

11. Message Queue Control IPC_RMID destroy the queue specified by msqid
struct msqid_ds {
/* pointers to first and last messages on queue */
__time_t msg_stime; /* time of last msgsnd command */
__time_t msg_rtime; /* time of last msgrcv command */
...
unsigned short int __msg_cbytes; /* current number of bytes on queue */
msgqnum_t msg_qnum; /* number of messages currently on queue */
msglen_t msg_qbytes; /* max number of bytes allowed on queue */
/* pids of last msgsnd() and msgrcv() */ };
int msgctl(int msqid, int cmd, struct msqid_ds * buf);
cmd can be one of:
IPC_RMID destroy the queue specified by msqid
IPC_SET set the uid, gid, mode, and qbytes for the queue
IPC_STAT get the current msqid_ds struct for the queue
example: query.c

12. Shared Memory
Normally, the Unix kernel prohibits one process from accessing (reading, writing) memory belonging to another process
Sometimes, however, this restriction is inconvenient
At such times, System V IPC Shared Memory can be created to specifically allow on process to read and/or write to memory created by another process

13. Advantages of Shared Memory
Random Access
you can update a small piece in the middle of a data structure, rather than the entire structure
Efficiency
unlike message queues and pipes, which copy data from the process into memory within the kernel, shared memory is directly accessed
Shared memory resides in the user process memory, and is then shared among other processes

14. Disadvantages of Shared Memory
No automatic synchronization as in pipes or message queues (you have to provide any synchronization). Synchronize with semaphores or signals.
You must remember that pointers are only valid within a given process. Thus, pointer offsets cannot be assumed to be valid across inter-process boundaries. This complicates the sharing of linked lists or binary trees.

15. Creating Shared Memory
int shmget(key_t key, size_t size, int shmflg);
key is either a number or the constant IPC_PRIVATE (man ftok)
a shmid is returned
key_t ftok(const char * path, int id) will return a key value for IPC usage
size is the size of the shared memory data
shmflg is a rights mask (0666) OR’d with one of the following:
IPC_CREAT will create or attach
IPC_EXCL creates new or it will error if it exists

16. Attaching to Shared Memory
After obtaining a shmid from shmget(), you need to attach or map the shared memory segment to your data reference:
void * shmat(int shmid, void * shmaddr, int shmflg)
shmid is the id returned from shmget()
shmaddr is the shared memory segment address. Set this to NULL and let the system handle it.
shmflg is one of the following (usually 0):
SHM_RDONLY sets the segment readonly
SHM_RND sets page boundary access
SHM_SHARE_MMU set first available aligned address

17. Shared Memory Control
struct shmid_ds {
int shm_segsz; /* size of segment in bytes */
__time_t shm_atime; /* time of last shmat command */
__time_t shm_dtime; /* time of last shmdt command */
...
unsigned short int __shm_npages; /* size of segment in pages */
msgqnum_t shm_nattach; /* number of current attaches */
/* pids of creator and last shmop */ };
int shmctl(int shmid, int cmd, struct shmid_ds * buf);
cmd can be one of:
IPC_RMID destroy the memory specified by shmid
IPC_SET set the uid, gid, and mode of the shared mem
IPC_STAT get the current shmid_ds struct for the queue
example: ~mark/pub/51081/shared.memory/linux/*

18. Example 1a
/* Our first program is a consumer. After the headers the shared memory segment (the size of our shared memory structure) is created with a call to shmget, with the IPC_CREAT bit specified. */
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
/* A common header file to describe the shared memory we wish to pass about. */
#define TEXT_SZ 2048
struct shared_use_st
{ int written_by_you;
char some_text[TEXT_SZ]; };
int main() {
int running = 1;
void *shared_memory = (void *)0;
struct shared_use_st *shared_stuff;
int shmid;
srand((unsigned int)getpid());
shmid = shmget((key_t)1234, sizeof(struct shared_use_st), 0666 | IPC_CREAT);
if (shmid == -1) {
fprintf(stderr, "shmget failed\n");
exit(EXIT_FAILURE); }
/* We now make the shared memory accessible to the program. */
shared_memory = shmat(shmid, (void *)0, 0);
if (shared_memory == (void *)-1) {
fprintf(stderr, "shmat failed\n");
printf("Memory attached at %X\n", (int)shared_memory);

19. /* The next portion of the program assigns the shared_memory segment to shared_stuff,
which then prints out any text in written_by_you. The loop continues until end is found
in written_by_you. The call to sleep forces the consumer to sit in its critical section,
which makes the producer wait. */
shared_stuff = (struct shared_use_st *)shared_memory;
shared_stuff->written_by_you = 0;
while(running) {
if (shared_stuff->written_by_you) {
printf("You wrote: %s", shared_stuff->some_text);
sleep( rand() % 4 ); /* make the other process wait for us ! */
if (strncmp(shared_stuff->some_text, "end", 3) == 0) {
running = 0; }
/* Lastly, the shared memory is detached and then deleted. */
if (shmdt(shared_memory) == -1) {
fprintf(stderr, "shmdt failed\n");
exit(EXIT_FAILURE);
if (shmctl(shmid, IPC_RMID, 0) == -1) {
fprintf(stderr, "shmctl(IPC_RMID) failed\n");
exit(EXIT_SUCCESS);

20. example 1b
/* The second program is the producer and allows us to enter data for consumers.
It's very similar to shm1.c and looks like this. */
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include "shm_com.h"
/*shm_com.h
/* A common header file to describe the shared memory we wish to pass about. */#define TEXT_SZ 2048struct shared_use_st { int written_by_you; char some_text[TEXT_SZ];}; */
int main() {
int running = 1;
void *shared_memory = (void *)0;
struct shared_use_st *shared_stuff;
char buffer[BUFSIZ];
int shmid;
shmid = shmget((key_t)1234, sizeof(struct shared_use_st), 0666 | IPC_CREAT);
if (shmid == -1) {
fprintf(stderr, "shmget failed\n");
exit(EXIT_FAILURE); }
shared_memory = shmat(shmid, (void *)0, 0);
if (shared_memory == (void *)-1) {
fprintf(stderr, "shmat failed\n");

21. printf("Memory attached at %X\n", (int)shared_memory);
shared_stuff = (struct shared_use_st *)shared_memory;
while(running) {
while(shared_stuff->written_by_you == 1) {
sleep(1);
printf("waiting for client...\n"); }
printf("Enter some text: ");
fgets(buffer, BUFSIZ, stdin);
strncpy(shared_stuff->some_text, buffer, TEXT_SZ);
shared_stuff->written_by_you = 1;
if (strncmp(buffer, "end", 3) == 0) {
running = 0;
if (shmdt(shared_memory) == -1) {
fprintf(stderr, "shmdt failed\n");
exit(EXIT_FAILURE);
exit(EXIT_SUCCESS);

22. Semaphores Shared memory is not access controlled by the kernel
This means critical sections must be protected from potential conflicts with multiple writers
A critical section is a section of code that would prove problematic if two or more separate processes wrote to it simultaneously
Semaphores were invented to provide such locking protection on shared memory segments

23. System V Semaphores 1 == unlocked (available resource) 0 == locked
You can create an array of semaphores that can be controlled as a group
Semaphores may be binary (0/1), or counting
1 == unlocked (available resource)
0 == locked
Thus:
To unlock a semaphore, you INCREMENT it
To lock a semaphore, you DECREMENT it
Spinlocks are busy waiting semaphores that constantly poll to see if they may proceed

24. How Semaphores Work A critical section is defined
A semaphore is created to protect it
The first process into the critical section locks the critical section
All subsequent processes wait on the semaphore, and they are added to the semaphore’s “waiting list”
When the first process is out of the critical section, it signals the semaphore that it is done
The semaphore then wakes up one of its waiting processes to proceed into the critical section
All waiting and signaling are done atomically

25. How Semaphores “Don’t” Work: Deadlocks and Starvation
When two processes (p,q) are both waiting on a semaphore, and p cannot proceed until q signals, and q cannot continue until p signals. They are both asleep, waiting. Neither can signal the other, wake the other up. This is called a deadlock.
P1 locks a which succeeds, then waits on b
P2 locks b which succeeds, then waits on a
Indefinite blocking, or starvation, occurs when one process is constantly in a wait state, and is never signaled. This often occurs in LIFO situations.
example: ~mark/pub/51081/semaphores/linux/shmem.matrix.multiplier2.c

26. semun.h #if defined(__GNU_LIBRARY__) && !defined(_SEM_SEMUN_UNDEFINED)
/* union semun is defined by including <sys/sem.h> */
#else
/* according to X/OPEN we have to define it ourselves */
union semun {
int val; /* value for SETVAL */
struct semid_ds *buf; /* buffer for IPC_STAT, IPC_SET */
unsigned short int *array; /* array for GETALL, SETALL */
struct seminfo *__buf; /* buffer for IPC_INFO */ };
#endif

27. #include <unistd.h> #include <stdlib.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/sem.h>
#include "semun.h"
static int set_semvalue(void);
static void del_semvalue(void);
static int semaphore_p(void);
static int semaphore_v(void);
static int sem_id;
int main(int argc, char *argv[]) {
int i;
int pause_time;
char op_char = 'O';
srand((unsigned int)getpid());
sem_id = semget((key_t)1234, 1, 0666 | IPC_CREAT);
if (argc > 1) {
if (!set_semvalue()) {
fprintf(stderr, "Failed to initialize semaphore\n");
exit(EXIT_FAILURE); }
op_char = 'X';
sleep(2);

28. /* Then we have a loop which enters and leaves the critical section ten times.
There, we first make a call to semaphore_p which sets the semaphore to wait, as
this program is about to enter the critical section. */
for(i = 0; i < 10; i++) {
if (!semaphore_p()) exit(EXIT_FAILURE);
printf("%c", op_char);fflush(stdout);
pause_time = rand() % 3;
sleep(pause_time);
/* After the critical section, we call semaphore_v, setting the semaphore available,
before going through the for loop again after a random wait. After the loop, the call
to del_semvalue is made to clean up the code. */
if (!semaphore_v()) exit(EXIT_FAILURE);
pause_time = rand() % 2; }
printf("\n%d - finished\n", getpid());
if (argc > 1) {
sleep(10);
del_semvalue();
exit(EXIT_SUCCESS);

29. /* The function set_semvalue initializes the semaphore using the SETVAL command in a
semctl call. We need to do this before we can use the semaphore. */
static int set_semvalue(void) {
union semun sem_union;
sem_union.val = 1;
if (semctl(sem_id, 0, SETVAL, sem_union) == -1) return(0);
return(1); }
/* The del_semvalue function has almost the same form, except the call to semctl uses
the command IPC_RMID to remove the semaphore's ID. */
static void del_semvalue(void)
if (semctl(sem_id, 0, IPC_RMID, sem_union) == -1)
fprintf(stderr, "Failed to delete semaphore\n");
/* semaphore_p changes the semaphore by -1 (waiting). */
static int semaphore_p(void)
struct sembuf sem_b;
sem_b.sem_num = 0;
sem_b.sem_op = -1; /* P() */
sem_b.sem_flg = SEM_UNDO;
if (semop(sem_id, &sem_b, 1) == -1) {
fprintf(stderr, "semaphore_p failed\n");
return(0);

30. /* semaphore_v is similar except for setting the sem_op part of the sembuf structure to 1,
so that the semaphore becomes available. */
static int semaphore_v(void) {
struct sembuf sem_b;
sem_b.sem_num = 0;
sem_b.sem_op = 1; /* V() */
sem_b.sem_flg = SEM_UNDO;
if (semop(sem_id, &sem_b, 1) == -1) {
fprintf(stderr, "semaphore_v failed\n");
return(0); }
return(1);
% ./3&
%./3 1