1. System Config IPC
Iris Zhu

2. Agenda IPC Overview System V IPC Posix IPC
Tools for performance analysis
System configuration
Hands On Lab

3. Process overview Concept Execution of program
Fundamental abstraction of unix system
Basic unit
Process
Virtual memory environment
Resource for Process
Memory
Open file list
Thread

4. Conceptual view of a Process

5. Process virtual address space

6. IPC history Message passing Data synchronization Pipe,named pipe(FIFO)
System V message queue
Posix message queue(1003.1b-1993)
RPC
Data synchronization
System V semaphore,System V shared memory
Posix semaphore
PIPE-ealiest IPC
System V message queue /semaphore 20century 80s
other types of thread lock/lock/mutex/condition variable

7. Inter Process Communication
Function
Sharing of data
shared memory
Exchange Information and data
Pipe,FIFO,message queues
Synchronization of access to shared resources semaphore,mutex,lock,
Remote procedure call
Sun RPC,Solaris Door

8. Unix Processes share data
Posix message queue/semaphore shared memory at least kernel persist ,dependent on implementation.

9. IPC object persistence
Process-persistent
Pipe
FIFO
Kernel persistent
System V Shared memory
System V Message queue
System V Semaphore
Filesystem persistent

10. Pipe
Special type of file that do not hold data but can be opened by 2 different processed so that data can be passed between them.
API
Int pipe(int fd[2]);
Fd[0] – read Fd[1] – write //For example:
Feature
Half duplex.
Atomic operation.
Usable between processes with the same parent.
Pipe rarely used in only 1 process
2 or more pipes used between parent/child process to exchange data.
Mostly used in shell
the reader will block if there were no data in the pipe

11. Named pipe - FIFO
Provide a bidirectional communication path between processed on the same system.
API
Int mkfifo(const char *pathname,mode_t mode);
fopen/fclose //For example:
Features
Half duplex
Automatic block
SIGPIPE is generated if a process write to a pipe and read is terminated
Atomic operation
Can be used between different processes.

12. System limitation -<limits.h>
OPEN_MAX – by sysconf()
PIPE_BUF pathconf() or fpathconf() in Posix standard

13. Other commands to get system variables
limit(csh)/unlimit(sh/bash/ksh)
Sysdef
lists all hardware devices, as well as pseudo devices, system devices, loadable modules, and the values of selected kernel tunable parameters.
For example,sysdef(1M) //For exercise:
Getconf
get configuration values
For example,getconf(1)
getconf OPEN_MAX //For exercise:

14. Doors
Provide a facility for processes to issue procedure calls to functions in other processes running on the same system. //Sun Solaris OS
Overview
libdoor.so
lightweight RPC 16bytes sent only for a remote call

15. Agenda IPC Overview System V IPC* Posix IPC
Tools for performance analysis
System configuration
Hands On Lab
Why smf
what can smf do

16. System V IPC Overview Types System V shared memory
System V message queues
System V semaphore
API

17. System V IPC overview key_t and ftok #include <sys/ipc.h>
key_t ftok (const char * pathname,int id);
Create and open IPC
key=id(8bit)+st_dev(12bit)+st_ino(12bit)
Same id+same path = same key
IPC_PRIVATE will create an unique key and return the unique Id

18. Data structure in kernel for IPC
ipc_perm data structure in /usr/include/sys/ipc.h

19. System V shared memory Concept Feature Kernel resource consumption
sharing the same physical(RAM) memory
pages by multi processes
Feature
Extremely efficient(*)
Dynamically loaded when required,eg.modload
Unload when system reboot or by command modunload
Kernel resource consumption
Shmid
actual shared RAM pages
data structure about the shared segment

20. System V shared memory - API
Header file and system calls
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
int shmget(key_t key, size_t size, int shmflg);
void *shmat(int shmid, const void *shmaddr, int shmflg);
int shmctl(int shmid, int cmd, struct shmid_ds *buf);
int shmdt(char *shmaddr);
Cmd – IPC_STAT
IPC_RMID
IPC_SET
SHM_LOCK/UNLOCK

21. Data structure Struct shmid_ds{
struct ipc_perm shm_perm /* operation permission structure */
size_t shm_segsz /* size of segment in bytes */
pid_t shm_lpid /* process ID of last shared memory operation */
pid_t shm_cpid /* process ID of creator */
shmatt_t shm_nattch /* number of current attaches */
time_t shm_atime /* time of last shmat() */
time_t shm_dtime /* time of last shmdt() */
time_t shm_ctime /* time of last change by shmctl() */ }

22. Example,process with shared memory

23. System overheads with shared memory
User level
Quantity of created shared memory
Size
Kernel level
System memory
Quantity of identifiers
Max size for shared memory segment
Translation table of shared memory
Swap space
Shared memory need more swap.
ISM locked in kernel memory and not able to be paged,thus others need to be swapped out.

24. System tunable parameters
System profile(before Solaris 10)
/etc/system

25. Caveat Static mechanism
Values specified are read and applied when system boot.
Any changes are not applied until the system reboot
Values specified in /etc/system are global and affect all processes on the system
The obsolete tunable settings are ignored from Solaris 10.

26. Resource control available
Dynamically resource control
Prctl(1)
get or set the resource controls of running processes, tasks, and projects
rctladm(1M)
display or modify global state of system resource controls
Man resource_controls(5)
Setrctl(2)
API,set or get resource control values
Man rctlblk_get_local_action(3C)

27. Dynamic resource control
Settings in Solaris 10

28. Obsolete tunable parameter

29. Example

30. Optimization of shared memory - ISM
Concept
ISM - Intimate shared memory
Sharing of the translation tables involved in virtual-to-physical address translation
No need to share the actual physical memory pages
Contrast
Non-ISM
per-processes mapping for shared memory pages

31. Background introduction – memory basic
MMU
- Virtual memory management unit
Management and translation of the virtual view of memory(address space) to physical memory.
HAT
- Hardware address translation layer
Management mapping of virtual to physical memory
TLB
- Translation lookaside buffer
Hardware cache of recent address translation information
MMU – V to P change table
HAT – get the physical address in RAM
TLB – HW cache for address transition.

32. Hardware address translation

33. Sharing the memory translation table
ISM
Sharing the memory translation table

34. Non-ISM

35. When attach shared memory by Shmat() ,use parameter SHM_SHARE_MMU
SHM_PAGEABLE virtual memory resources are shared and the dynamic shared memory (DISM) framework is created. The dynamic shared memory can be resized dynamically.

36. ISM Feature Avoid generate redundant mappings to physical pages
Intimate shared memory is an important optimization that makes more efficient use of the kernel and hardware resources involved in the implementation of virtual memory and provides a means of keeping heavily used shared pages locked in memory.
Large page size automatically enabled

37. Example - ISM used for Database
Without ISM
400 database processes
2GB shared segment ~ KB pages 8Bytes for each page mapping-->2M
2M*400
800Mbytes for mapping!
With ISM
2Mbytes only.

38. System V message queues
Concept
For process to send and receive messages in various size in an asynchronous fashion.
Feature
Dramatical loadable module
Depends on /kernel/sys/msgsys
/kernel/misc/ipc
Kernel resource consumption
Kernel memory
Resource map
Dramatical?

39. System V IPC - message queue API
Header file and system calls
#include <sys/msg.h>
int msgget(key_t key, int msgflg);
int msgctl(int msqid, int cmd, struct msqid_ds *buf);
int msgsnd(int msqid, const void *msgp, size_t msgsz, int msgflg);
size_t msgrcv(int msqid, void *msgp, size_t msgsz, long int msgtyp, int msgflg);
More available from man page
e,g, man msg

40. Data Structure Struct msqid_ds{
Struct ipc_perm msg_perm; /*read-write perms*/
Struct msg msg_first; /ptr to first message on queue*/
Struct msg *msg_last; /ptr to last message on queue*/
msglen_t msg_cbytes;/*current #bytes on queue*/
msgqnum_t msg_qnum; /*current #of messages on queue*/
msglen_t msg_qbytes;/*max # of bytes allowed on queue*/
pid_t msg_lspid; /*pid of last msgsnd()*/
pid_t msg_lrpid; /*pid of last msgrcv()*/
time_t msg_stime; /*time of last msgsnd()*/
time_t msg_rtime; /*time of last msgrcv()*/
time_t msg_ctime; /*pid of last msgctl()*/ }

41. System V message queues structures
Data on message queue
header with each message ID
address map from kernel memory to user memory
kernel msglock for each message queue

42. System overhead with message queues
Kernel memory
check first,no greater than 25% available kernel memory
Resource map
Identifier msqid_ds sys/msg.h
Struct msg

43. System tunable parameter
/etc/system

44. Available resource control
process.max-msg-messages
msginfo_msgtql(obsolete)
maximum number of messages on a message queue
process.max-msg-qbytes
msginfo_msgmnb(obsolete)
maximum number of bytes of messages on a message queue
project.max-msg-ids
msginfo_msgmni(obsolete)
maximum number of message queue Ids allowed for a project

45. System V Semaphore Concept
mechanical signaling device or a means of doing visual signaling.
A method of synchronizing to a sharable resource by multi processes.
Feature
P – try or attempt; V – increase
Semaphore sets available
Kernel resource consumption
Kernel memory
Resource map

46. System V IPC semaphore -API
Header file and system calls
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/sem.h>
int semget(key_t key, int nsems, int semflg);
int semctl(int semid, int semnum, int cmd, ...);
int semop(int semid, struct sembuf *sops, size_t nsops);
More available from man page
Man sem
e,g.semget(2)
semop(......,size_t nsops)-> how many arrays there is in sembuf data structure?

47. System V semaphore Binary semaphore Counting semaphore
0 or 1,like mutex
Counting semaphore
0~system limitation ,P/V
Set of counting semaphores
One or more,each set has a limitation
System limitation – for Posix IPC

48. Data structures for semaphore
Struct semid_ds
Struct sem_buf

49. System overhead with semaphore
Kernel memory
Must less than 25% of kernel memory
Resource map allocation
Sem structure based on semmns
Identifier semid_ds
Undo structure pointer
Undo structure themselves
Kernel mutex lock
Semmns -> total size of semaphore
semusz -> total bytes of undo structure

50. System tunable parameter
/etc/system(part)

51. System tunable parameter
/etc/system

52. Available resource control
process.max-sem-ops
seminfo_semopm(obsolete)
Maximum operations per semop(2) call
process.max-sem-nsems
seminfo_semmsl(obsolete)
Maximum semaphores per identifiers
project.max-sem-ids
seminfo_semmni(obsolete)
Number of semaphore identifiers

53. Agenda IPC Overview System V IPC Posix IPC
Tools for performance analysis
System configuration
Hands On Lab
Why smf
what can smf do

54. Types Posix message queues Posix semaphore Posix shared memory API
Posix IPC Overview
Types
Posix message queues
Posix semaphore
Posix shared memory
API
Based on file system
open files /memory spaces – the basic limitations.
OPEN_MAX - max # of files a process can have open

55. Posix IPC and System V IPC
Common points
Types/Function
Difference-Implementation
Library – libposix4.so vs libc.so
Implementation
Built on Solaris file memory mapping interface -- mmap(2)
acquired desired resource by using a file name convention

56. Difference with System V IPC
(continue...)
No APIs entering into kernel are executed
No tunable parameters are required
Limiting factors - Per-process limit
open files
memory spaces
Common routines
_pos4obj _open/_pos4obj _name
_open _nc/_close _nc
_pos4obj _lock/_pos4obj _unlock
All POSIX IPCs call mmap directly or indirectly.
Message Queue/Semaphore calls mmap directly.
Shared memory calls mmap in the middle.
open/shm_open ,then mmap()

57. mmap
Function
mapping a file or some other named objects into a process's space address.
Achievement
Common files used for memory reflecting
Special files for anonymous memory reflecting
shm_open for unallied processes shared memory
API #include <sys/mman.h>
void *mmap(void *addr,size_t len,int prot,int flags,int fd,off_t offset);
A file or shared object will be mapped into different process's address space.
Special files—not exist in the file system.
Process with different parents-- shm_open first
Not all the Fds can be used to mmap(),e.g.socket fd.
Only those who can be used by open/write will do this.
FIFO/Pipe/message queue will involved in kernel->user land data exchange. Mmap will not.
Kernel will do the mmap in the background and won't do any I/O operation.

58. Process address space with mmap(2)
Memory mapped – can be private or shared.
shm_open can decided that.

59. Posix Semaphores Named semaphore sem_open()/sem_close()/sem_unlink
Unnamed semaphore(memory based)
sem_init()/sem_destroy
Named-- process with no parents' relationship , easy to use by the name
unnamed-- unnecessary to get the name,different threads in a process.
3 types of realization of named semaphore :
FIFO/ memory mapping I/O and system V semaphore
named semaphore: based on file system files exists/non-exists.
unnamed :based on memory.
If the semaphore is used in 1 process- destroyed when process exit. Otherwise,kernel persist.
Used between threads/processes.
It will use FD when is created,but free it later.

60. System-imposed limits
<unistd.h>
SEM_NSEMS_MAX
Maximum number of semaphores per process
SEM_VALUE_MAX
Maximum value of a semaphore
e,g. #getconf -a|grep SEM

61. Posix message queue
With the right privilege will be able to write/read messages,with priority.
Different with System V message queue:
1.return the message with the highest priority
2.signal /thread will be issue whenever message is put onto the queue.
System V:
no priority
no signal

62. System limits in /usr/include/limits.h
Posix message queue
System limits in /usr/include/limits.h
MQ_OPEN_MAX
MQ_PRIO_MAX
e,g getconf -a |grep MQ
Tunable parameters
mq_open(3R)
mq_setattr(3R)

63. Agenda Solaris multi-thread process Inter-processes IPC System V IPC
Posix IPC
Tools for performance analysis
System configuration
Hands On Lab

64. Performance analysis process
Understand the problem.
Collect data with tools for performance analysis.
iostat,kstat,mdb,pmap,vmstat,ps,etc.
Proc tools
Dtrace
Sun studio11
Separate all the data you get from different layers.
CPU
Memory
System I/O
File system
...

65. Performance tuning process
Find the area and the period of the bottleneck.
Set up performance goal.
Performance tuning
E,g. Resource control with IPC.
Review and repeat until meet tuning goals .
Report with comparison available.
Tools for performance analysis related to IPC

66. kmstat(1M)

67. vmstat(1M)
Rports virtual memory statistics regarding kernel thread, virtual memory, disk, trap, and CPU activity.
memory-report on usage of virtual and real memory
mf-minor faults
pi- kilobytes paged in
po- kilobytes paged out

68. pmap -x(1M)

69. Pmap -x example

70. ipcs(1M) report inter-process communication facilities status
-m active shared memory segments.
-q active message queues.
-s active semaphores.
-i number of ISM attaches to shared memory segments.
-p process number
-A all print options, -b, -c, -i, -J, -o, -p, and -t.
-z information about facilities associated with the specified zone
-Z information about all zones

71. ipcrm(1) Remove a message queue, semaphore set, or shared memory ID
-m shmid
-q msqid
-s semid
-M shmkey
-Q msgkey
-S semkey
-z zone

72. dtrace(1M)
Comprehensive dynamic tracing framework for the Solaris Operating system.
Powerful infrastructure to permit administrators,developers to understanding behavior of the operating system and user processes.
Observe the system call related to ipc
fbt provider provides probe into every function in the kernel
shmsys provider provides probe into system API with ipc
ipc module,probe related to ipc in kernel

73. Sun Studio- Performance analyzer
Combination of compiler,libraries of optimized functions and tools for performance analysis.
Command and sub-command
er_print-- generate a plain-text report of performance data
Collect – simplest interface for collecting data
Dbx collector – performance analyzing for active process
Analyzer – GUI tool
#collect -L unlimited -A copy -F on -d /tmp/a.out ->/tmp/test.1.er
#analyzer test.1.er

74. Agenda Solaris multi-thread process Inter-processes IPC System V IPC
Posix IPC
Tools for performance analysis
System configuration
Hands On Lab
Why smf
what can smf do

75. How to modify kernel parameters?
System configuration
How to modify kernel parameters?
Modify the /etc/system
Use the kernel debugger(kmdb)
Use the modular debugger(Mdb)
Use the ndd command to set TCP/IP parameters
Modify the /etc/default files
Viewing Solaris system configuration information
Sysdef(1M) command
Kstat(1M) or Kstat(3Kstat)

76. Example Setting a parameter in /etc/system
Set nfs:nfs_nra=4(read-ahead blocks that are read for file system mounted using NFS version 2 software)
Using mdb to change a value

77. Sysdef command

78. Reference Solaris Internal Unix Network Programming Vol II
Solaris Dynamic Tracing Guide
Solaris Modular Debugger Guide
Solaris Tunable Parameters Reference Manual

79. System Config IPC
Iris Zhu