1. mgr inż. Marcin Borkowski UNIX POSIX Threads

2. mgr inż. Marcin Borkowski Introduction to Threads ● Threads provide different (to processes) method to compute in parallel ● Unlike processes threads: – are fast to start – use less system resources – share the same address space ● But many threads in single process: – share the same process environment – share per process limits (files, memory, CPU) – require synchronization (see: synchronization)

3. mgr inż. Marcin Borkowski Introduction to Threads ● All threads of one process share: – global memory (data and heap segments) – process ID – parent process ID – process group ID and session ID – controlling terminal – user and group IDs – open file descriptors – record locks – signal dispositions (handlers and ignoring but not masking) – file mode creation mask ( umask )

4. mgr inż. Marcin Borkowski Introduction to Threads ● All threads of one process share (cont.): – current directory and root directory ( chroot ) – interval timers ( setitimer ) and POSIX timers ( timer_create ) – nice value – resource limits – measurements of the consumption of CPU time and resources – semaphore undo values – static local variables

5. mgr inż. Marcin Borkowski Introduction to Threads ● All threads of one process have independent: – stack ● automatic local variables ● function calls – thread ID – signal mask – the errno variable – alternate signal stack – real-time scheduling policy and priority

6. mgr inż. Marcin Borkowski Introduction to Threads ● Threading models: – user level threads ● lower kernel resources consumption ● cannot benefit from SMP architecture – kernel level threads ● higher (sometimes comparable with process) kernel resources consumption ● can benefit from SMP – mixed (POSIX) threads ● user controls the scheduling scope per thread (see: attributes)

7. mgr inż. Marcin Borkowski Introduction to Threads ● To compile a multi-threaded program using POSIX threads special compiler flag may be required ( -pthread ), gcc lists the following architectures to require this: – IA-64 – RS/6000 – PowerPC – SPARC ● Library of POSIX threads must be linked, gcc requires -lpthread option

8. mgr inż. Marcin Borkowski Introduction to Threads ● Older non POSIX thread implementations are usually incompatible ● Linux used to have (and in some installations still has) its own implementation of threads: – threads implemented on lightweight processes (PIDs, locks, waiting for children etc.) – signal handling was incompatible with POSIX – signals SIGUSR1 and SIGUSR2 or first 3 Real Time signals were used internally

9. mgr inż. Marcin Borkowski Introduction to Threads ● Linux POSIX thread implementation is referred to as Native Posix Thread Library (NPTL) – use first two Real Time signals and application can not block, ignore or handle them – all threads share nice value (not POSIX compliant) – to determine threading library on Linux: $getconf GNU_LIBPTHREAD_VERSION (or) $/lib/libc.so.6 (the number of libc.so may change)

10. mgr inż. Marcin Borkowski Introduction to Threads ● Most of pthread_ functions return 0 on success and error code on failure, errno is not set thus function perror does not report errors, function strerror_r should be used instead ● Non of pthread_ functions can ever return EINTR as error code, those functions are required to restart automatically after interruption by signal

11. mgr inż. Marcin Borkowski Thread Lifetime ● Threads are identified by Thread ID (TID): – data type pthread_t can be implemented as structure! – function pthread_self returns TID of calling thread – function pthread_equal test for TID equality – function gettid is Linux specific and as not portable should not be used in students' work!

12. mgr inż. Marcin Borkowski Thread Lifetime ● Thread creation (f. pthread_create ): – created thread instantly starts to execute pointed function in separate thread – thread function ● takes a void pointer, all shared data accessed by thread should be addressable through data pointed by this pointer, inter-thread communication is not a good excuse to use global variables ● thread that exits its function is terminated as if function pthread_exit was called ● returns void pointer that can be used to return data to calling thread (see. thread_join), but if thread is detached this value will be ignored

13. mgr inż. Marcin Borkowski Thread Lifetime ● Thread creation (f. pthread_create ) cont.: – attributes (see next slides) ● can be set only at creation time ● NULL means default values for all the attributes ● attribute values are copied and stored internally, the same attribute structure can be later modified without affecting created thread – new thread TID is known to the creating thread – signals mask is inherited but pending signals are not

14. mgr inż. Marcin Borkowski Thread Lifetime ● Join or detach – if thread return status is required or other thread must synchronize on its termination – thread should be joined ● function pthread_join suspends the calling thread until specified thread (by TID) terminates ● the results of multiple simultaneous join calls specifying the same target thread are undefined ● thread that is about to be joined is called joinable ● this state (joinable) is default for newly created threads ● if thread is cancelled at pthread_join function it either finishes joining or cancellation handler is executed (see: cancellation) ● simultaneous joins in group of threads can cause a deadlock !

15. mgr inż. Marcin Borkowski Thread Lifetime ● Join or detach (cont.) – otherwise it is more comfortable to detach ● thread can terminate at any time, and canot be joined ● detaching can be set by calling pthread_detach, any thread in application can do this ● the results of multiple calls to pthread_detach specifying the same target thread are undefined ● detaching can be set as thread attribute on creation time – terminated threads that are neither detached nor joined occupy system resources, students are obliged to eliminate such threads ASAP.

16. mgr inż. Marcin Borkowski Thread Lifetime ● Termination: – call to exit terminates all threads (process as whole) – thread can terminate by exiting the thread function ( return (void*) ) or by calling thread_exit(void*) function – return data can not be a local auto variable – if thread has cancellation handlers not executed yet, they will be executed at exit in FILO order (see cancellation) – thread's local resources are released, no process level (opened descriptors, heap etc.) resources are altered

17. mgr inż. Marcin Borkowski Thread Lifetime ● Cancellation: – thread can force another thread to terminate – the cancellation process is asynchronous, function pthread_cancel only makes a request – not all threads can be cancelled ● function pthread_setcancelstate can set cancel to: ● PTHREAD_CANCEL_ENABLE (can be cancelled, default) ● PTHREAD_CANCEL_DISABLE (can not be cancelled) ● if thread re-enables cancel pending cancel will be executed

18. mgr inż. Marcin Borkowski Thread Lifetime ● Cancellation (cont.): – not all threads can be cancelled immediately ● two types of cancellation can be set with function pthread_setcanceltype to: ● PTHREAD_CANCEL_DEFERRED (default, thread can terminate only at cancellation points) ● PTHREAD_CANCEL_ASYNCHRONOUS (thread can terminate at any moment) – as a general rule, a function that changes its cancellation state or its type should restore the value before returning.

19. mgr inż. Marcin Borkowski Thread Lifetime ● Cancellation (cont.): – many system functions are cancellation points e.g.: ● aio_suspend, close, creat, msgrcv, msgsnd ● nanosleep, open, pause, poll, ● pthread_cond_timedwait, pthread_cond_wait ● pthread_join, read, select ● sem_wait, sigsuspend, sleep ● sync, system, wait, waitpid, write, fcntl – function pthread_testcancel works as cancellation point only

20. mgr inż. Marcin Borkowski Thread Lifetime ● Cancellation (cont.): – cancellation of thread can lead to inconsistent application state (semaphores, mutexes and conditionals) or memory leaks and resource waste – to release or free resources at cancellation or at the end of thread use stack of clean-up handlers ● handlers are pushed on the stack with pthread_cleanup_push function ● each handler is one parameter (void*) function ● handler can be popped (and executed if required) with pthread_cleanup_pop function

21. mgr inż. Marcin Borkowski Thread Lifetime ● Cancellation (cont.): – clean-up handlers are not called at exit function call – clean-up handler must also be called directly to release the resource as POSIX requires to match push and pop operation within lexical scope ({}) – a function that uses clean-up handlers and can always be safely cancelled is called cancel safe – cancelling semantics may add a lot of code thus not all portable functions are required to be cancel safe – use cancelling only when it cannot be avoided

22. mgr inż. Marcin Borkowski Thread Attributes ● All thread attributes can be set only at creation time (f. pthread_create ), except detach state ● Attributes are stored in pthread_attr_t structure and manipulated in object like fashion by get/set functions ● Set of Attributes can be extended in the future ● Structure must be initialised (default values) with pthread_attr_init and destroyed (set invalid values) with pthread_attr_destroy ● Structure can be used to create many threads

23. mgr inż. Marcin Borkowski Thread Attributes ● Detach state (see: join or detach section) – pthread_attr_getdetachstate pthread_attr_setdetachstate ● PTHREAD_CREATE_JOINABLE (default) ● PTHREAD_CREATE_DETACHED ● Thread stack – size and location of stack (allowed values are implementation depended, not portable attribute) – pthread_attr_getstack – pthread_attr_setstack – default stack address is assigned by system, ca. 1Mb size

24. mgr inż. Marcin Borkowski Thread Attributes ● Scheduling (threading model) – pthread_attr_getscope pthread_attr_setscope ● PTHREAD_SCOPE_PROCESS – default SUN, FreeBSD6 ● PTHREAD_SCOPE_SYSTEM – default on HP-UX, FreeBSD7, Linux ● Linux supports only system scope ! – scheduling algorithm and priority can be set as well (read man page on pthread_attr_[get|set]_[schedpolicy|schedp aram] )

25. mgr inż. Marcin Borkowski Threads Communication ● Memory (available via pointer to the thread function, not global variables) – requires synchronization ● Traditional IPC (pipe, FIFO, socket, POSIX message queue, SysV message queues) ● Signals (see Threads and Signals)

26. mgr inż. Marcin Borkowski Threads and Streams ● Operations on stream are atomic i.e. many threads do not mix data sent or received from the stream in single function call (e.g. printf ) ● Thread can control atomicity of many function calls by means of – flockfile – ftrylockfile – funlockfile (see files chapter)

27. mgr inż. Marcin Borkowski Thread Safe ● Thread safe function or code can be executed simultaneously in two or more threads without spurious results – most of POSIX functions used in this lecture are thread-safe (see exceptions in Not Thread Safe section) – errno is thread safe as it is usually implemented as macro, each thread has its own errno value – malloc/free is thread safe – operation on thread's private data is thread safe

28. mgr inż. Marcin Borkowski Not Thread Safe ● Thread must treat not safe functions and code as critical section and synchronize access with IPC ● Application logic must ensure that only one thread at given time uses such function or code – functions that: ● write to static buffers ● modify process specific resources (like CWD) will never be thread safe – access to shared (not private) data in program is not thread safe

29. mgr inż. Marcin Borkowski Not Thread Safe ● POSIX functions that are not required to be thread- safe and were discussed in this lecture: – ftw, nftw – gethostbyaddr – gethostbyname – rand – readdir – strtok – strerror ● Most of not thread safe functions have ( _r ) alternative

30. mgr inż. Marcin Borkowski Threads and Signals ● Signals sent to process (f. kill ) are handled by arbitrarily selected thread that is not masking given signal (usually only one thread does not mask anticipated signals) ● Signal sent to specific thread (f. pthread_kill ) – asynchronous (RT signal can be lost if pending signals queue is overflown) ● Implicitly generated signal (e.g. SIGPIPE) is delivered to the thread that triggered the signal

31. mgr inż. Marcin Borkowski Threads and Signals ● Default signal actions (kill, suspend) always affect entire process ● All threads of one process share signal handlers ● Each thread has its own signal mask, initial mask is inherited from the creating thread ● pthread_sigmask function sets per thread signal masking, sigprocmask should not be used when multiple threads exist within a process

32. mgr inż. Marcin Borkowski Threads and Signals ● If two or more identical signals are delivered to process and more than one thread is not blocking them it may happen that the same signal handler will be executed simultaneously by two or more threads as signal handler block the signal only for current thread ● The best signal handling strategy for threads is to block communication (not system generated like SIGFPE or SIGSEGV) signals before threads are actually being spawned and dedicate one thread that should unblock and handle those signals

33. mgr inż. Marcin Borkowski Threads and Processes Mixed ● fork function does not copy threads, only fork calling thread is copied, all address space including mutexes, semaphores etc. states is copied ● Child process of multi-threaded program usually cannot continue to work normally as all other threads are missing thus fork use is limited, commonly it is used to start another program ( exec functions family) ● Processes should be created before threads (Apache does it with MPM_worker)

34. mgr inż. Marcin Borkowski Threads and Processes Mixed ● Any thread can wait for any child of the process ● Function pthread_atfork installs fork handlers: – prepare (executed prior to fork, e.g. lock all mutexes ) – parent (executed after fork in parent process e.g. release mutexes) – child (executed after fork in child process, e.g. release mutexes) – can be used to set mutexes and other resources to consistent state

35. mgr inż. Marcin Borkowski Threads and Processes Mixed ● Function pthread_atfork (cont.): – multiple pthread_atfork calls install multiple handlers – handler function is parameterless – function widely used in multi-threaded libraries

36. mgr inż. Marcin Borkowski Threads Synchronization ● Posix synchronisation tools: – mutex – conditional variable – read-write lock (not covered here) – semaphore ● SysV IPC semaphores can be used but: – there is no good reason to mix standards – SEM_UNDO counter is implemented per process not per thread

37. mgr inż. Marcin Borkowski Threads Synchronization ● Mutex – two states: available or locked – atomic access – locking thread owns the mutex – only one thread can own the mutex, the rest must wait – queue of waiting threads ● POSIX does not enforce any order of access thus starvation is possible ● many implementations provide reasonable scheduling – mostly used for short time critical section access – can be used to protect not thread safe system functions

38. mgr inż. Marcin Borkowski Threads Synchronization ● Mutex (cont.) – static initialisation: pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER; – dynamic or with non-default attributes: pthread_mutex_init function – static initialiser is guaranteed to be executed only once before any thread begins execution, default value is “available” – already initialised mutex (f. pthread_mutex_init ) should not be reinitialised – thread must not use a copy of mutex structure

39. mgr inż. Marcin Borkowski Threads Synchronization ● Mutex (cont.) – mutex must be destroyed after it turns useless – only unlocked mutex can be destroyed – function: pthread_mutex_destroy voids the mutex – destroyed mutex cannot be used but can be reinitialised – mutex operations: ● pthread_mutex_lock – blocks until mutex is available ● pthread_mutex_trylock – never blocks ● pthread_mutex_timedlock – time-outed locking ● pthread_mutex_unlock – release the mutex

40. mgr inż. Marcin Borkowski Threads Synchronization ● Mutex (cont.) – attributes can be set to non default values at mutex initialisation, attribute structure is manipulated with: ● pthread_mutexattr_init and pthread_mutexattr_destroy – attribute structure can be reused – mutex attribute: process shared – pthread_mutexattr_getpshared – pthread_mutexattr_setpshared – if mutex is PTHREAD_PROCESS_SHARED then it can be used from any process that can access mutex (via mmap ) otherwise it must be PTHREAD_PROCESS_PRIVATE

41. mgr inż. Marcin Borkowski Threads Synchronization – mutex attribute: type ● pthread_mutexattr_gettype ● pthread_mutexattr_settype ● PTHREAD_MUTEX_NORMAL does not check deadlock, cannot be locked again by the same thread without unlocking first, cannot be unlock by a different thread than the owner, can not be unlocked if available. If any of above is attempted the result is undefined ● PTHREAD_MUTEX_ERRORCHECK check deadlock, cannot be locked again by the same thread without unlocking first, cannot be unlock by a different thread than the owner, can not be unlocked if available. If any of above is attempted the error is reported

42. mgr inż. Marcin Borkowski Threads Synchronization – mutex attribute: type (cont.) ● PTHREAD_MUTEX_RECURSIVE – check for deadlock – can be locked again by the same thread without unlocking first – cannot be unlocked by a different thread than the owner, cannot be unlocked if available. If above is attempted the error is reported – can break conditional variable synchronization - not advised – must be unlocked the same number of times as it was locked ● PTHREAD_MUTEX_DEFAULT – default type – works as PTHREAD_MUTEX_NORMAL but implementation can remap it to any other type

43. mgr inż. Marcin Borkowski Threads Synchronization ● Conditional Variable – tightly connected with mutex – method to wait for certain condition on data protected by mutex - value of some variable or set of variables – eliminates need for constant polling the value as thread is suspended until the value is changed – many threads can wait on the same condition – the wakeup of waiting threads is controlled by other threads – similar to synchronized block in Java or monitor idea

44. mgr inż. Marcin Borkowski Threads Synchronization (conditional variable use case) ● Thread A – lock associated mutex and check condition – if condition is not true call pthread_cond_wait () to perform a blocking wait for signal from Thread- B. Call to pthread_cond_wait() atomically unlocks the associated mutex – when signalled, wake up. Mutex is implicitly and atomically locked. – unlock mutex ● Thread B – lock associated mutex – change the value of the global variable that Thread-A is waiting upon. – check value of the global Thread-A wait variable. If it fulfils the desired condition, signal Thread-A. – unlock mutex.

45. mgr inż. Marcin Borkowski Threads Synchronization ● Conditional Variable (cont.) – static initialisation: pthread_cond_t cond = PTHREAD_COND_INITIALIZER; – dynamic or with non-default attributes: pthread_cond_init function – static initialiser is guaranteed to be executed only once before any thread begins execution – already initialised conditional (f. pthread_cond_init ) should not be reinitialised – thread must not use a copy of conditional structure

46. mgr inż. Marcin Borkowski Threads Synchronization ● Conditional Variable (cont.) – conditional must be destroyed after it turns useless – only conditional without waiting threads can be destroyed – function: pthread_cond_destroy voids the conditional variable – destroyed conditional can not be used but can be reinitialised

47. mgr inż. Marcin Borkowski Threads Synchronization ● Conditional Variable (cont.) – waiting on condition with pthread_cond_wait ● mutex must be locked ● atomically unlock mutex and suspend the thread until signalled ● when signalled acquires the mutex before returning (mutex is owned by the thread as execution continues) ● there condition may not be true if wakening thread is not working properly or in some implementations (e.g. spurious wakeup on SMP), it is advisable to always recheck the condition before continuing (test it in a loop) ● if cancelled before being signalled mutex is acquired so clean-up handler can release it as if cancellation occurred before or after the conditional wait

48. mgr inż. Marcin Borkowski Threads Synchronization ● Conditional Variable (cont.) – waiting on condition with pthread_cond_timedwait ● adds time-out to conditional wait ● if time-out occurs the condition can be true as there can be a race condition between time-out and condition change – signalling (wakening) with pthread_cond_signal mutex should be locked, at least one waiting thread is resumed – signalling (wakening) with pthread_cond_broadcast ● all waiting threads are signalled, the order of execution is enforced by order of mutex acquisition

49. mgr inż. Marcin Borkowski Threads Synchronization ● Conditional Variable (cont.) – attributes can be set to non default values at conditional initialisation, attribute structure is manipulated with: ● pthread_condattr_init and pthread_condattr_destroy – attribute structure can be reused – attribute: process shared – pthread_condattr_getpshared – pthread_condattr_setpshared – if conditional is PTHREAD_PROCESS_SHARED then it can be used from any process that can access variable (via mmap ) otherwise it must be PTHREAD_PROCESS_PRIVATE

50. mgr inż. Marcin Borkowski Semaphores – Thread and Process Synchronization ● Semaphores semantics is similar to sysV, the interface is different: – semaphores are created as separate entities not arrays – named semaphores ● available for any process or thread to synchronise on ● kernel persistence (if not unlinked) ● stored in /dev/shm on Linux – unnamed semaphores ● available only to threads and processes that can access the memory occupied by semaphore ● process persistence

51. mgr inż. Marcin Borkowski Semaphores – Thread and Process Synchronization – require librt (-lrt) on Linux – does not require libpthread – semaphore functions set errno – some of semaphore functions can be interrupted by EINTR

52. mgr inż. Marcin Borkowski Semaphores – Thread and Process Synchronization ● Creating an unnamed semaphore: – function sem_init – semaphore is stored in sem_t variable – if pshared argument is non-zero then any process that can access the memory of structure sem_t can synchronise on the semaphore – if pshared is zero then only threads of the same process can use the semaphore – semaphore initial value is set upon creation ( value parameter) – no flow similar to sysV semaphores – semaphore structure copy is not a semaphore

53. mgr inż. Marcin Borkowski Semaphores – Thread and Process Synchronization ● Creating of named semaphore: – function sem_open – semaphore address is returned as sem_t pointer, constant SEM_FAILED indicates error – semaphore initial value is set upon creation ( value parameter) - no flow similar to sysV semaphores – flags parameter can be O_CREAT optionally bitwise or with O_EXCL – name of semaphore should start with slash (/) – multiple calls to sem_open without O_CREAT flag must return the same address

54. mgr inż. Marcin Borkowski Semaphores – Thread and Process Synchronization – both sem_init and sem_close (with O_CREAT) should be called only once for one semaphore ● Function sem_close – indicates that named semaphore in no longer used by the process (all threads) – does not affect semaphore value ● Function sem_destroy – removes unnamed semaphore – only available semaphore (no process blocks on it) can be destroyed

55. mgr inż. Marcin Borkowski Semaphores – Thread and Process Synchronization ● Function sem_unlink – removes the named semaphore so no new process/thread can not open it – has no effect on processes/threads that already reference the semaphore, postponed destruction – does not change semaphore value – if new semaphore is created with the same name it is not connected to the old one

56. mgr inż. Marcin Borkowski Semaphores – Thread and Process Synchronization ● semaphore operations (obvious) – decrease semaphore value ● sem_wait (blocks if value is 0) ● sem_trywait (never blocks) ● sem_timedwait (time-out) – increase semaphore value (never blocks) ● sem_post

57. mgr inż. Marcin Borkowski Multi Threaded Library Helper Functions ● pthread_once – initialise the library ● pthread_setspecific – set library internal data for thread (global variables are common to all threads), data is identified by key ● pthread_getspecific - read library internal data for thread, requires valid key ● pthread_key_create – generate key ● pthread_key_delete – destroy key ● pthread_atfork – see Threads and Processes

58. mgr inż. Marcin Borkowski Threads - Examples ● How to create and join threads (by Jerzy Bartuszek) for (i = 0; i<5; i++){ targ[i].id = i + 1; targ[i].cond = &cond; targ[i].mutex = &mutex; if(pthread_create(&thread[i], NULL, threadfunc,\ (void *)&targ[i])!=0) PTERR(); }... for (i = 0; i<5; i++) if(pthread_join(thread[i], NULL)!=0) PTERR();

59. mgr inż. Marcin Borkowski Threads - Examples ● How to create detached thread pthread_t thread; pthread_attr_t attr; if(pthread_attr_init(&attr)!=0) PTERR(); pthread_attr_setdetachstate(&attr, \ PTHREAD_CREATE_DETACHED); if(pthread_create(&thread, &attr, &gra, sock)!=0) PTERR(); pthread_attr_destroy(&attr);

60. mgr inż. Marcin Borkowski Threads - Examples ● How to use clean-up handler (by Jerzy Bartuszek) void cleanup(void *arg){ pthread_mutex_unlock((pthread_mutex_t *)arg); } void *threadfunc(void *arg){ struct thread_arg targ; memcpy(&targ, arg, sizeof(targ)); pthread_cleanup_push(cleanup, (void *)targ.mutex); pthread_mutex_lock(targ.mutex);... pthread_cleanup_pop(1);...

61. mgr inż. Marcin Borkowski Threads - Examples ● How to use conditional variable /*thread A*/ pthread_mutex_lock(mutex); while(cond_var!=0) pthread_cond_wait(cond,mutex); cond_var++; pthread_mutex_unlock(mutex);... /*thread B*/ pthread_mutex_lock(mutex); cond_var--; if(0==cond_var) pthread_cond_signal(cond); pthread_mutex_unlock(mutex);