1. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Multitasking & Synchronization

2. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Introduction (1) There are two classes of media: There are two classes of media: 1) Static media do not have a time dimension, and their contents and meaning do not depend on the presentation of time. Static media include alphanumeric data, graphics, and still images. 2) Dynamic media have a time dimension, and their meanings and correctness depend on the rate at which they are presented. Dynamic media include animation, audio, and video. 2) Dynamic media have a time dimension, and their meanings and correctness depend on the rate at which they are presented. Dynamic media include animation, audio, and video.

3. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Introduction (2) Video must be played at 25 frames/s ( or 30 frames/s, depending on the system ) in order to have perceptually smooth movement. Video must be played at 25 frames/s ( or 30 frames/s, depending on the system ) in order to have perceptually smooth movement. Similarly, only one playback rate is sensible when we play audio. Playback at slower or faster rates will distort the meaning / quality of sound. Similarly, only one playback rate is sensible when we play audio. Playback at slower or faster rates will distort the meaning / quality of sound. Since these media must be played back continuously at a fixed rate, they are often called continuous media. They are also called isochronous media, because of the fixed relationship between each media unit and time.

4. Lars A. Conrad-Hansen, Introduction to Multimedia Systems We define multimedia systems as capable of handling at least one type of continuous media in digital form as well as static media ( e.g., image + sound) Introduction (3)

5. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Classes of Multimedia Systems Multimedia systems can be classified into: 1) Standalone systems use dedicated system resources. The amount of multimedia information may be limited and multimedia data communications are not supported. 2) Distributed systems share both system resources and information resources and can support communication among users. 2) Distributed systems share both system resources and information resources and can support communication among users.

6. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Challenges of Multimedia Computing and Communications Multimedia data have a time dimension and must be transmitted, processed and presented in a fixed rate in most applications. So, the multimedia computing and communications system must meet this timeliness requirement. Multimedia data have a time dimension and must be transmitted, processed and presented in a fixed rate in most applications. So, the multimedia computing and communications system must meet this timeliness requirement. Multimedia applications normally use multiple related media simultaneously. Multimedia applications normally use multiple related media simultaneously. Multimedia data are data intensive, so they must be compressed, and high-speed communications networks and powerful computers are required to handle them. Multimedia data are data intensive, so they must be compressed, and high-speed communications networks and powerful computers are required to handle them.

7. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Networked Multimedia Synchronization (1) is defined as the correct or desired temporal appearance (presentation) of the media items. Synchronization is defined as the correct or desired temporal appearance (presentation) of the media items.

8. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Networked Multimedia Synchronization (2) Each Package contains a bit of information, that tells at which time it needs to be handled !

9. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Networked Multimedia Synchronization (3) The aim of temporal synchronization specification is to represent all possible temporal relationships among media. The aim of temporal synchronization specification is to represent all possible temporal relationships among media.

10. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Networked Multimedia Synchronization (4) In time-line-based temporal specification, the presentation time of each stream is specified to a common clock. In time-line-based temporal specification, the presentation time of each stream is specified to a common clock.

11. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Required Synchronization Accuracy (1) Synchronization can be measured by four parameters: Synchronization can be measured by four parameters: 1) Delay : Measures end-to-end delay in live applications and response time in retrieval applications 2) Delay jitter : Measures the deviation of presentation time of continuous media samples from their fixed or desired presentation time 3) Intermedia skew : Measures the time shift between related media from the desired temporal relationship 4) Tolerable error rate : Measures the allowable bit error rate and packet error rate for a particular stream in a particular application. There are two types of errors: - - Bit error - - Packet loss

12. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Required Synchronization Accuracy (2) Very sensitive, since we use it every day to Localize sounds !

13. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Synchronization Problems The clock rates at the transmitter and the receiver might be different. If the clock rate is faster at the transmitter than at the receiver, the transmitter sends more data than what the receiver can consume, causing data overflow at the receiver. On the other hand, if the clock rate is slower at the transmitter than at the receiver, the receiver has a data-starvation problem. The clock rates at the transmitter and the receiver might be different. If the clock rate is faster at the transmitter than at the receiver, the transmitter sends more data than what the receiver can consume, causing data overflow at the receiver. On the other hand, if the clock rate is slower at the transmitter than at the receiver, the receiver has a data-starvation problem. When connectionless transport protocol is used, packets belonging to a stream may arrive at the receiver out of order. When connectionless transport protocol is used, packets belonging to a stream may arrive at the receiver out of order. When multiple sources are involved, it may be difficult to coordinate the stream transmission times at different locations. When multiple sources are involved, it may be difficult to coordinate the stream transmission times at different locations. It is important to maintain synchronization during and after user interactions. F. ex., when a presentation is resumed after pausing for some time, the presentation should be resumed synchronously. Thus, synchronization is complicated by user interactions. It is important to maintain synchronization during and after user interactions. F. ex., when a presentation is resumed after pausing for some time, the presentation should be resumed synchronously. Thus, synchronization is complicated by user interactions. Coding / decoding times for different streams may vary. Coding / decoding times for different streams may vary.

14. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Mechanisms to achieve multimedia synchronization (1)

15. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Mechanisms to achieve multimedia synchronization (2)

16. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Mechanisms to achieve multimedia synchronization (3) Measures to counter delay jitter can be divided into two groups: Measures to counter delay jitter can be divided into two groups: Corrective measures do not do anything to the transport protocol and underlying networks. They try to smooth out the jitter at the destination before media data are presented to the user. Corrective measures do not do anything to the transport protocol and underlying networks. They try to smooth out the jitter at the destination before media data are presented to the user. Preventive measures try to minimize delay and delay jitter by improving transport protocol and underlying networks. Preventive measures try to minimize delay and delay jitter by improving transport protocol and underlying networks.

17. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Mechanisms to achieve multimedia synchronization (4) Corrective measures : Buffering Corrective measures : Buffering Preventive Measures : When delay jitter is too large, corrective measures are not very useful, because they require a large buffer and long buffering time, which is inappropriate for many applications. Preventive Measures : When delay jitter is too large, corrective measures are not very useful, because they require a large buffer and long buffering time, which is inappropriate for many applications.

18. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Mutual Exclusion The simplest way for two or more threads in a java program to interact is via an object, whose methods can be invoked by the set of threads. This shared objects state can be observed and modified by its methods. The simplest way for two or more threads in a java program to interact is via an object, whose methods can be invoked by the set of threads. This shared objects state can be observed and modified by its methods. Consequently, two threads can communicate by one thread writing the state of the shared object and the other thread reading that state. Consequently, two threads can communicate by one thread writing the state of the shared object and the other thread reading that state. Trouble !! Trouble !!

19. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Mutual Exclusion, Interference (1) The execution of the instructions from a set of threads can be interleaved in an arbitrary fashion. This interleaving can (and will ) result in incorrect updates to the state of a shared object. This is called interference. The execution of the instructions from a set of threads can be interleaved in an arbitrary fashion. This interleaving can (and will ) result in incorrect updates to the state of a shared object. This is called interference. To simplify the problem, the garden is designed so that people can enter but never leave. To simplify the problem, the garden is designed so that people can enter but never leave. Ornamental Garden

20. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Mutual Exclusion, Interference (2) The general solution to the problem of interference is to give methods that access a shared object with mutually exclusive access to that object. The general solution to the problem of interference is to give methods that access a shared object with mutually exclusive access to that object. This ensures that an update is not interupted by concurrent updates ! This ensures that an update is not interupted by concurrent updates !

21. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Mutual Exclusion, Interference (3) Concurrent activations of a method in Java can be made mutually exclusive by prefixing the method with the keyword synchronized. Concurrent activations of a method in Java can be made mutually exclusive by prefixing the method with the keyword synchronized.  Java associates a lock with every object. The Java compiler inserts code to acquire the lock before executing the body of a synchronized method, and code to release the lock before the method returns.  If an object has more than one method, all the methods should be synchronized in order to ensure mutually exclusive access to the state of the object. Ornamental Garden

22. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Mutual Exclusion, Interference (4) Once a thread has acquired the lock on an object by executing a synchronized method, that method may itself call another synchronized method from the same object without having to wait to acquire the lock again. Once a thread has acquired the lock on an object by executing a synchronized method, that method may itself call another synchronized method from the same object without having to wait to acquire the lock again. The lock counts how many times it has been acquired by the same thread and does not allow another thread to access the object until there has been an equivalent number of releases. The lock counts how many times it has been acquired by the same thread and does not allow another thread to access the object until there has been an equivalent number of releases. This locking strategy is termed recursive locking, since it permits recursive synchronized methods. This locking strategy is termed recursive locking, since it permits recursive synchronized methods.

23. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Mutual Exclusion, Interference (5) Example: public synchronized void increment(int n){ if(n > 0){ ++value; increment(n-1); } else return; } If locking in Java was not recursive, this example would cause a calling thread to be blocked forever, waiting to acquire a lock which it already holds. If locking in Java was not recursive, this example would cause a calling thread to be blocked forever, waiting to acquire a lock which it already holds.

24. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Monitors and Condition Synchronization (1)  A monitor encapsulates data, which can only be observed and modified by monitor access procedures. Only a single access procedure may be active at a time. An access procedure thus has mutually exclusive access to the data variables encapsulated in the monitor.  Monitors support condition synchronization in addition to ensuring that access to the data they encapsulate is mutually exclusive. Condition synchronization permits a monitor to block threads until a particular condition holds (such as a count becoming non-zero, a buffer becoming empty, or a new input becoming available). A monitor is simply represented in Java as a class that has synchronized methods.

25. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Monitors and Condition Synchronization (2) Each guarded action in the model of a monitor is implemented as a synchronized method which uses a while loop and wait() to implement the guard. Each guarded action in the model of a monitor is implemented as a synchronized method which uses a while loop and wait() to implement the guard. Changes in the state of the monitor are signalled to waiting threads using notify() or notifyAll(). Changes in the state of the monitor are signalled to waiting threads using notify() or notifyAll(). Car Park

26. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Semaphores (1) A semaphore S is an integer variable that can take only non-negative values. Once S has been given an initial value, the only operations permitted on S are up(S) and down(S). A semaphore S is an integer variable that can take only non-negative values. Once S has been given an initial value, the only operations permitted on S are up(S) and down(S). down(S) : when S > 0 do decrement S; up(S) : increment S; Semaphores are usually implemented by a blocking wait() as follows: down(S) : if S > 0 decrement S else block execution of calling process up(S) : if processes blocked on S awaken one of them else increment S

27. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Semaphores (2) class Buffer { protected Object[] buf; protected int in = 0; protected int out= 0; protected int count = 0; protected int size; public Buffer(int size) { this.size = size; buf = new Object[size]; } public synchronized void put(Object o) throws InterruptedException { while (count == size) wait(); buf[in] = o; ++count; in = (in+1) % size; notify(); } public synchronized Object get() throws InterruptedException { while (count == 0) wait(); Object o = buf[out]; buf[out] = null; --count; out = (out+1) % size; notify(); return o; }

28. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Semaphores (3) Implementations of semaphores usually manage processes, blocked on a semaphore by down(), as a First-In-First-Out queue (FIFO). The operation up() awakens the process at the head of the queue. Implementations of semaphores usually manage processes, blocked on a semaphore by down(), as a First-In-First-Out queue (FIFO). The operation up() awakens the process at the head of the queue. FIFO in out Semaphore

29. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Semaphores (4) The producer-consumer model with a bounded buffer is an example of a program that handles data items without altering them. The producer-consumer model with a bounded buffer is an example of a program that handles data items without altering them. In addition, the behaviour of the producer, the consumer, and the buffer itself are not affected by the value of the items they handle. In other words, they do not test the value of these data items. The behaviour is said to be data- independent. In addition, the behaviour of the producer, the consumer, and the buffer itself are not affected by the value of the items they handle. In other words, they do not test the value of these data items. The behaviour is said to be data- independent. BoundedBounded Buffer

30. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Deadlock (1) Deadlock occurs in a system when all its constituent processes are blocked; i.e., there are no eligible actions it can perform. Deadlock occurs in a system when all its constituent processes are blocked; i.e., there are no eligible actions it can perform. Four necessary and sufficient conditions for the occurrence of deadlock: Four necessary and sufficient conditions for the occurrence of deadlock:  Serially reusable resources : The processes involved share resources which they use under mutual exclusion  Incremental acquisition : Processes hold on to resources already allocated to them while waiting to acquire additional resources  No preemtion : Once acquired by a process, resources cannot be preemted (forcibly withdrawn), but are only released voluntarily  Wait-for-cycle : A circular chain (or cycle) of processes exists, such that each process holds a resource which its successor in the cycle is waiting to acquire

31. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Deadlock (2)

32. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Deadlock (3)

33. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Deadlock (4)

34. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Dining Philosophers The problem : each philosopher is modelled by the process The problem : each philosopher is modelled by the process Phil = (sit down →right.get →left.get →eat →left.put →right.put →rise) One solution to the dining philosophers problem depends on ensuring that a wait-for-cycle cannot exist. To do so, some assymmetry is introduced into the definition of a philosopher. One solution to the dining philosophers problem depends on ensuring that a wait-for-cycle cannot exist. To do so, some assymmetry is introduced into the definition of a philosopher. Odd numbered philosopher gets right fork first and even-numbered philosopher gets left fork first. Philosophers Philosophers (fixed)

35. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Racing conditions (1) Racing conditions can occur when two or more processes are reading some shared data and the final result depends on who runs exactly when. Racing conditions can occur when two or more processes are reading some shared data and the final result depends on who runs exactly when.

36. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Racing conditions (2)

37. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Racing conditions (3) So, put the thread to sleep ! So, put the thread to sleep !

38. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Threads (1) API

39. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Threads (2) Processes are not really parallel, but sequential ! Processes are not really parallel, but sequential !

40. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Threads (3)

41. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Threads (4) void interrupt()  First the method of this thread is invoked, which may cause a to be thrown  First the checkAccess() method of this thread is invoked, which may cause a SecurityException to be thrown  It the thread is blocked in an invokation of or methods of the Object class, then it will receive an.  It the thread is blocked in an invokation of wait(), join(), sleep() or methods of the Object class, then it will receive an InterruptedException.

42. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Threads (5) join()   Waits for this thread to die, which ensures sequential execution !

43. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Threads (6) Thread A joins thread B (waits for thread B to finish) but gets interrupted by thread C Thread A joins thread B (waits for thread B to finish) but gets interrupted by thread C

44. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Threads (7) Thread A interrupts itself, but the interrupt throws first an exception when the method is activated, since does not affect the interrupted status of the thread Thread A interrupts itself, but the interrupt throws first an exception when the sleep() method is activated, since isInterrupted() does not affect the interrupted status of the thread

45. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Threads (8) Here, the thread will enter everlasting sleep, since the method cleares the interrupted status of the thread. So, unless another thread comes and wakes the sleeping thread with another it will sleep forever : Here, the thread will enter everlasting sleep, since the interrupted() method cleares the interrupted status of the thread. So, unless another thread comes and wakes the sleeping thread with another interrupt(), it will sleep forever : The sleeping- beauty phenomenon

46. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Threads (9)

47. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Threads (10)

48. Lars A. Conrad-Hansen, Introduction to Multimedia Systems Threads (11) Solution: F.ex. use on one of the threads prior to acquiring locks Solution: F.ex. use join() on one of the threads prior to acquiring locks