Multimedia Presentation and Delivery Ch.13 Principles of Multimedia Database Systems. V.S. Subrahmanian, 1998

MM Presentation & Delivery2 Presentation & Retrieval n When creating a multimedia presentation, three basic questions must be answered What objects should be included in the presentation ?What objects should be included in the presentation ? When should these objects to be presented to the user ?When should these objects to be presented to the user ? Where should the objects appear on the screen ?Where should the objects appear on the screen ? n These questions must be answered by the individual who creates the presentation (i.e., author) n Once these have been specified, the Presentation Engine must generate a Retrieval Plan like: when the objects need to be presentedwhen the objects need to be presented bandwidth limitation on the networkbandwidth limitation on the network resource (load, buffer) limitation on the server & clientresource (load, buffer) limitation on the server & client mismatches between delivery rate and client consumption rate.mismatches between delivery rate and client consumption rate.

MM Presentation & Delivery3 Interaction Between Presentation & Retrieval Schedules

MM Presentation & Delivery4 Creating a Presentation n Suppose objects o 1,...,o n. are to be presented. n Temporal constraints specify how the objects should be laid out in time. Example: The presentation of objects 1 and 2 must start at the same time.The presentation of objects 1 and 2 must start at the same time. The termination of objects 2 and 3 must occur at the same time.The termination of objects 2 and 3 must occur at the same time. Object 3 must start at the time object 1 ends.Object 3 must start at the time object 1 ends. n Spatial constraints specify how the objects should be laid out in space. Example: Object 1 must be to the left of object 2.Object 1 must be to the left of object 2. Object 1 must be above object 3.Object 1 must be above object 3.

MM Presentation & Delivery5 Seven Possible Temporal Relationships

MM Presentation & Delivery6 Constraint Language n Constants: Every integer (positive and negative) is a constant. n Variables: Associated with each o i are two integer variables, s i (denoting the "start" of o i ) and e i (denoting the "end" of o i ) n Elementary Terms: Elementary terms are defined inductively as follows: Every constant is an elementary termEvery constant is an elementary term Every variable is an elementary termEvery variable is an elementary term n Difference Constraint: If tl, t2 are elementary terms, and c is a Constant, then tl - t2 < Ctl - t2 < C is a difference constraint

MM Presentation & Delivery7 Examples of Difference Constraints n e1 - s1  10. This constraint says that object o 1 must end within 10 time units of the time its presentation starts. n s2 - e1  0; e1 - s2  0 These two constraints jointly state that object o 1 's presentation starts as soon as object o 1 's presentation ends, i.e., s2 - e1 = 0 n s2 - e1  3 This constraint says that object o 2 's presentation starts within 3 time units of the end of the presentation of object o 1.

MM Presentation & Delivery8 A Definition of Temporal Presentation n A temporal presentation is a pair TP = (O, DC) where O is a finite set of objects, and DC is a finite set of difference constraints in the constraint language generated by O. n Suppose DC is a set of difference constraints over O of n virtual objects. A solution of DC is an assignment of integers to each of the variables, s1,... sn, e1,... en, which makes all the constraints true. n E.g., n Sets of constraints can have no, one, multiple solutions.

MM Presentation & Delivery9 n Possible solutions: n A temporal presentation TP = (O, DC) is feasible iff the set, DC, of difference constraints, has a solution , I.e., a feasible schedule for TP.

MM Presentation & Delivery10 Definitions about the schedule n The start and end of , denoted start(  ) and end(  ), are defined to be: start(  ) = min({  (si) I1  i  n}).start(  ) = min({  (si) I1  i  n}). end(  ) = max({  (ei) I1  i  n}).end(  ) = max({  (ei) I1  i  n}). n We are assuming that the constraints {si - ei  0|1  i  n} are included in DC, i.e., a default DC. n For our example:

MM Presentation & Delivery11 How to create a presentation schedule? n We are facing a constraint satisfaction problem which can be solved by linear programming algorithms like simplex method. n There is another classic algorithm called the Bellman Ford Algorithm which takes as input, a set S of difference constraints.takes as input, a set S of difference constraints. Convert S to a graph.Convert S to a graph. S has a solution iff the graph has no negative cycles.S has a solution iff the graph has no negative cycles. A negative cycle is a cycle whose edges sum up to a negative number.A negative cycle is a cycle whose edges sum up to a negative number.

MM Presentation & Delivery12 An example: Given a set S of DC’s: Form a graph for S:

MM Presentation & Delivery13 An example: (cont.) n Compute the shortest/cheapest paths n If no negative cycles (i.e., costs are not negative), we have

MM Presentation & Delivery14 More about Bellman-Ford Algorithm n Let the cost of a path be the sum of the costs of edges along the path. A cycle in graph G is a sequence of nodes v 1, …, v k such that v 1 = v k and  1  j < k, (v j, v j+1 ) is an edge in the graph. A cycle in graph G is a sequence of nodes v 1, …, v k such that v 1 = v k and  1  j < k, (v j, v j+1 ) is an edge in the graph. v 1, …, v k is said to be a negative cycle in G iff it (total) cost is a negative number v 1, …, v k is said to be a negative cycle in G iff it (total) cost is a negative number n A set of difference constraints DC has no solution iff G DC has a negative cycle. n Suppose a graph has no negative cycles. Then the cost of the shortest/cheapest path from the start node to that node (si or ei) directly allows us to get a solution to the original set of difference constraints.

MM Presentation & Delivery15 Algorithm Implementation The Bellman Ford algorithm associates with each node N in G DC, the following two fields: Bestval : This specifies the cheapest path from the start node to the node N, that has been discovered thus far Bestval : This specifies the cheapest path from the start node to the node N, that has been discovered thus far Bestpar : Bestpar(N) specifies the immediate predecessor of node N along the best path from the start node to node N, that we have found thus far Bestpar : Bestpar(N) specifies the immediate predecessor of node N along the best path from the start node to node N, that we have found thus far n Then, n Execute Algorithm 13.3 (which includes Algorithm 13.1, 13.2) in the text.

MM Presentation & Delivery16 A Broader Issue of MM Presentation: Multimedia Synchronization n Attempts to ensure the desired temporal ordering among a set of MM objects in a multimedia scenario, where “MM scenario” denotes the temporal semantics of a MM session n The goals of MM synchronization mechanism are to preserve the playback continuity of media frames within single continuous media and the desired temporal dependencies among multiple related data objects, given that user interactions are allowed. n Temporal dependency can be either implicit (lip synchronization) or explicit (a slide show) n MM synchronization is an important set of QoS parameters n MM synch. differs from traditional synch. mechanisms such as monitor and semaphore in that it imposes real-time constraints (deadlines) on events, and differs from real-time systems in that such deadlines are predetermined, periodic, and soft

MM Presentation & Delivery17 Three Levels of MM Synchronization Intramedia synchronization n also known as intrastream or serial synchronization n preserves the temporal relationship between consecutive frames and the continuity of playback within a single media stream n the basic idea is to smooth the delay jitters experienced in the storage system, network, and host system Intermedia synchronization n also known as interstream or parallel synchronization n coordinates the different media streams to achieve the desired temporal relationships among them

MM Presentation & Delivery18 Intra and Intermedia Synchronization Requirements

MM Presentation & Delivery19 Three Levels of MM Synchronization (cont.) Interparty synchronization n also known as event synchronization n maintains the intramedia and/or intermedia synch. among different participants at distributed locations n e.g., remote learning and video conferencing

MM Presentation & Delivery20 Temporal Synchronization Mechanisms (in a distributed environment) n Synchronization mechanism renders the synchronization specification into a set of playback schedules and retrieval schedule and enforces the playback deadlines by taking into considerations of the network characteristics n e.g. to display a MM object at time s (i.e., the playback deadline), the presentation site sends a request to the corresponding media server at request time r, which is R time units prior to s. R(=s-r)is the response time to account for possible experienced end-to-end delays and packet losses.

MM Presentation & Delivery21 Temporal Synchronization Mechanisms: delay For discrete object, the playback deadline can be met when R the response time is larger than the total delay (server d ser + network d net + host delay d host ) For discrete object, the playback deadline can be met when R the response time is larger than the total delay (server d ser + network d net + host delay d host ) e.g. to retrieve an image of size 1.3MB from a remote site with d ser =50  s connected by a link of 100Mbps bandwidth, we should have the response time e.g. to retrieve an image of size 1.3MB from a remote site with d ser =50  s connected by a link of 100Mbps bandwidth, we should have the response time R  50  s+1.3x8Mbits/100Mbits+ d host d host is normally negligible. d host is normally negligible.

MM Presentation & Delivery22 Temporal Synchronization Mechanisms: delay jitter For a continuous media object, each media frame/packet may experience different delay and such intrastream synchronization can be achieved by delaying the the playback deadline of the first media frame by d syn, i.e., For a continuous media object, each media frame/packet may experience different delay and such intrastream synchronization can be achieved by delaying the the playback deadline of the first media frame by d syn, i.e., which is the difference between the maximum and minimum network delays of stream i. Therefore, the response time should be at least (server d ser + network d net + intramedia synchronization d syn + host delay d host ) Therefore, the response time should be at least (server d ser + network d net + intramedia synchronization d syn + host delay d host ) e.g., to retrieve a video clip of fixed frame size 40kB from a remote site with d ser =50  s connected by a link of 100Mbps bandwidth and 10  s intramedia, we should have response time e.g., to retrieve a video clip of fixed frame size 40kB from a remote site with d ser =50  s connected by a link of 100Mbps bandwidth and 10  s intramedia, we should have response time R  50  s + 40x8kbits/100Mbits + 10  s + d host

MM Synchronization23 More on Synchronization Requirements n Synchronization can be measured by four major parameters: delay, delay jitter (intramedia skew), intermedia skew, and tolerable error n Delay measures end-to-end delay or response time n Delay jitter measures delay variations of continuous media Intermedia skew measures the time shift between related media from the desired temporal relationship, e.g., lip synchronization requires intermedia skew within ±80ms, stereo audio (audio- audio synchronization) requires ±11  s, video-text synchronization requires ±240ms Intermedia skew measures the time shift between related media from the desired temporal relationship, e.g., lip synchronization requires intermedia skew within ±80ms, stereo audio (audio- audio synchronization) requires ±11  s, video-text synchronization requires ±240ms n Tolerable error measures the allowable bit error rate and packet error rate for a particular stream in a particular application, e.g., bit error rate should be smaller than 0.1 (for uncompressed voice), 0.01 (for uncompressed TV-quality video), (for uncompressed image)

MM Synchronization24 Synchronization Specifications & Mechanisms n Synchronization can be refer to time (our focus here), space, and content. n General MM synchronization can be divided into two levels: (temporal) synchronization specification (upper level)(temporal) synchronization specification (upper level) (temporal) synchronization mechanism (lower level) !(temporal) synchronization mechanism (lower level) ! n Synchronization specification models the temporal relationships among the set of related abstract MM objects (e.g., video clip, text) of a presentation scenario n Synchronization mechanism translates the temporal synchronization specification into the desired presentation scheduling sequences and enforces the playback deadlines despite the indeterministic delays due to the server, the network and the host system

MM Synchronization25 Specification by Scripts (Language-Based Approach) n A script uses a structural language (concurrent programming language) to specify the temporal relationships, e.g., Communication Sequential Processing (CSP) scheme n Scripts are powerful and can be used to specify all types of relationships and user interactions Script slide-show {Parallel Display picture1.jpg for 5 seconds Display audio1.au for 5 seconds Parallel Display picture2.jpg for 6 seconds Display audio2.au for 7 seconds Parallel Display picture3.jpg for 10 seconds Display audio3.au for 10 seconds }

MM Synchronization26 Specification by Scripts (cont.) n The first line specifies the name of the script Parallel, Display, For, and Seconds are keywords of the script Parallel, Display, For, and Seconds are keywords of the script n A parallel structure completes when all statements within it complete; so in the second parallel structure, the audio continues to play for 1 sec after the display of picture 2 has ended (A starts B). n Advantage of using scripts: they can be directly lead to implementation n Disadvantage: they are hard to conceptually visualize and are difficult to verify

MM Synchronization27 Specification by Time Lines n In this specification, the presentation time of each stream is specified relative to a common clock. n Streams are treated independently. n Synchronization is achieved by ensuring that each stream is presented at its specified time relative to the common clock n Advantages: simple and easy to use and implement because streams are treated independently during execution n Disadvantages: cannot specify user interaction or streams with unpredictable duration refer to Fig.9.2, p.279 of reference material

MM Synchronization28 Specification by Graphs n Synchronization can be specified or modeled by graph, in particular, object composition petri nets (OCPN) n The basic idea is to represent various components of MM objects as places and describe their inter-relations in the form of transitions n It is capable to explicitly capture all the necessary temporal relations, and to provide simulation of presentation in both the forward and reverse directions (better user interaction) n Each place in the model represents the play-out of a MM object while transitions represent synchronization points n It specifies exact presentation time play-out semantics, which is useful in real-time scheduling

MM Synchronization29 Specification by OCPN n The slide show example: n Another example: