1. Rutgers - CIMC Automatic Manifestation of Composite Multimedia Objects Ahmed Gomaa Nabil Adam Vijay Atluri

2. Rutgers - CIMC 2 Outline Motivating Example Introduction to CMO Types of constraints Universal Access Framework –CMO specification –Petri net layer MMCTPN –SMIL layer Demo User Interaction Related work Future Work

3. Rutgers - CIMC 3 Current Situation What to do to select your business location? Currently available on NJ commerce commission and partners sites

4. Rutgers - CIMC 4 Current Situation Easy to get lost between the different links. Inserts the profile several times because dealing with different web sites ( state /realtors).

5. Rutgers - CIMC 5 Composite multimedia Object (CMO)

6. Rutgers - CIMC 6 Composite multimedia Object (CMO) –Comprised of different media component such as text, image, video, audio, shape files,... –Variety of relationships among components. –different types of constraints that must be adhered to when rendering it. (e.g., Fidelity >= 640*480). –Associated with each component is a set of parameters. (e.g., Fidelity = 800*600).

7. Rutgers - CIMC 7 Types of Constraints Synchronization constraints –Map must appears in sync.with Text 1 Fidelity Constraints –Map must be displayed at a resolution of at least 640*480 Spatial Constraints –Fly by is above Text Security Constraints –Sales representative audio and video are accessed by subscribers only.

8. Rutgers - CIMC 8 CMO Over Any Network To Any One with varying expertise, capabilities, and preferences To Any Device Universal Access Need to facilitate access to desired CMO according to the various user’s: Capabilities ( e.g, devices) Characteristics (e.g, expertise) Credentials (e.g, subscribers) When a subject requests a CMO, he may not be able to view the entire CMO. –For example, due to the limitations of his appliances, or due to lack of his credentials to satisfy the security requirements.

9. Rutgers - CIMC 9 Framework CMO layer –Comprises of the formal specification of the CMO, such that specifying each components in the CMO along with its associated constraints. Petri Net layer –Formal model that can describe,visualize and validate the CMO constraints. –Conceptual tool. SMIL layer –Implementation and automatic rendering purpose.

10. Rutgers - CIMC 10 CMO specification CMO = ({c 1...,c n }, S ): –Each c i is component of CMO –S is a set of synchronization,spatial,fidelity, and security constraints. Each component c i in turn,is a tuple c =(ml,fd,pd): –ml represents the modality –fd represents the fidelity –pd represents the playback duration

11. Rutgers - CIMC 11 Constraints Synchronization Constraints –We only need 3 from the 7 temporal relations –Meets: video meets image –Sync: video sync text Before: video image video text  video1 video2

12. Rutgers - CIMC 12 Constraints Fidelity Constraints –A set of fidelity constraints for modality ml. –FD (ml )is set of fidelity ranges fd –for example >= 640 * 480 is to specify that the resolution be at least 640 *480 to view the object. Spatial Constraints –Where to place the components on the screen

13. Rutgers - CIMC 13 Constraints Security Constraints Let CR = {cr 1, cr 2... }denote the set of all distinct credentials relevant to the multimedia object. Each subject S possesses set of credentials denoted as s CR A security constraint can be: –(1)Each component c i of the multimedia object is associated with set of credentials,denoted c i CR. –(2)A set of components (c i c j,...)is associated with a set of credentials,denoted (c i c j,...) CR. E.g., video2 ( v 2 ) need subscription credentials so, C i CR = V 2 subscription

14. Rutgers - CIMC 14 Petri net layer Token Place –A marking is an arrangement of tokens in places, representing state Arc –An arc connects a place and a transition Transition –Inter-object sync. –A transition is enabled if there is at least one token at each of its input places Firing rule –the activity may take place if transition enabled. t1 t0 p0 t2 p1 p2 p3 p4

15. Rutgers - CIMC 15 Colored Petri Nets (CPN) Before firing: After firing with substitution {a|x, b|y, c|z}: Execution 1.p1 loses 2 tokens of x; 2.p2 loses and ; 3.p3 gets ; 4.p4 get e (constant). Initial marking: 1.p1 has 4 tokens, 2 a’s and 2 d’s; 2.p2 has 6 tokens,,, ; 3.p3 and p4 has no tokens.

16. Rutgers - CIMC 16 MMCTPN Model for a CMO plan MMCTPN (Multimedia color-time Petri-Net) consists of –Color-time Petri-Net –Tokens: Color sets that represent: Types of Fidelity (e.g.,the resolution of the MM component) –Absence of fidelity capabilities is represented as Holes Types of credentials (e.g., credentials for researchers) –Absence of credentials of the subject requesting the object is represented as Holes Types of Modality (e.g., image, video,text, audio) –Absence of modality capabilities is represented as Holes –Places: Includes the length of time the multimedia component is played. A color set that is needed to activate/ play the component. Null place in each subMMCTPN (A part of the net between two subsequent transitions)

17. Rutgers - CIMC 17 MMCTPN Credentials and capabilities are represented as tokens and absence of them are represented as holes. Author rules are represented as doted holes. The initial place is marked with these tokens to start with. Tokens are said to be available if it remains in a place for their specified duration. Transitions are enabled once any token or hole are available in all input places. t t Before Firing Pa, dur:10 Pb, dur:15 After Firing Pa, dur:10 Pb, dur:15 t = 0

18. Rutgers - CIMC 18 MMCTPN – Object Manifestation Dur = 10 S A A C C Dur = 0 S B B

19. Rutgers - CIMC 19 P2P2 P1P1 PsPs PfPf P5P5 P3P3 nul l P9P9 P6P6 P4P4 P8P8 P7P7 dur :40s dur :15s dur :10s dur :15s CMO Plan

20. Rutgers - CIMC 20 MMCTPN execution dur :40s P2P2 P1P1 PsPs PfPf P5P5 P3P3 nul l P9P9 P6P6 P4P4 P8P8 P7P7 dur :40s dur :15s dur :10s dur :15s

21. Rutgers - CIMC 21 Modified CMO Plan P2P2 PsPs PfPf P5P5 nul l P9P9 P6P6 P4P4 dur :40s dur :10s dur :15s

22. Rutgers - CIMC 22 Petri Net To SMIL Equals ( SYNC): |------------| image |------------| text |------------| Audio 30s After ( Before): |----------| 6 sec |---------| 4sec |--------------------------| 11 seconds

23. Rutgers - CIMC 23 SMIL layer SMIL-based rich multimedia presentations integrate several types of media. SMIL is XML-based, thus flexible and extensible. SMIL is W3C supported, thus enjoys cross industry support. SMIL is supported by readily available tools and parsers.

24. Rutgers - CIMC 24 Demo http://cimic.rutgers.edu/~ahgomaa/ua/presentations/demo3/egov.smil http://cimic.rutgers.edu/~ahgomaa/ua/presentations/demo3/egov-adj.smil

25. Rutgers - CIMC 25 SMIL 2.0 Implementation we have adopted the security model and the language (XACL) in specifying the security constraints on the multimedia object. We have used the XACL visual tool,tool from IBM XML Security Suite.

26. Rutgers - CIMC 26 Incorporating User interactivity Identify and model the different types of user interactions. Make sure that different constraints are still valid. Visualize and analyze the effect of the user interaction on different constraints.

27. Rutgers - CIMC 27 User interaction categories Temporal Interaction: –continuous component, ( ex. video ) ''play'', '‘fast forward'' –non-continuous component(ex. image) "begin", "end", "pause". Spatial interaction: "move “, "Zoom",” maximize”, "change font". Undo – Redo The user may want to interact with the CMO as one object, e.g., pause, begin,…

28. Rutgers - CIMC 28 User Interaction example If a user wants to stop the fly by video, what will be the effect on the following text? (rendering) –Appear immediately after stop. –Appear after a defined amount of time. If a user wants to maximize the map, what will be the effect on rendering the following image? ( conflict) –Rendered behind the map. –Rendered after another action. The CMO author need to: –Visualize –Analyze

29. Rutgers - CIMC 29 User Interaction Problems How to model different types of user interaction on the composite multimedia object plan? How can the author visualize and analyze the impact of potential user interaction on the rendering of the CMO? How to detect potential conflicts between different types of constraints? How to resolve different conflicts between constraints?

30. Rutgers - CIMC 30 CMO constraints Uniform Model Conflict resolution PETRI-NET Modeling for validation and analysis Implementation with user/web friendly with flexibility and interoperability (SMIL 2.0)

31. Rutgers - CIMC 31 Related Work Multimedia object rendering Security User interaction

32. Rutgers - CIMC 32 Related Work Multimedia Object Rendering: Adam and Atluri et al.(2001) Presented the Universal Access problem and the Oblet approach. –Uses Petri Net to present temporal, spatial, modality and fidelity constraints. Bertino et al.(2000) present a system called MPGS - Multimedia Presentation Generator System. –Enables specification of synchronization and spatial constraints –Capable of analyzing the consistency among these two types of constraints. None of those papers addressed the security constraints in a CMO

33. Rutgers - CIMC 33 Related Work Security: Damiani et al.(2000),Bertino et Al.(2000) and Kudo et al (2000) addressed access control models in XML. We adopted Kudo et al. approach in presenting a new Petri Net model that incorporate the security constraints in the CMO.

34. Rutgers - CIMC 34 Related Work User interaction: User interaction research within CMO may be divided into three categories: –Formal modeling –Programming approach –Statistical techniques Formal modeling, using Petri net : –Song et al.(1996) presented Timed Petri Net (TPN). –Guan et al. (1998) presented a distributed object composition Petri net (DOCPN). –Prabhakaran et al.(1993) presented a dynamic timed Petri nets (DTPN) model. –All models are restrictive in types of interactions. –They only deal with the synchronization constraints.

35. Rutgers - CIMC 35 Related Work Programming approach –Partially support user interactions for online CMO. –LimSee(2002), Grins (2002), Yang (2001). –All of which based on time line events –which leads to dead times if the CMO is adaptable. Statistical techniques –Predict online user interactions –Hollfelder et al. (2000) where they model the user behavior as a Continuous Time Markov chain (CTMC). –Boll et al.(2001) implementing adaptive streaming of MPEG videos for interactive internet applications to support jumping to bookmarks within the MPEG

36. Rutgers - CIMC 36 Future Work Enforcing Security Constraints. Secure the plan on the client side ( certificates, relational transducer). Allowing User Interaction. Conflict Identification and Resolution Strategy.