1. Thursday, 18 January 2007 Multimedia Architectures Ines Färber and Alexander G.M. Hoffmann Advisor: Thorsten Karrer Post Desktop User Interfaces

2. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 2/35 Motivation multimedia applications are of growing importance have to satisfy hard real-time and interaction requests  good architectures are required

3. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 3/35 Introduction “multimedia architecture” is not clearly defined multimedia is an integration of multiple digital media into one big media at least one combined media has to be time- dependent

4. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 4/35 Introduction (2) typical applications: video games, video conferencing systems, speech recognition, animation, internet dealing with creation, manipulation, presentation, storage or communication high requirements concerning real-time, low latency and precise synchronisation all influences have to be well-regulated

5. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 5/35 Outline SAI - unifying approach to the distributed implementation of algorithms and their easy integration into complex systems Fran - Architecture for functional reactive animation generation Multimedia Information Service Enabling Conclusion

6. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 6/35 SAI - Software Architecture for Immersipresence SAI is a new software architecture model for designing, analyzing and implementing applications performing distributed, asynchronous parallel processing of generic data streams. (Francois 2005) general formalism high bandwidth low latency A. R.J. Francois. Software Architecture for Immersipresence, IMSC 2005

7. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 7/35 SAI (2) problems in actual complex systems  integration  resource intensive activity  countless unforeseen problems SAI  distributed implementation  easy integration

8. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 8/35 SAI (3) dataflow architectures  simple and intuitive design  support parallel and distributed processing problems:  limitation of efficiency and modelling power  shared data excess blackboards  but: not adapted to online and real-time processing

9. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 9/35 SAI – Fundamental principles explicit account of time both in data and processing models  Cells  Sources distinction between persistent and volatile data asynchronous parallelism

10. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 10/35 SAI – graphical approach data encapsulated in pulses cells=processing centres sources=persistent information repository persistent data (constant) volatile data (changing) VisualSAI

11. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 11/35 Architecture conceptual level  set of sources and cells  inter-connections  descriptions of tasks logical level  cell: active and passive filters  source: structure of passive pulse

12. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 12/35 MuSA.RT – Music on the Spiral Array. Real-Time (ACM ‘03) These computations are defined by four independent streams: MIDI input and event processing tonal analysis rendering of the Spiral Array structures control device (game pad) input and camera manipulation

13. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 13/35 Screenshot The active triad is indicated by a coloured triangle  red = major  blue = minor grey dots = actual key(s) green line = main melody

14. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 14/35 Video http://www-rcf.usc.edu/~mucoaco/

15. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 15/35 SAI - unifying approach to the distributed implementation of algorithms and their easy integration into complex systems Fran - Architecture for functional reactive animation generation Multimedia Information Service Enabling Conclusion Outline

16. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 16/35 Problems of Creating Animations missing differentiation between presentation and modelling animation is conceptually continuous and parallel, whereas a PC works discretely and sequentially  visual authoring tool  change of language  Fran Conal Elliott and Paul Hudak. Functional reactive animation IFCP’97

17. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 17/35 Fran (Functional Reactive Animation) implemented in Haskell

18. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 18/35 Pascal vs. Haskell procedure quicksort(l,r : integer); var x,i,j,tmp : integer; begin if r>l then begin x:=a[l]; i:=l; j:=r+1; repeat repeat i:=i+1 until a[i]>=x; repeat j:=j-1 until a[j]<=x; tmp:=a[j]; a[j]:=a[i]; a[i]:=tmp; until j<=i; a[i]:=a[j]; a[j]:=a[l]; a[l]:=tmp; quicksort(l,j-1); quicksort(j+1,r) end end. quicksort [] = [] quicksort [x:xs] = quicksort [n | n =x]

19. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 19/35 Fran (Functional Reactive Animation) implemented in Haskell afford a declarative instead of an imperative kind of programming modelling-oriented high level vocabulary of data types, functions and primitive graphic routines

20. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 20/35 Fran: key concepts behaviours: time-varying, reactive values events: sets of arbitrarily complex conditions, carrying possibly rich information modelling approach is established by four features:  temporal modelling  event modelling  declarative modelling  polymorphic media implicit treatment of time method for event detection based on interval analysis

21. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 21/35 Fran – examples behaviours upDownPat = moveXY 0 waggle pat pat = importBitmap "../Media/pat.bmp" leftRightCharlotte = moveXY wiggle 0 charlotte charlotte = importBitmap "../Media/charlotte.bmp" Conal Elliott and Paul Hudak.1997 Modeling Interactive 3D and Multimedia Animation with an Embedded Language

22. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 22/35 Fran – examples behaviours charlottePatDance = leftRightCharlotte `over` upDownPat

23. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 23/35 Fran – example event redBlue u = buttonMonitor u `over` withColor c circle where c = red `untilB` lbp u -=> blue Fran – example 3D teapot = stretch3 2 (importX "../Media/tpot2.x") redSpinningPot = turn3 zVector3 time ( withColorG red teapot)

24. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 24/35 Outline SAI - unifying approach to the distributed implementation of algorithms and their easy integration into complex systems Fran - Architecture for functional reactive animation generation Multimedia Information Service Enabling Conclusion

25. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 25/35 Multimedia Information Services Enabling 3-level Multimedia-Search Architecture extendable and scaleable multimedia information management systems extendable: easily adjusted to new data types scaleable: no performance lose while growing workload Erik Boertjes, W. Jonker, and J. Wijnands. Multimedia information services enabling: An architectural approach. ACM 2001

26. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 26/35 Modules Communication and Data Transfer User Interfaces and Presentation Module Raw Data Servers Metadata Module User Profiles and Personalization Module

27. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 27/35 Three – layer model

28. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 28/35 Peggy Internet movie search retrieval service movie trailers are stored (rawdata) MPEG-7 format (metadata)

29. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 29/35 camera find free parking slot Feature - and Conceptual database PDOM search engine generates voice messages call-service Spot – a – spot 0900 - SPOT “Turn left, the empty spot Is between the black car and the red car”

30. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 30/35 Outline SAI - unifying approach to the distributed implementation of algorithms and their easy integration into complex systems Fran - Architecture for functional reactive animation generation Multimedia Information Service Enabling Conclusion

31. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 31/35 Conclusion - SAI universal framework distributed asynchronous algorithms integration into complex systems graphical interface VisualSAI components, connectors and constraints

32. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 32/35 Conclusion Fran embedded language simplifies the animation programming process via declarative programming techniques presentation details are left to the underlying implementation possible improvements: more features for 2D, 3D and sound, improving performance the underlying ideas of Fran also formed the basis of Microsoft’s DirectAnimation

33. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 33/35 Conclusion MISE 5 modules 3 layer model with new concept of raw- and metadata databases universal search extendable and scaleable modify and adjustable

34. Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January 2007 34/35 Conclusion Multimedia Architectures no universal architecture various approaches different approaches  SAI is a graphical approach  FRAN is animation oriented  MISE handles Interactive Multimedia retrieval requests incompatibility if we exchange media

35. Thursday, 18 January 2007 Multimedia Architectures Ines Färber and Alexander G.M. Hoffmann Advisor: Thorsten Karrer Thanks for your attention