Thursday, 18 January 2007 Multimedia Architectures Ines Färber and Alexander G.M. Hoffmann Advisor: Thorsten Karrer Post Desktop User Interfaces

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Motivation multimedia applications are of growing importance have to satisfy hard real-time and interaction requests  good architectures are required

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Introduction “multimedia architecture” is not clearly defined multimedia is an integration of multiple digital media into one big media at least one combined media has to be time- dependent

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Introduction (2) typical applications: video games, video conferencing systems, speech recognition, animation, internet dealing with creation, manipulation, presentation, storage or communication high requirements concerning real-time, low latency and precise synchronisation all influences have to be well-regulated

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Outline SAI - unifying approach to the distributed implementation of algorithms and their easy integration into complex systems Fran - Architecture for functional reactive animation generation Multimedia Information Service Enabling Conclusion

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 SAI - Software Architecture for Immersipresence SAI is a new software architecture model for designing, analyzing and implementing applications performing distributed, asynchronous parallel processing of generic data streams. (Francois 2005) general formalism high bandwidth low latency A. R.J. Francois. Software Architecture for Immersipresence, IMSC 2005

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 SAI (2) problems in actual complex systems  integration  resource intensive activity  countless unforeseen problems SAI  distributed implementation  easy integration

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 SAI (3) dataflow architectures  simple and intuitive design  support parallel and distributed processing problems:  limitation of efficiency and modelling power  shared data excess blackboards  but: not adapted to online and real-time processing

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 SAI – Fundamental principles explicit account of time both in data and processing models  Cells  Sources distinction between persistent and volatile data asynchronous parallelism

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 SAI – graphical approach data encapsulated in pulses cells=processing centres sources=persistent information repository persistent data (constant) volatile data (changing) VisualSAI

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Architecture conceptual level  set of sources and cells  inter-connections  descriptions of tasks logical level  cell: active and passive filters  source: structure of passive pulse

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 MuSA.RT – Music on the Spiral Array. Real-Time (ACM ‘03) These computations are defined by four independent streams: MIDI input and event processing tonal analysis rendering of the Spiral Array structures control device (game pad) input and camera manipulation

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Screenshot The active triad is indicated by a coloured triangle  red = major  blue = minor grey dots = actual key(s) green line = main melody

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Video

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 SAI - unifying approach to the distributed implementation of algorithms and their easy integration into complex systems Fran - Architecture for functional reactive animation generation Multimedia Information Service Enabling Conclusion Outline

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Problems of Creating Animations missing differentiation between presentation and modelling animation is conceptually continuous and parallel, whereas a PC works discretely and sequentially  visual authoring tool  change of language  Fran Conal Elliott and Paul Hudak. Functional reactive animation IFCP’97

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Fran (Functional Reactive Animation) implemented in Haskell

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Pascal vs. Haskell procedure quicksort(l,r : integer); var x,i,j,tmp : integer; begin if r>l then begin x:=a[l]; i:=l; j:=r+1; repeat repeat i:=i+1 until a[i]>=x; repeat j:=j-1 until a[j]<=x; tmp:=a[j]; a[j]:=a[i]; a[i]:=tmp; until j<=i; a[i]:=a[j]; a[j]:=a[l]; a[l]:=tmp; quicksort(l,j-1); quicksort(j+1,r) end end. quicksort [] = [] quicksort [x:xs] = quicksort [n | n =x]

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Fran (Functional Reactive Animation) implemented in Haskell afford a declarative instead of an imperative kind of programming modelling-oriented high level vocabulary of data types, functions and primitive graphic routines

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Fran: key concepts behaviours: time-varying, reactive values events: sets of arbitrarily complex conditions, carrying possibly rich information modelling approach is established by four features:  temporal modelling  event modelling  declarative modelling  polymorphic media implicit treatment of time method for event detection based on interval analysis

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Fran – examples behaviours upDownPat = moveXY 0 waggle pat pat = importBitmap "../Media/pat.bmp" leftRightCharlotte = moveXY wiggle 0 charlotte charlotte = importBitmap "../Media/charlotte.bmp" Conal Elliott and Paul Hudak.1997 Modeling Interactive 3D and Multimedia Animation with an Embedded Language

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Fran – examples behaviours charlottePatDance = leftRightCharlotte `over` upDownPat

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Fran – example event redBlue u = buttonMonitor u `over` withColor c circle where c = red `untilB` lbp u -=> blue Fran – example 3D teapot = stretch3 2 (importX "../Media/tpot2.x") redSpinningPot = turn3 zVector3 time ( withColorG red teapot)

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Outline SAI - unifying approach to the distributed implementation of algorithms and their easy integration into complex systems Fran - Architecture for functional reactive animation generation Multimedia Information Service Enabling Conclusion

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Multimedia Information Services Enabling 3-level Multimedia-Search Architecture extendable and scaleable multimedia information management systems extendable: easily adjusted to new data types scaleable: no performance lose while growing workload Erik Boertjes, W. Jonker, and J. Wijnands. Multimedia information services enabling: An architectural approach. ACM 2001

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Modules Communication and Data Transfer User Interfaces and Presentation Module Raw Data Servers Metadata Module User Profiles and Personalization Module

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Three – layer model

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Peggy Internet movie search retrieval service movie trailers are stored (rawdata) MPEG-7 format (metadata)

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 camera find free parking slot Feature - and Conceptual database PDOM search engine generates voice messages call-service Spot – a – spot SPOT “Turn left, the empty spot Is between the black car and the red car”

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Outline SAI - unifying approach to the distributed implementation of algorithms and their easy integration into complex systems Fran - Architecture for functional reactive animation generation Multimedia Information Service Enabling Conclusion

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Conclusion - SAI universal framework distributed asynchronous algorithms integration into complex systems graphical interface VisualSAI components, connectors and constraints

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Conclusion Fran embedded language simplifies the animation programming process via declarative programming techniques presentation details are left to the underlying implementation possible improvements: more features for 2D, 3D and sound, improving performance the underlying ideas of Fran also formed the basis of Microsoft’s DirectAnimation

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Conclusion MISE 5 modules 3 layer model with new concept of raw- and metadata databases universal search extendable and scaleable modify and adjustable

Multimedia Architecture Ines Färber and Alexander G.M. Hoffmann 18 January /35 Conclusion Multimedia Architectures no universal architecture various approaches different approaches  SAI is a graphical approach  FRAN is animation oriented  MISE handles Interactive Multimedia retrieval requests incompatibility if we exchange media

Thursday, 18 January 2007 Multimedia Architectures Ines Färber and Alexander G.M. Hoffmann Advisor: Thorsten Karrer Thanks for your attention