1 Constraints for Multimedia Presentation Generation Joost Geurts, Multimedia and Human-Computer Interaction CWI Amsterdam

2 Talk overview Generating multimedia automatically Cuypers multimedia generation engine Multimedia and constraints –Quantitative constraints –Qualitative constraints Cuypers demo Conclusion, future directions

3 Multimedia Presentation –Image, Text, Video, Audio –Based on Temporal and Spatial Synchronization Multimedia Document –SMIL, SVG, HTML –WYSIWYG –Static Content Problem: Dynamic Content

4 Generating adaptive multimedia Content –Large multimedia database System profile –PC, PDA, WAP Network profile –Modem, Gigabit User profile –Language, Interests, Abilities, Preferences Too costly to author manually

5 Cuypers multimedia generation engine

6 Cuypers is based on –media independent presentation abstractions –transformation rules with built-in backtracking and constraint solving

7 Semantic structure Author does not specify complete presentation… …but only rhetoric relations

8 Communicative Devices …rhetoric relations are than transformed into presentation independent communicative devices…

9 Automatic multimedia generation Designer does not specify complete presentation… …but only specifies requirements System automatically finds a solution which meets requirements How should the requirements be specified? –Declarative constraints

10 Constraint satisfaction Constraints occur often in our daily lives –Agenda, Travelling, Shopping Constraint paradigm for Problem Solving –Declarative Used for problems with: –Many variables –Large domains –Based on domain reduction paradigm

11 Intelligent reduction of possible values X  {1,2,3,4,5}, Y  {1,2} ; X  Y X  {1,2}, Y  {1,2} ; X  Y

12 Traditional use of constraints Quantitative constraints –Integer domain –Reduction by arithmetic relations Greater than (>) Less than (<) Equals (=) –Example (x < y ; x  [0..10], y  [5..10] ) (x  + y  = z 3, x = u  + 1 ; x  , y  , z  , u   )

13 Solving a Constraint Satisfaction Problem Problem SEND + MORE = MONEY Modeling 1000 x S x E + 10 x N + D x M x O + 10 x R + E = x M x O x N + 10 x E + Y Domain reduction / Search Solution S=9, E=5, N=6, D=7, M=1, O=0, R=8, Y=2

14 Quantitative Constraints in Multimedia …Communicative devices generate constraint-graph which the system tries to satisfy…

15 Drawbacks of quantitative constraints Too many (trivial) solutions that differ by: –1 pixel position, or –1 milliseconds in timing Not sufficiently expressive cannot specify “no overlap” constraint Too low level A.X2  B.X1

16 Allen’s 13 temporal relations Allen’s relations are used for both spatial and temporal lay-out

17 Solution: qualitative constraints For non-typical domains –Boolean, –Three valued logics, –Allen’s relation Advantages for Multimedia generation: –More intuitive –More expressive –Smaller domains

18 Domain Reduction Rules Inverse A before B  B after A A equal B  B equal A Transitive A before B, B before C  A before C A overlaps B, B during C  A overlap C or A during C or A starts C Equals A overlap C, A [o,d,s] C  A overlap C

19 Qualitative Constraints …Qualitative solutions translate automatically to lower level quantitative constraints…

20 New problem: What if constraints are insoluble? Combine Prolog unification and backtracking with constraint solving Use Prolog rules to generate constraints Backtrack when constraints are insoluble Solution: Constraint Logic Programming

21 Cuypers generation engine Multiple layers: –Communicative devices generate constraints –Qualitative constraints translate to quantitative constraints –Solution of both constraints provides sufficient information for final presentation

22 Cuypers demo: scenario Client Server Client User is interested in Rembrandt and wants to know about about the “chiaroscuro” technique Query database Generate constraints according to: –System profile –User profile –Network profile Solve constraints / revise constraints Generate SMIL presentation PlayPlay presentation

23 Conclusions Quantitative constraints are insufficient for automatic multimedia presentation generation. Also need Qualitative constraints to allow intuitive and effective high level specification, and Backtracking for revising specific constraints which otherwise cause the entire set to fail

24 Discussion Labeling –Choice of candidate variable –Choice of candidate value Transitive Reasoning Rule –Infer implicit relations –Redundant Allen’s Relations –Not very well suited for generating MM –Non interactive

25 Future directions Best-first instead of depth-first –Choose “best” among possible solutions –Needs evaluation criteria Improve knowledge management –Make design knowledge declarative and explicit –Preserve metadata in final presentation –Use standardized and reusable profiles

26 Thank you

27 Need to make trade-offs Semantics –Convey message Aesthetics –Clear / nice layout Resources –Screen size, bandwidth Dimension may result in conflicting goals

28 Quantitative Constraints % csp(+Ids, -Boxes) csp([IdA,IdB],[box(IdA,[x1:AX1, …]), box(IdB,[x1:BX1,…])]) :- % get values maxX(MaxX), maxY(MaxY), height(IdA,HeightA), widtht(IdA,WidthA), … % define domains [AX1,AX2,BX1,BX2]::[0..MaxX], [AY1,AY2,BY1,BY2]::[0..MaxY], % set width & height AX2 – AX1 #= WidthA, AY2 – AY1 #= HeightA, … % constraints AX2 #< BX1,% A left-of B AY1 #= BY1,% A top-align B, … true.

29 Multimedia and Constraints Constraint Logic Programming –Domain reduction –Backtracking –Unification (matching rules) Qualitative Constraints –Non-integer domain –Allen’s 13 temporal interval relations in three dimensions

30 Qualitative Constraints Example: Two images, A,B A left or right of B A not above or below B

31 Qualitative Constraints % csp(+Ids, -Graph) csp([IdA, Idb], [edge(IdA,IdB,x,NoOverlap),…]) :- % define domains NoOverlap :: [b,b-,m,m-], Overlap :: [d,d-,s,s-,f,f-,e], % constraints edge(IdA,IdB,x,NoOverlap),% B not-overlap A edge(IdA,IdB,y,Overlap),% B overlap A true.

32 Qualitative Constraints Reasoning –Inverse: edge(A,B,D,Value) inverse(Value,RValue),edge(B,A,D,RValue). –Equality edge(A,B,D,V1), edge(A,B,D,V2) => V1 #= V2 –Transitive edge(A,B,D,VAB), edge(B,C,D,VBC) => tr(VAB,VBC,VAC), % rule generation algorithm edge(A,C,D,VAC). Translation rules to quantitative domain edge(A,B,D,b) => node(A,D/2,V2), node(B,D/1,V1) V1 #< V2.

33 Problems in generating multimedia Text documents are flexible –Add page, scrollbar, –Template models –Wrap text around images Multimedia documents are less flexible –No pages or scrollbars, no line-breaking or hyphenation –Not based on text-flow –Feedback needed Linear process model does not work for multimedia

34 Quantitative Constraints Example: Two images, A,B A left-of B A top-align B

35 Cuypers generation engine Rhetoric/Semantic –Sequence, Example Communicative devices –Bookshelf, Slideshow Qualitative Constraints –A before B Quantitative Constraints –A.X2 < B.X1 Presentation –SMIL