1 Application-specific constraints for multimedia presentation generation Joost Geurts, Jacco van Ossenbruggen and Lynda Hardman CWI Amsterdam

2 Talk overview Generating multimedia automatically Cuypers multimedia generation engine Multimedia and constraints –Quantitative constraints –Qualitative constraints Cuypers demo Conclusion, future directions

3 Generating adaptive multimedia Content –Large multimedia database System profile –PC, PDA, WAP Network profile –Modem, Gigabit User profile –Language, Interests, Abilities, Preferences Too costly to author manually

4 Cuypers multimedia generation engine

5 Automatic multimedia generation Designer does not specify complete presentation… …but only specifies requirements System automatically finds a solution which meets requirements How should the requirements be specified? –Declarative constraints

6 Traditional use of constraints Constraint solving used for problems with: –Many variables –Large domains Based on domain reduction paradigm Quantitative constraints –Integer domain –Reduction by arithmetic relations Greater than (>) Less than (<) Equals (=)

7 Drawbacks of quantitative constraints Too many (trivial) solutions that differ by: –1 pixel position, or –1 milliseconds in timing Not sufficiently expressive e.g. cannot specify “no overlap” constraint Too low level e.g. A.X2  B.X1

8 Solution: qualitative constraints Example “A not overlap B”, “B after C” Advantages: –More intuitive –More expressive –Smaller domains Qualitative solutions translate automatically to lower level quantitative constraints New problem: What if constraints are insoluble?

9 Solution: Constraint Logic Programming Combine Prolog unification and backtracking with constraint solving Use Prolog rules to generate constraints Backtrack when constraints are insoluble

10 Cuypers generation engine Multiple layers: –Prolog rules to generate constraints –Qualitative constraints translate to quantitative constraints –Solution of both constraints provides sufficient information for final presentation

11 Cuypers demo:Cuypers demo: scenario Client: User is interested in Rembrandt and wants to know about about the “chiaroscuro” technique Server: Query database Server: Generate constraints according to: –System profile –User profile –Network profile Server: Solve constraints / revise constraints Server: Generate SMIL presentation Client: Play presentation

12 Conclusions Quantitative constraints are insufficient for automatic multimedia presentation generation. Also need Qualitative constraints to allow intuitive and effective high level specification, and Backtracking for revising specific constraints that cause the entire set to fail

13 Future directions Best-first instead of depth-first –Choose “best” among possible solutions –Needs evaluation criteria Improve knowledge management –Make design knowledge declarative and explicit –Preserve metadata in final presentation –Use standardized and reusable profiles

14

15 Need to make trade-offs Semantics –Convey message Aesthetics –Clear / nice layout Resources –Screen size, bandwidth Dimension may result in conflicting goals

16 Quantitative Constraints % csp(+Ids, -Boxes) csp([IdA,IdB],[box(IdA,[x1:AX1, …]), box(IdB,[x1:BX1,…])]) :- % get values maxX(MaxX), maxY(MaxY), height(IdA,HeightA), widtht(IdA,WidthA), … % define domains [AX1,AX2,BX1,BX2]::[0..MaxX], [AY1,AY2,BY1,BY2]::[0..MaxY], % set width & height AX2 – AX1 #= WidthA, AY2 – AY1 #= HeightA, … % constraints AX2 #< BX1,% A left-of B AY1 #= BY1,% A top-align B, … true.

17 Multimedia and Constraints Constraint Logic Programming –Domain reduction –Backtracking –Unification (matching rules) Qualitative Constraints –Non-integer domain –Allen’s 13 temporal interval relations in three dimensions

18 Qualitative Constraints Example: Two images, A,B A left or right of B A not above or below B

19 Qualitative Constraints % csp(+Ids, -Graph) csp([IdA, Idb], [edge(IdA,IdB,x,NoOverlap),…]) :- % define domains NoOverlap :: [b,b-,m,m-], Overlap :: [d,d-,s,s-,f,f-,e], % constraints edge(IdA,IdB,x,NoOverlap),% B not-overlap A edge(IdA,IdB,y,Overlap),% B overlap A true.

20 Qualitative Constraints Reasoning –Inverse: edge(A,B,D,Value) inverse(Value,RValue),edge(B,A,D,RValue). –Equality edge(A,B,D,V1), edge(A,B,D,V2) => V1 #= V2 –Transitive edge(A,B,D,VAB), edge(B,C,D,VBC) => tr(VAB,VBC,VAC), % rule generation algorithm edge(A,C,D,VAC). Translation rules to quantitative domain edge(A,B,D,b) => node(A,D/2,V2), node(B,D/1,V1) V1 #< V2.

21 Problems in generating multimedia Text documents are flexible –Add page, scrollbar, –Template models –Wrap text around images Multimedia documents are less flexible –No pages or scrollbars, no line-breaking or hyphenation –Not based on text-flow –Feedback needed Linear process model does not work for multimedia

22 Quantitative Constraints Example: Two images, A,B A left-of B A top-align B

23 Cuypers generation engine Rhetoric/Semantic –Sequence, Example Communicative devices –Bookshelf, Slideshow Qualitative Constraints –A before B Quantitative Constraints –A.X2 < B.X1 Presentation –SMIL