1. Multimodal Plan Representation for Adaptable BML Scheduling Dennis Reidsma, Herwin van Welbergen, Job Zwiers

2. SAIBA

3. BML

4. BML behavior and sync points

5. The BML scheduling process Scheduling the multimodal behavior plan

6. The behavior plan BML Specifies behaviors and the constraints between them The multimodal behavior plan Specified on the basis of, e.g., communicative intent Incrementally constructed from BML blocks The motor plan Executes the behavior plan on the embodiment of the virtual human, using sound, joint rotations, FAPs, … Making the motor plan flexible Achieve interpersonal coordination Allow on the fly adjustments of ongoing behavior But do not violate the specified constraints

7. BML constraints Explicit constraints Before/after At Implicit constraints Sync points should not occur before the start of their block Behaviors should have >0 duration The default sync points (e.g. start, ready, stroke_start, stroke, stroke_end, relax, end) must remain in order

8. BML constraints Realizer specific constraints Due to technical limitation or Theoretically motivated Block level constraints Merge: start block at current time Append: start as soon as possible after all behaviors in the current plan (but not earlier than current time)

9. Additional constraints in Elckerlyc Elckerlyc specific constraints Whitespace constraint: no unnecessary ‘whitespace’ between behaviors Append-after(X) block constraint

10. Flexible plan representation The (motor) plan Describes the low level execution of motor behavior on the embodiment of the virtual human: joint rotations, MPEG4 FAP movement, sound, …etc To be flexible, the plan representation must maintain information about the relation between elements in the plan and the original BML expressions from which they originated be capable of expressing all the constraints be capable of being modified… …in such a way that the (remaining) constraints automatically stay satisfied

11. Central: the PegBoard Each sync point is assigned to a TimePeg on this board Sync points that are connected by an ‘at’ constraint share the same TimePeg TimePegs can be moved, changing the time of the associated sync(s) TimePegs provide local timing (irt the start of the BML block) Each TimePeg is connected to a BMLBlockPeg BMLBlockPegs provide global timing Elckerlyc’s plan representation

12. OffsetPeg: links to another TimePeg and moves with it, retaining a time offset All ‘at’ constraints can be expressed in Pegs, OffsetPegs en Blockpegs After en before constraints require a specific pair of Pegs – Not implemented yet – (but: trivial to design and implement)

13. Example: solving BML to TimePegs As you can see on this painting,... <gesture id="point1" start="walk1:relax" type="POINT" target="painting1" stroke="speech1:s1+0.5"/>

14. Managing block timing

15. Architecture Separated parsing, scheduling, execution Central: the PegBoard Engines handle unimodal motor plans Sync points of PlanUnits are connected by TimePegs Anticipators may make time adjustments through TimePegs

16. Architecture The scheduler communicates with a set of engines to set up the multimodal behavior plan It knows for each BML behavior what Engine handles it Engines have a standardized interface with functions to: – Add a BML behavior to the motor plan – Remove a BML behavior from the motor plan – Resolve unknown time constraints on a BML behavior given its known time constraints – Check which BML behavior in the motor plan are currently invalid Note that communication with Engines is in terms of BML behaviors

17. Managing adjustments of the plan Interruption of a behavior Will automatically shorten the block’s end Satisfies block append constraint Time adjustment of a sync point Through its associated TimePeg Will also move syncs that are connected to it with ‘at’ constraints Might invalidate other constraints – Each Engine has functionality to check for this – Solution: drop behavior, drop constraint,.. – And communicate back to the behavior planner

18. Conclusion BML scheduling can be viewed as a constraint problem Elckerlyc uses this view to maintain a flexible behavior plan representation – Allows micro (timing) adjustments – But maintains constraints Enables applications that require tight mutual coordination between a user and a virtual human These capabilities have been used in: Attentive Speaker, recent version of Virtual Trainer

19. Thanks for your attention

20. Further work Capture before/after constraints Multithreaded scheduling