1. ARPUPPET – A CHARACTER Animation FRAMEWORK for Augmented Reality Istvá n Barakonyi StbZwerg Day’03

2. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 2/18 Main goal Combining AR applications with intelligent, animated characters NO augmented paper clip NO AI, avoiding the term 'agent‘ Interface to reasoning engines, behavior generators Main focus on Researching important aspects and implications for intelligent animated characters  Visualization, appearance  Behavior  Required technology  Applications

3. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 3/18 Vision AR applications with embedded intelligent animated characters with multiple representations that are aware of Own Other characters UsersPropsPoseBehaviorContext Storage

4. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 4/18 ARPuppet Framework Several different characters acting together  Puppet theater metaphor Author/StorytellerDirector Choreographer Puppeteer 1Puppeteer 2 Puppeteer n Puppet 1.1Puppet 1.PPuppet 2.1Puppet n.1Puppet n.Q … … …

5. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 5/18 Different representations Director Choreographer Puppeteer i Physical Virtual Augmented

6. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 6/18 Different devices/platforms Director Choreographer Puppeteer i Projection screen PC monitor PocketPC

7. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 7/18 Puppeteer features Command parsing /w type checking  commandType %s|%f|%d Unified command queue: parallel / sequential Unified feedback mechanism  If unsupported, provide default feedback Default implementation of commands  Position, orientation, scale  goTo, lookAt, turnTowards, pathMove etc. Conversation: command - response Multi-level compound commands  goTo = { turnTowards, go, stop }, go = { walk_anim, setPosition } Handling multiple puppets and attributes  Puppet-specific command execution Tracking functions  Tracking on/off, Relative pose to tracking station, snap-to-station Default bounding box for collision detection/path planning (IV) Hotspots  Particular meaning or function (e.g. buttons, rooms)  Points of interaction (hand, head etc.) External speech & sound support if not featured by puppets

8. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 8/18 Choreographer features Command parsing  with type checking: own cmds  without type checking: puppeteer attributes can be params  Easy scripting Command synchronization  Easy handling of group commands [targetList:] commandType [parameterList] Conversation: command – response  Choreographer’s own commands - Getter functions - Events: collision, facing, near, approaching etc. - Utilities: motion planning, path following, terrain following  Puppeteer commands - Choreography - e.g. goTo  pathMove, walk  run

9. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 9/18 Scripting examples Example 1: { LEGOrobot: motorState +--, LEGOmaxl: goTo legorobot.position 2000 }, LEGOmaxl: pointAt legorobot.hotspot#motorA Example 2: TourGuide: pathMove (building.hotspot#signpost.path).position 5000, {TourGuide: pointAt building.hotspot#signpost.currentRoom 3000, TourGuide: speak `This is our seminar room.` } Example 3: chess.nextStart: goTo chess.nextDest.position 2000, chess.nextStart, chess.nextDest: play `fight` 3000, chess.nextStart: setAttribute `name` chess.nextDest

10. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 10/18 Existing characters

11. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 11/18 Creating characters  a new Puppeteer Puppets  Prepare to communicate with all representations  Adapt commands Attributes  Sub-scene graphs + field connections Supported commands  Reimplementing standard commands (Walk, Run, Grab, Drop, Point_at, Look_at, Turn towards/away, Idle, etc.)  Own commands Feedback  Which command has finished and when  Set up triggering structure Bounding box calculation structure  e.g. for collision detection, path planning Hotspots  Particular meaning or function (e.g. buttons, rooms)  Points of interaction (hand, head etc.)

12. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 12/18 Configuring / Interaction / Embedding ChoreographerPuppeteerPuppet command attributes 1..j IV, C++, python, … APRILPUC PUC device component class, wrapper… response Application attributes 1..i Story engine Conditions, events wrapper

13. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 13/18 Personal Universal Controller (PUC) Carnegie Mellon University (Pebbles project) + Thomas Psik: Inventor integration Automatically generate control GUI  Statically configured  Dynamically changing (coming soon) Goal: walk up & connect to any object in a running ARPuppet framework to configure generate GUI to configure and control  Puppet  Puppeteer - Command types (e.g. setPosition) - Command format (e.g. %f %f %f %d)  Choreographer - Currently loaded puppeteers puppeteerName: puppeteerCmd [parameterList] choreographerCmd [parameterList]

14. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 14/18 Enhancing applications /w characters Characters  Gestures  Positioning  Highlighting  Animation  Auxiliary objects - Arrows - Labels - Markers - Other MM objects Interaction, communication  Multimodal  2D/3D GUIs (PIP, PUC)  Users’ own behavior

15. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 15/18 Test applications Dummy examples ARSensei – early demo ARLego Virtual Tour Guide Planned demos:  Battlechess - Testing scripting power  “Geist 2” - Ultimate demo - Multiuser role playing game

16. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 16/18 Contribution and current status Combination of a collaborative AR framework and character animation framework has not existed before Working prototype Possibility to experiment with numerous characters with several representations Possibility to experiment with many applications to prove usefulness  own tests/apps  VRLU projects Default values and methods provided  very easy and quick to add own character without much extra work Easy scripting Can be smoothly integrated with other projects  Studierstube apps  PUC  APRIL

17. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 17/18 Future plans Adding lots of extra functionality to the Puppeteer and Choreographer objects Lots of implementation work Closer cooperation with APRIL Studierstube + APRIL + ARPuppet + PUC  powerful combination for AR storytelling Searching for new applications  GEIST 2 ?

18. ISTVÁ N BARAKONYI BARA @ IMS.TUWIEN.AC.AT SLIDE 18/18 Questions?