1. SIG Center for Computer Graphics Pragbot 2.0 Moving Pragbot Language Interactions Toward More Realistic Situations Oct 2009 Review Chris Czyzewicz, Norm Badler

2. SIG Center for Computer Graphics Purpose We were tasked with collecting a corpus of data to study language interaction – Emphasis on intentions Last year, we presented Pragbot, which collected 50 transcripts – Thanks Karl Schultz et. al.

3. SIG Center for Computer Graphics Pragbot % pragbot_chat_log_2007.11.01 AD at 19.48.20 EDT.txt Player 2: i have 4H Player 1: I want it! Player 1: where is it? Player 2: should i leave it for you somewhere? Player 1: sure Player 1: where are you? Player 2: okay, where are you? Player 1: I'm near the top Player 2: i'm left side. Player 1: next to the gap near the middle Player 2: i'll leave the card in the upper left corner. Player 1: awesome

4. SIG Center for Computer Graphics Pragbot 1.0 Review

5. SIG Center for Computer Graphics Pragbot 1.0 Two player game played through an applet Players searched through an environment to find cards Objective was to find 6 cards in a row, all from the same suit Players discussed strategy Transcripts were recorded and studied

6. SIG Center for Computer Graphics Pragbot 1.0 Very successful in gathering transcripts Flexible architecture – No installation (played through an applet) – Platform independent Game requires some strategy, without a steep learning curve – Easy controls and interface

7. SIG Center for Computer Graphics Pragbot 1.0 panel comments We received feedback from the panel last year 2D experience was not immersive Not very realistic for the target of our application Metaphor of card finding should be changed

8. SIG Center for Computer Graphics Pragbot 2.0 Objective of Pragbot 2.0 is the same as the original – Collect transcripts from participants – Determine how they convey intention – Specifically how a human might convey intention to a robot – Examine any other interesting bits of the conversation

9. SIG Center for Computer Graphics Pragbot 2.0 Important that we retain the portability of the original Pragbot – Shouldn't need to install anything – Should be platform independent Maintain ease of control and interface – Game should be simple to play

10. SIG Center for Computer Graphics Pragbot 2.0

11. SIG Center for Computer Graphics Pragbot 2.0 implementation First Person Shooter view Nothing to install – Uses JNLP to download and launch the game – Needs >= Java 1.5 – Windows/Linux at the moment – Runs in its own, full screen window

12. SIG Center for Computer Graphics Pragbot 2.0 implementation Game metaphor has changed – A commander and a robot looking to save hostages, eliminate bombs and terrorists – Game takes place in a Wolfenstein like environment Characters have different roles in the game – Requires coordination between the two players

13. SIG Center for Computer Graphics Pragbot 2.0 implementation Guidance: “Develop a common simulation and experimentation scenario … which is demanding and sophisticated….”, The new metaphor and implementation of the game have helped us with the scenario – On the same page with scenario demo Promoted good discussion during the year about what would be good/bad for a demo Nice sandbox for testing potential scenario ideas

14. SIG Center for Computer Graphics Pragbot 2.0

15. SIG Center for Computer Graphics Pragbot 2.0

16. SIG Center for Computer Graphics Pragbot 2.0

17. SIG Center for Computer Graphics Pragbot 2.0

18. SIG Center for Computer Graphics Pragbot 2.0

19. SIG Center for Computer Graphics Game Dynamics Two possible characters – Commander (CMDR), a human – Junior (JR), a robot Hostages that need to be rescued Terrorists – Being improved as we speak Bombs

20. SIG Center for Computer Graphics Transcripts Data can be collected in the same way as Pragbot 1.0 Players communicate via chat

21. SIG Center for Computer Graphics JR One player plays as JR Defuses bombs, cannot remove barriers Cannot rescue hostages He does the dangerous work Does not receive mission updates

22. SIG Center for Computer Graphics CMDR Other player is CMDR Kills terrorists Is killed by bombs if he is in the same room/hallway Rescues hostages Gets mission updates from HQ

23. SIG Center for Computer Graphics Typical Gameplay A mission is described to CMDR He relays that information to JR, who then begins to explore JR updates CMDR, and requests help when he can't handle something Game is completed when mission is accomplished (terrorists killed, hostages rescued, etc.)

24. SIG Center for Computer Graphics Sample Transcript

25. SIG Center for Computer Graphics Pragbot 2.0 video

26. SIG Center for Computer Graphics Data Collection We have collected several transcripts Some transcripts not incredibly interesting Good ideas for next game objective – More strategy aspects

27. SIG Center for Computer Graphics Website If you would like to try out Pragbot 2.0, you can go to the website and play a game – Plus we would love your transcript! www.seas.upenn.edu/~eczyzewi/pragbot Alpha game is being updated regularly Original game is stable and playable

28. SIG Center for Computer Graphics Future Work Currently in the process of adding more intelligent enemies Terrorists will attempt to flee the building Players will have to coordinate to keep them pinned down Multiple bots?

29. SIG Center for Computer Graphics Future PhD Work Bots with memory of what they, and other bots, have done – Also memory of objects, world Memory = list of PARs Reasoning based on this memory bot/agent PAR-1PAR-2PAR-3PAR-4 ‘me’ … PAR-aPAR-bPAR-cPAR-d ‘b2’ … PAR-APAR-BPAR-CPAR-D ‘b6’ …

30. SIG Center for Computer Graphics Future PhD Work As mentioned earlier, would be ideal to play the game with only one person Person gives commands, bot reacts Will require memory and 'envisionment' – Ability to determine whether or not proposed action is possible, or potential pitfalls Please wait until next year to grill me on this stuff

31. SIG Center for Computer Graphics Thanks! Questions?