1. STEP: a scripting language for embodied agents Zhisheng Huang, Anton Eliens, and Cees Visser (Scripting Technology for Embodied Persona) http://wasp.cs.vu.nl/step

2. Embodied Agent and 3D Web Agent Embodied agents are autonomous agents which have bodies by which the agents can perceive their world directly through sensors and act on the world directly through effectors. 3D Web agents are embodied agents whose experienced worlds are the Web, and whose 3D avatars can interact with each other or with users via Web browsers

3. Motivation Why Embodied Agents Sound, graphics, and knowledge could convey ideas and feelings faster than technical documents.[Bickmore(1998)] Multimodal human-computer interface, in particular, human- Web interface

4. Motivation Why Embodied Agents User interface is better if implemented with respect to what people would expect from the same kind of creature in the real world, regarding personality and emotion[Reeves(1996)] The use of an anthropomorphic agent may affect the way in which a user interacts with an application or interface[Koda(1996)]

5. Motivation Why Scripting Languages Embodied agent oriented. More convenience and efficiency for authoring. Reusability of scripts for different agents in different application contexts.

6. Motivation Why Scripting Language Scripting languages focus on communicative acts, like gesture and posture, facial expression, and speech. The specification of external- oriented communicative acts are separated from the specification of the internal states of embodied agents to reduce the complexity

7. Principles Convenience: natural-language-like terms turn(Agent, left_arm, front, slow) turn(Agent, left_arm, rotation(1,0,0,1.57), 3) Compositional semantics: combining operators. par([turn(Agent, left_arm, front, slow), turn(Agent, right_arm, front, slow)]). Re-definability: for high-level specification of actions run_then_kick(Agent)=seq([run(Agent),kick(Agent)]). Parametrization: for the adaptation of actions Introduce variables in actions. Interaction: with a (virtual) environment

8. STEP and Dynamic Logic Distinction between an action and a state Semantically a state represents the properties at a particular moment, whereas an action consists of a set of state pairs, which represent a relation between two states.

9. Dynamic Logic  [a]  if the state  is true, then after doing action a, the state  is true. Pre-condition:  and post-condition:  two sub languages: a language for action specification, and a meta language for state specification

10. STEP and its Meta Language STEP is a scripting language for action specification. In order to interact with virtual worlds/states, STEP needs a meta language, like DLP, Prolog, Java, etc.

11. Main Features of STEP X3D/VRML H-anim avatars based, the standard Web3D Technology. Dynamic logic based, for solid semantic foundation. Declarative semantics, for knowledge representation and reasoning, and for agent technology

12. Implementation STEP has been implemented by using Distributed Logic Programming Language (DLP) STEP (version 1.0) and its authoring tools are available from the STEP website: http://wasp.cs.vu.nl/step

13. Why DLP Object-oriented parallel Prolog Web3D support TCP/IP support for network communication Java-based. All DLP programs are compiled into Java classes Efficiency for programming Reasonably good performance

14. FACT : STEP Source Code (1.0)

15. FACT: Web_oriented Web deliverable animation. No need to install any software or tool to run and test STEP scripts. No DLLs are needed to run STEP facial animation engine. Animation and presentation can be created on-the-fly, well suitable for web agents which need interaction with users in real time with reasonably good performance.

16. Functionalities Gesture and Posture (version 1.0) Inverse Kinematics I(version 1.0) XSTEP (version 1.0) Text to Speech (version 1.1) Facial Animation and Emotion Expression (version 2.0)

17. Reference System of STEP Direction Reference for Humanoid

18. Combination of Directions

19. Body Reference based on H-anim specification

20. Time Reference Duration: very_fast, fast, slow, very_slow, very_very_fast Time: time(3, second)

21. Action Operators turn(Agent, BodyPart, Direction, Duration). move(Agent, BodyPart, Direction, Duration). seq([Action 1, …, Action m ]) par([Action 1, …,Action m ]) repeat(Action, T).

22. Interaction Operators scripting language versus meta language execution: do(State) where State is a formula in the meta language: do(setSFVec3f(Agent,position,X,Y,Z)) conditional: if_then_else(Condition, Action1, Action2) until: until(Action, UntilCondition)

23. Actions in Script Libraries Script(action_name(Agent, OtherVariables), Action):- Action = ComplexAction. Prolog-like syntax

24. Example: Walk

25. Walk in STEP script(walk_pose(Agent), Action):- Action = seq([par([ turn(Agent,r_shoulder,back_down2,fast), turn(Agent,r_hip,front_down2,fast), turn(Agent,l_shoulder,front_down2,fast), turn(Agent,l_hip,back_down2,fast)]), par([turn(Agent,l_shoulder,back_down2,fast), turn(Agent,l_hip,front_down2,fast), turn(Agent,r_shoulder,front_down2,fast), turn(Agent,r_hip,back_down2,fast)])]). Script(walk(Agent, N), Action):- Action = repeat(walk_pose(Agent),N).

26. Variants of walk script(walk_forward_step0(Agent,StepLength),Action):- Action = par([walk_pose(Agent), move(Agent,increment(0.0,0.0,StepLength),fast)]).

27. Execution of scripting actions In DLP Format:do_script(Script) Example:do_script(walk(humanoid, 10))

28. STEP Authoring Tools

29. Run

30. Run in STEP script(start_run(Agent), Action) :- Action = seq([par([turn(Agent, r_elbow, front, fast), turn(Agent, l_elbow, front, fast)]), par([turn(Agent, l_hip, front_down2, fast), turn(Agent, r_hip, front_down2, fast), turn(Agent, l_knee, back_down, fast), turn(Agent, r_knee, back_down, fast)]) ]), !. script(run(Agent), Action) :- Action = seq([par([turn(Agent,r_shoulder,back_down2,fast), turn(Agent,r_hip,front_down,fast), turn(Agent,l_shoulder,front_down2,fast), turn(Agent,l_hip,back_down2,fast)]), par([turn(Agent,l_shoulder,back_down2,fast), turn(Agent,l_hip,front_down,fast), turn(Agent,r_shoulder,front_down2,fast), turn(Agent,r_hip,back_down2,fast)]) ]), !.

31. Example: Taichi

32. STEP script for Taichi script(taichi(Agent,beginning_posture),Action):- Action =seq([ turn(Agent,l_hip,side1_down,fast), turn(Agent,r_hip,side1_down,fast), par([turn(Agent,l_shoulder,front,slow), turn(Agent,r_shoulder,front,slow)])]). script(taichi(Agent,push_down_posture),Action) :- Action =seq([ par([turn(Agent,l_shoulder,front_down,slow), turn(Agent,r_shoulder,front_down,slow), turn(Agent,l_elbow,front_right2,slow), …...

33. Variables and Parametrization Agent names and their corresponding parameters are specified as variables in script libraries, by which the same scripting actions can be re- used for different embodied agents under different situations by different authors. It improves the reusability of scripting actions for the purpose of productivity.

34. Example: Interaction with virtual worlds

35. STEP script: look-at-ball script(turnToDirection(Object,SrcVector,DestVector),Action):- Action = seq([ do(vectorCrossProduct(SrcVector,DestVector,vector(X,Y,Z),R)), do(setRotation(Object,X,Y,Z,R))]). script(look_at_position(Agent,X1,_Y1,Z1),Action):- Action = seq([do(getPosition(Agent,X,_Y,Z)), do(Xdif is X1-X), do(Zdif is Z1-Z), turnToDirection(Agent,vector(0.0,0.0,1.0), vector(Xdif,0.0,Zdif))]).

36. Interaction Operators for Computation High-level interaction operators can be used to access the computational capabilities of the meta language in order to find the computation solutions, which involve inverse kinematics.

37. Example: Touch an inverse kinematics problem

38. STEP script: Touch script(getABvalue(Agent,position(X0,Y0,Z0),Hand,A,B),Action):- Action = seq([getDvalue(Agent,position(X0,Y0,Z0),Hand, D), get_upperarm_length(Agent,L1), get_forearm_length(Agent,L2), do(D1 is L1 + L2), if_then_else(sign(D1-D)>sign(0.001-D), seq([do(cosine_law(L1,L2,D,A)), do(cosine_law(L1, D, L2, B))]), seq([do(A is 1.57*(1+sign(D-0.001))), do(B is 0.0)]))]).

39. script(getVvalue(Agent,position(X0,Y0,Z0),Hand,V),Action):- Action = seq([get_shoulder_center(Agent,Hand, position(X2,Y2,Z2)), do(direction_vector(position(X2,Y2,Z2),position(X0,Y0,Z0),V))]). script(touch(Agent, position(X0,Y0,Z0),l),Action):- Action = seq([getABvalue(Agent,position(X0,Y0,Z0),l,A,B), do(R1 is 3.14-A), getVvalue(Agent,position(X0,Y0,Z0),l,V), get_arm_vector(Agent,l,V0), do(vector_cross_product(V0,V,vector(X3,Y3,Z3),C)), do(R2 is C-B), par([turn(Agent,l_shoulder,rotation(X3,Y3,Z3,R2),fast), turn(Agent,l_elbow,rotation(X3,Y3,Z3,R1),fast), turn(Agent,l_wrist,rotation(X3,Y3,Z3,-0.5),fast)])]).

40. script(touch_absolutePosition(Agent,position(X1,Y1,Z1),Hand),Acti on):- Action = seq([do(getPosition(Agent,X,Y,Z)), do(getRotation(Agent, X2,Y2,Z2,R)), do(X3 is X1-X), do(Y3 is Y1-Y), do(Z3 is Z1-Z), do(R1 is -R), do(position_rotation(position(X3,Y3,Z3), rotation(X2,Y2,Z2,R1),position(X4,Y4,Z4))), touch(Agent,position(X4,Y4,Z4), Hand)]).

41. STEP Architecture

42. STEP Engine

43. XSTEP: A Markup Language for Embodied Agent XSTEP is XML-based STEP. For XSTEP we have developed an XSLT style sheet, using Saxon package, that transforms an XSTEP specification into STEP, thus DLP.

44. An Example in XSTEP ……

45. XSTEP Editor

46. Installation of STEP on DLP 1. Download step1.0 Lite (which includes STEP engine, templates, authoring tools, and the avatar ‘yt’) and unzip it to the DLP directory dlp. Make sure that do not overwrite the existing files in your system when you use your own unzip tool. 2. STEP templates : the application template ‘steptest.pl’ the script library template ‘stepscriptlib.pl’ the step setting template ‘stepsetting.pl’

47. Default Script Library Name step_scriptlib Problem: Due to the limit of the Java virtual machine, the size of each script library cannot be larger than 64KB after the compiling. Solution: Split a large library into multiple ones.

48. Hierarchical Library Claims :-object step_scriptlib : [ bcilib ]. script(ScriptAction, Action):- step_myscriptlib <- script(ScriptAction,Action), !. :-end_object step_scriptlib. Your own library object name

49. Humanoid Objects humanoid object: humanoid is defined based on a prototype non humanoid object: humanoid is not defined based on a prototype.

50. stepsetting.pl :-object stepsetting. humanoid_objects(humanoid1, false):-!. % humanoid1 is not a humanoid object humanoid_objects(humanoid2, true):-!. % humanoid2 is a humanoid object :-end_object stepsetting.

51. STEP application template :-object my_step_application : [bcilib, step_kernel]. var url = './yt/yt.wrl'. main:- text_area(BrowserStream), set_output(BrowserStream), loadURL(url), sleep(6000), script_threads, /* initialize parallel processing*/ do_script(walk(humanoid)), do_script(run(humanoid)). :-end_object my_step_application.

52. Exercises Define at least five postures/gestures in STEP from the following set: kick, beat, sit, kneeing, squat, crouch, crawl, climb, lie, sleep, fall, stretch, greeting, hello, bye-bye,...