1. Robot Metaphors and Models

2. Animatronic “Robot” or device
brain
effectors

3. Perceiving “Robot”
brain
sensors

4. Reactive Robot is the simplest behavioral robot
Brain
is a
mapping
sensors
effectors
This is the simplest robot that satisfies the definition of a robot

5. Reactive Robot in environment
ENVIRONMENT is a feedback
brain
sensors
effectors
This is the simplest robot that satisfies the definition of a robot

6. Braitenberg Vehicles and Quantum Automata Robots

7. Another Example: Braitenberg Vehicles and Quantum BV

8. Braitenberg Vehicles

9. Emotional Robot has a simple form of memory or state
Brain
is a
Finite
State
Machine
sensors
effectors
This is the simplest robot that satisfies the definition of a robot

10. Behavior as an interpretation of a string
Newton, Einstein and Bohr.
Hello Professor
Hello Sir
Turn Left . Turn right.
behavior

11. Behavior as an interpretation of a tree
Newton, Einstein and Bohr.
Hello Professor
Hello Sir
Turn Left . Turn right.
behavior
Grammar. Derivation. Alphabets.

12. Our Base Model and Designs

13. Neck and upper body movement generation

14. Robot Head Construction, 1999
High school summer camps, hobby roboticists, undergraduates
Furby head with new control
Jonas
We built and animated various kinds of humanoid heads with from 4 to 20 DOF, looking for comical and entertaining values.

15. Mister Butcher
Latex skin from Hollywood
4 degree of freedom neck

16. Robot Head Construction, 2000
Skeleton
Alien
We use inexpensive servos from Hitec and Futaba, plastic, playwood and aluminum.
The robots are either PC-interfaced, use simple micro-controllers such as Basic Stamp, or are radio controlled from a PC or by the user.

17. Technical Construction, 2001 Details
Marvin the Crazy Robot
Adam

18. Virginia Woolf
2001
heads equipped with microphones, USB cameras, sonars and CDS light sensors

19. 2002 BUG (Big Ugly Robot) Max
Image processing and pattern recognition uses software developed at PSU, CMU and Intel (public domain software available on WWW).
Software is in Visual C++, Visual Basic, Lisp and Prolog.

20. Visual Feedback and Learning based on Constructive Induction
2002
Uland Wong, 17 years old

21. Professor Perky 2002, Japan 1 dollar latex skin from China
Professor Perky with automated speech recognition (ASR) and text-to-speech (TTS) capabilities
We compared several commercial speech systems from Microsoft, Sensory and Fonix.
Based on experiences in highly noisy environments and with a variety of speakers, we selected Fonix for both ASR and TTS for Professor Perky and Maria robots.
We use microphone array from Andrea Electronics.
1 dollar latex skin from China

22. Maria, 2002/2003
20 DOF

23. Construction details of Maria
location of head servos
skull
location of controlling rods
location of remote servos
Custom designed skin

24. Animation of eyes and eyelids

25. Cynthia, 2004, June

26. Currently the hands are not moveable.
We have a separate hand design project.

27. Software/Hardware Architecture
Network- 10 processors, ultimately 100 processors.
Robotics Processors. ACS 16
Speech cards on Intel grant
More cameras
Tracking in all robots.
Robotic languages – Alice and Cyc-like technologies.

28. Face detection localizes the person and is the first step for feature and face recognition.
Acquiring information about the human: face detection and recognition, speech recognition and sensors.

29. Face features recognition and visualization.

30. Use of Multiple-Valued (five-valued) variables Smile, Mouth_Open and Eye_Brow_Raise for facial feature and face recognition.

31. HAHOE KAIST ROBOT THEATRE, KOREA, SUMMER 2004
Czy znacie dobra sztuke dla teatru robotow?
Sonbi, the Confucian Scholar
Paekchong, the bad butcher

32. Editing movements

33. Yangban the Aristocrat and Pune his concubine
The Narrator

34. The Narrator

36. We base all our robots on inexpensive radio-controlled servo technology.

37. We are familiar with latex and polyester technologies for faces
Martin Lukac and Jeff Allen wait for your help, whether you want to program, design behaviors, add muscles, improve vision, etc.

38. New Silicone Skins

39. A simplified diagram of software explaining the principle of using machine learning based on constructive induction to create new interaction modes of a human and a robot.

40. Probabilistic and Finite State Machines

41. Probabilistic State Machines to describe emotions
“you are beautiful”
/ ”Thanks for a compliment”
P=1
Happy state
“you are blonde!”
/ ”I am not an idiot”
P=0.3
“you are blonde!”
/ Do you suggest I am an idiot?”
Unhappy state
P=0.7
Ironic state

42. Facial Behaviors of Maria
Maria asks:
Do I look like younger than twenty three?
Response:
“no”
“no”
“yes”
0.7
0.3
Maria smiles
Maria frowns

43. Probabilistic Grammars for performances
Speak ”Professor Perky”, blinks eyes twice
P=0.1
Speak ”Professor Perky”
Where?
P=0.3
Who?
P=0.5
P=0.5
P=0.5
Speak “in some location”, smiles broadly
Speak “In the classroom”, shakes head
Speak ”Doctor Lee”
What?
P=0.1
P=0.1
Speak “Was singing and dancing”
P=0.1
P=0.1
Speak “Was drinking wine” ….

44. Human-controlled modes of dialog/interaction
Human teaches
“Thanks, I have a lesson”
“Hello Maria”
“Lesson finished”
Robot
performs
Robot asks
“Question”
“Stop performance”
“Questioning finished”
“Command finished”
“Thanks, I have a question”
“Thanks, I have a command”
Human asks
Human commands

45. Descriptions of Motions and Behaviors

46. Motion Descriptions Two dimensional matrix.
Rows are rotations of servos,
Column are robot poses.
Matrix is a robot gesture
WAVEFORM 1
WAVEFORM 2
WAVEFORM 3
WAVEFORM N
GESTURE 1
WE CAN USE ALGEBRA OF MATRICES, EIGENVALUES, EIGENVECTORS, MATRIX DECOMPOSITION ETC
GESTURE 2
GESTURE 3

47. Encoding poses by symbolsA B
AAB
BABA
You tall me?

48. Behavior Descriptions
Two dimensional matrix.
Rows are conditions of executing this pose,
Column are robot poses.
Matrix is a robot gesture with conditions.
Two dimensional matrix.
Rows are rotations of servos,
Column are robot poses.
Matrix is a robot gesture
WAVEFORM 1
WAVEFORM 2
WAVEFORM 3
WAVEFORM N
GESTURE 1
WE CAN USE ALGEBRA OF MATRICES, EIGENVALUES, EIGENVECTORS, MATRIX DECOMPOSITION ETC
GESTURE 2
GESTURE 3

49. If A and B then do X1 and X3 A B Partial conditions (Boolean) C D A B
Partial motions (servos or servo sequences) - X3 - X4 X1
previous
Partial motions (servos or servo sequences) X3
previous -

50. If A and C’ and D’ then do X1, X2 and X3B
Partial conditions (Boolean) C D A X1
Does not care
Partial conditions (Boolean) X2
Partial motions (servos or servo sequences) C’ X3 D’ X4 X1 X2
Partial motions (servos or servo sequences) X3
previous -

51. Behaviors as trees Braitenberg Vehicle Condition A Braitenberg Faces
yes no
Braitenberg
Bipeds
TURN RIGHT
Condition B
yes no
Condition C
yes no
GO STRAIGHT
TURN LEFT
Go Back

52. Behaviors as strings A A’B’ A’ B C’ A’ B C condition Motion or pose
Condition A
Condition C
Condition B
yes no
TURN RIGHT
GO STRAIGHT
TURN LEFT
Go Back A
TURN RIGHT
A’B’
GO STRAIGHT
A’ B C’
Go Back
A’ B C
TURN LEFT
condition
Motion or pose
Condition can be generalized to a Boolean or Multivalued expression

53. Behaviors as state machines
I am in a happy state
Condition A
Condition C
Condition B
yes no
GO STRAIGHT
TURN LEFT
Go Back
I am in an angry state

54. Dialog and Robot’s Knowledge

55. Robot-Receptionist Initiated Conversation
Human
Robot
What can I do for you?
Robot asks
This represents operation mode

56. Robot-Receptionist Initiated Conversation
Human
Robot
What can I do for you?
I would like to order a table for two
Robot asks

57. Robot-Receptionist Initiated Conversation
Human
Robot
Smoking or non-smoking?
Robot asks

58. Robot-Receptionist Initiated Conversation
Human
Robot
Smoking or non-smoking?
I do not understand
Robot asks

59. Robot-Receptionist Initiated Conversation
Human
Robot
Do you want a table in a smoking or non-smoking section of the restaurant?
Non-smoking section is near the terrace.
Robot asks

60. Robot-Receptionist Initiated Conversation
Human
Robot
Do you want a table in a smoking or non-smoking section of the restaurant?
Non-smoking section is near the terrace.
A table near the terrace, please
Robot asks

61. Human-Initiated Conversation
Robot
Hello Maria
initialization
Robot asks

62. Human-Initiated Conversation
Robot
Hello Maria
What can I do for you?
Robot asks

63. Human-Asking
Human
Robot
Question
Question
Robot asks
Human asks

64. Human-Asking
Human
Robot
Yes, you ask a question.
Question
Human asks

65. Human-Asking Yes, you ask a question. What book wrote Lee? Human asks
Robot
Yes, you ask a question.
What book wrote Lee?
Human asks

66. Human-Asking I have no sure information. What book wrote Lee?
Robot
I have no sure information.
What book wrote Lee?
Human asks

67. Human-Asking I have no sure information. Try to guess. Human asks
Robot
I have no sure information.
Try to guess.
Human asks

68. Human-Asking Lee wrote book “Flowers”. Try to guess. Human asks Human
Robot
Lee wrote book “Flowers”.
Try to guess.
Human asks

69. Human-Asking Lee wrote book “Flowers”. This is not true. Human asks
Robot
Lee wrote book “Flowers”.
This is not true.
Human asks

70. Human-Teaching Human ends questioning Questioning finished
Robot
Questioning finished
Human asks
Human teaches
“Questioning finished”
Robot asks
Thanks, I have a lesson

71. Human-Teaching Robot enters asking mode Questioning finished
What can I do for you?
Questioning finished
Human asks
Human teaches
“Questioning finished”
Robot asks
Thanks, I have a lesson

72. Human-Teaching Human starts teaching Thanks, I have a lesson
Robot
What can I do for you?
Thanks, I have a lesson
Human asks
Human teaches
“Questioning finished”
Robot asks
Thanks, I have a lesson

73. Human-Teaching
Human
Robot
Thanks, I have a lesson
Yes
Human teaches

74. Human-Teaching I give you question-answer pattern Yes Human teaches
Robot
I give you question-answer pattern
Yes
Human teaches

75. Human-Teaching Question pattern: What book Smith wrote? Yes
Robot
Question pattern:
What book Smith wrote?
Yes
Human teaches

76. Human-Teaching Answer pattern: Smith wrote book “Automata Theory” Yes
Robot
Answer pattern:
Smith wrote book “Automata Theory”
Yes
Human teaches

77. Human-Teaching Checking question: What book wrote Smith? Yes
Robot
Checking question:
What book wrote Smith?
Yes
Human teaches

78. Human-Teaching Checking question: What book wrote Smith?
Robot
Checking question:
What book wrote Smith?
Smith wrote book “Automata Theory”
Human teaches

79. Human-Teaching I give you question-answer pattern Yes Human teaches
Robot
I give you question-answer pattern
Yes
Human teaches

80. Human-Teaching Question pattern: Where is room of Lee? Yes
Robot
Question pattern:
Where is room of Lee?
Yes
Human teaches

81. Human-Teaching Answer pattern: Lee is in room 332 Yes Human teaches
Robot
Answer pattern:
Lee is in room 332
Yes
Human teaches

82. Human-Checking what robot learned
Lesson finished
Human teaches
“Lesson finished”
Question
Robot asks
Human asks

83. Human-Checking what robot learned
Lesson finished
What can I do for you?
Human teaches
“Lesson finished”
Question
Robot asks
Human asks

84. Human-Checking what robot learned
Question
What can I do for you?
Human teaches
“Lesson finished”
Question
Robot asks
Human asks

85. Human-Asking Yes, you ask a question. Question Human teaches
Robot
Yes, you ask a question.
Question
Human teaches
“Lesson finished”
Question
Robot asks
Human asks

86. Human-Asking Yes, you ask a question. What book wrote Lee? Human asks
Robot
Yes, you ask a question.
What book wrote Lee?
Human asks

87. Human-Asking I have no sure information. What book wrote Lee?
Robot
I have no sure information.
What book wrote Lee?
Human asks

88. Human-Asking I have no sure information. Try to guess. Human asks
Robot
I have no sure information.
Try to guess.
Human asks

89. Human-Asking Lee wrote book “Automata Theory” Try to guess. Human asks
Robot
Lee wrote book “Automata Theory”
Try to guess.
Observe that robot found similarity between Smith and Lee and generalized (incorrectly)
Human asks

90. Behavior, Dialog and Learning
The dialog/behavior has the following components:
(1) Eliza-like natural language dialogs based on pattern matching and limited parsing.
Commercial products like Memoni, Dog.Com, Heart, Alice, and Doctor all use this technology, very successfully – for instance Alice program won the 2001 Turing competition.
This is a “conversational” part of the robot brain, based on pattern-matching, parsing and black-board principles.
It is also a kind of “operating system” of the robot, which supervises other subroutines.

91. Behavior, Dialog and Learning
(2) Subroutines with logical data base and natural language parsing (CHAT).
This is the logical part of the brain used to find connections between places, timings and all kind of logical and relational reasonings, such as answering questions about Japanese geography.

92. Behavior, Dialog and Learning
(3) Use of generalization and analogy in dialog on many levels.
Random and intentional linking of spoken language, sound effects and facial gestures.
Use of Constructive Induction approach to help generalization, analogy reasoning and probabilistic generations in verbal and non-verbal dialog, like learning when to smile or turn the head off the partner.

93. Behavior, Dialog and Learning
(4) Model of the robot, model of the user, scenario of the situation, history of the dialog, all used in the conversation.
(5) Use of word spotting in speech recognition rather than single word or continuous speech recognition.
(6) Continuous speech recognition (Microsoft)
(7) Avoidance of “I do not know”, “I do not understand” answers from the robot.
Our robot will have always something to say, in the worst case, over-generalized, with not valid analogies or even nonsensical and random.

94. Questions for Homeworks, Quizzes and Projects (1)
What is Braitenberg Vehicle?
Explain simple Braitenberg vehicles that operate on continuous (analog), binary and multiple-valued data.
Explain a model of robot behavior based on a single Finite State Machine.
Explain a model of robot behavior based on several communicating probabilistic Finite State Machines.
Explain a natural language model for a robot based on probabilistic and deterministic state machines.
What is the internal state of the robot?
Explain the concept of a behavior editor for a humanoid robot.
Find on internet about Eliza, Alice and other chatbots and explain how they can be integrated in a complete robotic theatre of humanoid robots.
What is a single application of Machine Learning in robotics.
Think about possible applications of Machine Learning in humanoid robot theatre.
Draw a state diagram of FSM controlling a Braitenberg Vehicle type of robot with three inputs that can be switched from Aggressive to Shy behavior.
Draw a state diagram of a humanoid simple robot that behaves similarly to Aggressive and Shy Braitenberg Vehicles.

95. Questions (2) What is the use of Avatar in robot design?1
What is the use of Avatar in robot design?
Give example of Multiple-Valued logic in Machine Learning applied to a robot.
Invent a humanoid robot head that would behave similarly to any Braitenberg Vehicle. Draw figure and State Machine for servos.
Explain simple Braitenberg vehicles that operate on fuzzy signals using Post Literals and MIN, MAX operators.
Write a software program that would use natural language and probabilistic state machine, similarly to examples above.
Design a robot FSM with three states : Happy, Unhappy and Neutral that would behave accordingly to one of the three above moods.
Give concepts for a humanoid robot head with moveable eyes, eyebrows, ears, lips, cheeks and jaw. How to build it mechanically. Look critically at the pictures in this set of slides.
How to control all servos of these head parts – give ideas and explain how to relate them to facial gestures.
Think how to use Boolean Minimization to find generalized answers in question-answering.
Apply Braitenberg Vehicle ideas to Jimmy robot.
Apply Braitenberg Vehicle ideas to the robot from your project.