Robot Metaphors and Models

Animatronic “Robot” or device brain effectors

Perceiving “Robot” brain sensors

Reactive Robot is the simplest behavioral robot Brain is a mapping sensors effectors This is the simplest robot that satisfies the definition of a robot

Reactive Robot in environment ENVIRONMENT is a feedback brain sensors effectors This is the simplest robot that satisfies the definition of a robot

Braitenberg Vehicles and Quantum Automata Robots

Another Example: Braitenberg Vehicles and Quantum BV

Braitenberg Vehicles

Emotional Robot has a simple form of memory or state Brain is a Finite State Machine sensors effectors This is the simplest robot that satisfies the definition of a robot

Behavior as an interpretation of a string Newton, Einstein and Bohr. Hello Professor Hello Sir Turn Left . Turn right. behavior

Behavior as an interpretation of a tree Newton, Einstein and Bohr. Hello Professor Hello Sir Turn Left . Turn right. behavior Grammar. Derivation. Alphabets.

Our Base Model and Designs

Neck and upper body movement generation

Robot Head Construction, 1999 High school summer camps, hobby roboticists, undergraduates Furby head with new control Jonas We built and animated various kinds of humanoid heads with from 4 to 20 DOF, looking for comical and entertaining values.

Mister Butcher Latex skin from Hollywood 4 degree of freedom neck

Robot Head Construction, 2000 Skeleton Alien We use inexpensive servos from Hitec and Futaba, plastic, playwood and aluminum. The robots are either PC-interfaced, use simple micro-controllers such as Basic Stamp, or are radio controlled from a PC or by the user.

Technical Construction, 2001 Details Marvin the Crazy Robot Adam

Virginia Woolf 2001 heads equipped with microphones, USB cameras, sonars and CDS light sensors

2002 BUG (Big Ugly Robot) Max Image processing and pattern recognition uses software developed at PSU, CMU and Intel (public domain software available on WWW). Software is in Visual C++, Visual Basic, Lisp and Prolog.

Visual Feedback and Learning based on Constructive Induction 2002 Uland Wong, 17 years old

Professor Perky 2002, Japan 1 dollar latex skin from China Professor Perky with automated speech recognition (ASR) and text-to-speech (TTS) capabilities We compared several commercial speech systems from Microsoft, Sensory and Fonix. Based on experiences in highly noisy environments and with a variety of speakers, we selected Fonix for both ASR and TTS for Professor Perky and Maria robots. We use microphone array from Andrea Electronics. 1 dollar latex skin from China

Maria, 2002/2003 20 DOF

Construction details of Maria location of head servos skull location of controlling rods location of remote servos Custom designed skin

Animation of eyes and eyelids

Cynthia, 2004, June

Currently the hands are not moveable. We have a separate hand design project.

Software/Hardware Architecture Network- 10 processors, ultimately 100 processors. Robotics Processors. ACS 16 Speech cards on Intel grant More cameras Tracking in all robots. Robotic languages – Alice and Cyc-like technologies.

Face detection localizes the person and is the first step for feature and face recognition. Acquiring information about the human: face detection and recognition, speech recognition and sensors.

Face features recognition and visualization.

Use of Multiple-Valued (five-valued) variables Smile, Mouth_Open and Eye_Brow_Raise for facial feature and face recognition.

HAHOE KAIST ROBOT THEATRE, KOREA, SUMMER 2004 Czy znacie dobra sztuke dla teatru robotow? Sonbi, the Confucian Scholar Paekchong, the bad butcher

Editing movements

Yangban the Aristocrat and Pune his concubine The Narrator

The Narrator

We base all our robots on inexpensive radio-controlled servo technology.

We are familiar with latex and polyester technologies for faces Martin Lukac and Jeff Allen wait for your help, whether you want to program, design behaviors, add muscles, improve vision, etc.

New Silicone Skins

A simplified diagram of software explaining the principle of using machine learning based on constructive induction to create new interaction modes of a human and a robot.

Probabilistic and Finite State Machines

Probabilistic State Machines to describe emotions “you are beautiful” / ”Thanks for a compliment” P=1 Happy state “you are blonde!” / ”I am not an idiot” P=0.3 “you are blonde!” / Do you suggest I am an idiot?” Unhappy state P=0.7 Ironic state

Facial Behaviors of Maria Maria asks: Do I look like younger than twenty three? Response: “no” “no” “yes” 0.7 0.3 Maria smiles Maria frowns

Probabilistic Grammars for performances Speak ”Professor Perky”, blinks eyes twice P=0.1 Speak ”Professor Perky” Where? P=0.3 Who? P=0.5 P=0.5 P=0.5 Speak “in some location”, smiles broadly Speak “In the classroom”, shakes head Speak ”Doctor Lee” What? P=0.1 P=0.1 Speak “Was singing and dancing” P=0.1 P=0.1 Speak “Was drinking wine” ….

Human-controlled modes of dialog/interaction Human teaches “Thanks, I have a lesson” “Hello Maria” “Lesson finished” Robot performs Robot asks “Question” “Stop performance” “Questioning finished” “Command finished” “Thanks, I have a question” “Thanks, I have a command” Human asks Human commands

Descriptions of Motions and Behaviors

Motion Descriptions Two dimensional matrix. Rows are rotations of servos, Column are robot poses. Matrix is a robot gesture WAVEFORM 1 WAVEFORM 2 WAVEFORM 3 WAVEFORM N GESTURE 1 WE CAN USE ALGEBRA OF MATRICES, EIGENVALUES, EIGENVECTORS, MATRIX DECOMPOSITION ETC GESTURE 2 GESTURE 3

Encoding poses by symbols A B AAB BABA You tall me?

Behavior Descriptions Two dimensional matrix. Rows are conditions of executing this pose, Column are robot poses. Matrix is a robot gesture with conditions. Two dimensional matrix. Rows are rotations of servos, Column are robot poses. Matrix is a robot gesture WAVEFORM 1 WAVEFORM 2 WAVEFORM 3 WAVEFORM N GESTURE 1 WE CAN USE ALGEBRA OF MATRICES, EIGENVALUES, EIGENVECTORS, MATRIX DECOMPOSITION ETC GESTURE 2 GESTURE 3

If A and B then do X1 and X3 A B Partial conditions (Boolean) C D A B Partial motions (servos or servo sequences) - X3 - X4 X1 previous Partial motions (servos or servo sequences) X3 previous -

If A and C’ and D’ then do X1, X2 and X3 B Partial conditions (Boolean) C D A X1 Does not care Partial conditions (Boolean) X2 Partial motions (servos or servo sequences) C’ X3 D’ X4 X1 X2 Partial motions (servos or servo sequences) X3 previous -

Behaviors as trees Braitenberg Vehicle Condition A Braitenberg Faces yes no Braitenberg Bipeds TURN RIGHT Condition B yes no Condition C yes no GO STRAIGHT TURN LEFT Go Back

Behaviors as strings A A’B’ A’ B C’ A’ B C condition Motion or pose Condition A Condition C Condition B yes no TURN RIGHT GO STRAIGHT TURN LEFT Go Back A TURN RIGHT A’B’ GO STRAIGHT A’ B C’ Go Back A’ B C TURN LEFT condition Motion or pose Condition can be generalized to a Boolean or Multivalued expression

Behaviors as state machines I am in a happy state Condition A Condition C Condition B yes no GO STRAIGHT TURN LEFT Go Back I am in an angry state

Dialog and Robot’s Knowledge

Robot-Receptionist Initiated Conversation Human Robot What can I do for you? Robot asks This represents operation mode

Robot-Receptionist Initiated Conversation Human Robot What can I do for you? I would like to order a table for two Robot asks

Robot-Receptionist Initiated Conversation Human Robot Smoking or non-smoking? Robot asks

Robot-Receptionist Initiated Conversation Human Robot Smoking or non-smoking? I do not understand Robot asks

Robot-Receptionist Initiated Conversation Human Robot Do you want a table in a smoking or non-smoking section of the restaurant? Non-smoking section is near the terrace. Robot asks

Robot-Receptionist Initiated Conversation Human Robot Do you want a table in a smoking or non-smoking section of the restaurant? Non-smoking section is near the terrace. A table near the terrace, please Robot asks

Human-Initiated Conversation Robot Hello Maria initialization Robot asks

Human-Initiated Conversation Robot Hello Maria What can I do for you? Robot asks

Human-Asking Human Robot Question Question Robot asks Human asks

Human-Asking Human Robot Yes, you ask a question. Question Human asks

Human-Asking Yes, you ask a question. What book wrote Lee? Human asks Robot Yes, you ask a question. What book wrote Lee? Human asks

Human-Asking I have no sure information. What book wrote Lee? Robot I have no sure information. What book wrote Lee? Human asks

Human-Asking I have no sure information. Try to guess. Human asks Robot I have no sure information. Try to guess. Human asks

Human-Asking Lee wrote book “Flowers”. Try to guess. Human asks Human Robot Lee wrote book “Flowers”. Try to guess. Human asks

Human-Asking Lee wrote book “Flowers”. This is not true. Human asks Robot Lee wrote book “Flowers”. This is not true. Human asks

Human-Teaching Human ends questioning Questioning finished Robot Questioning finished Human asks Human teaches “Questioning finished” Robot asks Thanks, I have a lesson

Human-Teaching Robot enters asking mode Questioning finished What can I do for you? Questioning finished Human asks Human teaches “Questioning finished” Robot asks Thanks, I have a lesson

Human-Teaching Human starts teaching Thanks, I have a lesson Robot What can I do for you? Thanks, I have a lesson Human asks Human teaches “Questioning finished” Robot asks Thanks, I have a lesson

Human-Teaching Human Robot Thanks, I have a lesson Yes Human teaches

Human-Teaching I give you question-answer pattern Yes Human teaches Robot I give you question-answer pattern Yes Human teaches

Human-Teaching Question pattern: What book Smith wrote? Yes Robot Question pattern: What book Smith wrote? Yes Human teaches

Human-Teaching Answer pattern: Smith wrote book “Automata Theory” Yes Robot Answer pattern: Smith wrote book “Automata Theory” Yes Human teaches

Human-Teaching Checking question: What book wrote Smith? Yes Robot Checking question: What book wrote Smith? Yes Human teaches

Human-Teaching Checking question: What book wrote Smith? Robot Checking question: What book wrote Smith? Smith wrote book “Automata Theory” Human teaches

Human-Teaching I give you question-answer pattern Yes Human teaches Robot I give you question-answer pattern Yes Human teaches

Human-Teaching Question pattern: Where is room of Lee? Yes Robot Question pattern: Where is room of Lee? Yes Human teaches

Human-Teaching Answer pattern: Lee is in room 332 Yes Human teaches Robot Answer pattern: Lee is in room 332 Yes Human teaches

Human-Checking what robot learned Lesson finished Human teaches “Lesson finished” Question Robot asks Human asks

Human-Checking what robot learned Lesson finished What can I do for you? Human teaches “Lesson finished” Question Robot asks Human asks

Human-Checking what robot learned Question What can I do for you? Human teaches “Lesson finished” Question Robot asks Human asks

Human-Asking Yes, you ask a question. Question Human teaches Robot Yes, you ask a question. Question Human teaches “Lesson finished” Question Robot asks Human asks

Human-Asking Yes, you ask a question. What book wrote Lee? Human asks Robot Yes, you ask a question. What book wrote Lee? Human asks

Human-Asking I have no sure information. What book wrote Lee? Robot I have no sure information. What book wrote Lee? Human asks

Human-Asking I have no sure information. Try to guess. Human asks Robot I have no sure information. Try to guess. Human asks

Human-Asking Lee wrote book “Automata Theory” Try to guess. Human asks Robot Lee wrote book “Automata Theory” Try to guess. Observe that robot found similarity between Smith and Lee and generalized (incorrectly) Human asks

Behavior, Dialog and Learning The dialog/behavior has the following components: (1) Eliza-like natural language dialogs based on pattern matching and limited parsing. Commercial products like Memoni, Dog.Com, Heart, Alice, and Doctor all use this technology, very successfully – for instance Alice program won the 2001 Turing competition. This is a “conversational” part of the robot brain, based on pattern-matching, parsing and black-board principles. It is also a kind of “operating system” of the robot, which supervises other subroutines.

Behavior, Dialog and Learning (2) Subroutines with logical data base and natural language parsing (CHAT). This is the logical part of the brain used to find connections between places, timings and all kind of logical and relational reasonings, such as answering questions about Japanese geography.

Behavior, Dialog and Learning (3) Use of generalization and analogy in dialog on many levels. Random and intentional linking of spoken language, sound effects and facial gestures. Use of Constructive Induction approach to help generalization, analogy reasoning and probabilistic generations in verbal and non-verbal dialog, like learning when to smile or turn the head off the partner.

Behavior, Dialog and Learning (4) Model of the robot, model of the user, scenario of the situation, history of the dialog, all used in the conversation. (5) Use of word spotting in speech recognition rather than single word or continuous speech recognition. (6) Continuous speech recognition (Microsoft) (7) Avoidance of “I do not know”, “I do not understand” answers from the robot. Our robot will have always something to say, in the worst case, over-generalized, with not valid analogies or even nonsensical and random.

Questions for Homeworks, Quizzes and Projects (1) What is Braitenberg Vehicle? Explain simple Braitenberg vehicles that operate on continuous (analog), binary and multiple-valued data. Explain a model of robot behavior based on a single Finite State Machine. Explain a model of robot behavior based on several communicating probabilistic Finite State Machines. Explain a natural language model for a robot based on probabilistic and deterministic state machines. What is the internal state of the robot? Explain the concept of a behavior editor for a humanoid robot. Find on internet about Eliza, Alice and other chatbots and explain how they can be integrated in a complete robotic theatre of humanoid robots. What is a single application of Machine Learning in robotics. Think about possible applications of Machine Learning in humanoid robot theatre. Draw a state diagram of FSM controlling a Braitenberg Vehicle type of robot with three inputs that can be switched from Aggressive to Shy behavior. Draw a state diagram of a humanoid simple robot that behaves similarly to Aggressive and Shy Braitenberg Vehicles.

Questions (2) What is the use of Avatar in robot design? 1 What is the use of Avatar in robot design? Give example of Multiple-Valued logic in Machine Learning applied to a robot. Invent a humanoid robot head that would behave similarly to any Braitenberg Vehicle. Draw figure and State Machine for servos. Explain simple Braitenberg vehicles that operate on fuzzy signals using Post Literals and MIN, MAX operators. Write a software program that would use natural language and probabilistic state machine, similarly to examples above. Design a robot FSM with three states : Happy, Unhappy and Neutral that would behave accordingly to one of the three above moods. Give concepts for a humanoid robot head with moveable eyes, eyebrows, ears, lips, cheeks and jaw. How to build it mechanically. Look critically at the pictures in this set of slides. How to control all servos of these head parts – give ideas and explain how to relate them to facial gestures. Think how to use Boolean Minimization to find generalized answers in question-answering. Apply Braitenberg Vehicle ideas to Jimmy robot. Apply Braitenberg Vehicle ideas to the robot from your project.