Probabilistic State Machines to describe emotions Happy state Ironic state Unhappy state “you are beautiful” / ”Thanks for a compliment” “you are blonde!” / ”I am not an idiot” P=1 P=0.3 “you are blonde!” / Do you suggest I am an idiot?” P=0.7 Human speaks to robot Robot speaks to human (will be in italic in next slides)

Facial Behaviors of Maria Do I look like younger than twenty three? Maria asks:  “yes”  “no” Response from a human: Maria smiles Maria frowns In word spotting mode human responses are short

Probabilistic Grammars for performances Who? What? Where? Speak ”Professor Perky”, blinks eyes twice Speak “In the classroom”, shakes head P=0.1 Speak “Was drinking wine” P=0.1 P=0.3 P=0.5 Speak ”Professor Perky” Speak ”Doctor Lee” Speak “in some location”, smiles broadly Speak “Was singing and dancing” P=0.5 P=0.1 …. P=0.1 A complete behavior of a robot described as a probabilistic (quantum entangled) machine

Human-controlled modes of dialog/interaction Robot asks Human teaches Human commandsHuman asks Robot performs “Hello Maria” “Thanks, I have a question” “Thanks, I have a lesson” “Thanks, I have a command” “Lesson finished” “Questioning finished” “Command finished” “Stop performance” “Question” initialization This slide shows only human speech controls, no robot responses showing its state

The next slides will illustrate the following robot behaviors: Robot Initiated Conversation.Robot Initiated Conversation. Human-Initiated Conversation.Human-Initiated Conversation. Human asking.Human asking. Human teaching.Human teaching. Human verifying what robot learned.Human verifying what robot learned. Robot Performing (text, music, movements). Robot Performing (text, music, movements). Game Playing. Game Playing. Other robot behaviors:

Robot-Receptionist Initiated Conversation Robot What can I do for you? Human Robot asks This represents operation mode

Robot-Receptionist Initiated Conversation Robot What can I do for you? Human I would like to order a table for two Robot asks

Robot-Receptionist Initiated Conversation Robot Smoking or non- smoking? Human Robot asks

Robot-Receptionist Initiated Conversation Robot Smoking or non- smoking? Human I do not understand Robot asks

Robot-Receptionist Initiated Conversation Robot Do you want a table in a smoking or non-smoking section of the restaurant? Non-smoking section is near the terrace. Human Robot asks

Robot-Receptionist Initiated Conversation Robot Do you want a table in a smoking or non-smoking section of the restaurant? Non-smoking section is near the terrace. Human A table near the terrace, please Robot asks

Human-Initiated Conversation Robot Human Hello Maria Robot asks initialization

Human-Initiated Conversation Robot Human Hello Maria What can I do for you? Robot asks

Human-Asking Robot Human Question Human asks Question Robot asks

Human-Asking Robot Human Question Human asks Yes, you ask a question.

Human-Asking Robot Human What book wrote Lee? Human asks Yes, you ask a question.

Human-Asking Robot Human What book wrote Lee? Human asks I have no sure information.

Human-Asking Robot Human Try to guess. Human asks I have no sure information.

Human-Asking Robot Human Try to guess. Human asks Lee wrote book “Flowers”.

Human-Asking Robot Human This is not true. Human asks Lee wrote book “Flowers”.

Human-Teaching Robot Human Questioning finished Human teaches “Questioning finished” Robot asks Human asks Thanks, I have a lesson Human ends questioning

Human-Teaching Robot Human Questioning finished Human teaches “Questioning finished” Robot asks Human asks Thanks, I have a lesson Robot enters asking mode What can I do for you?

Human-Teaching Robot Human Thanks, I have a lesson Human teaches “Questioning finished” Robot asks Human asks Thanks, I have a lesson Human starts teaching What can I do for you?

Human-Teaching Robot Human Thanks, I have a lesson Yes Human teaches

Human-Teaching Robot Human I give you question- answer pattern Yes Human teaches

Human-Teaching Robot Human Question pattern: What book Smith wrote? Yes Human teaches

Robot Human Answer pattern: Smith wrote book “Automata Theory” Yes Human teaches Human-Teaching

Human-Teaching Robot Human Checking question: What book wrote Smith? Yes Human teaches

Human-Teaching Robot Human Checking question: What book wrote Smith? Smith wrote book “Automata Theory” Human teaches

Human-Teaching Robot Human I give you question- answer pattern Yes Human teaches

Human-Teaching Robot Human Question pattern: Where is room of Lee? Yes Human teaches

Human-Teaching Robot Human Answer pattern: Lee is in room 332 Yes Human teaches

Human-Checking what robot learned Robot Human Lesson finished Human asks Question Robot asks Human teaches “Lesson finished”

Human-Checking what robot learned Robot Human Lesson finished Human asks Question Robot asks Human teaches “Lesson finished” What can I do for you?

Human-Checking what robot learned Robot Human Question Human asks Question Robot asks Human teaches “Lesson finished” What can I do for you?

Human-Asking Robot Human Question Human asks Question Robot asks Human teaches “Lesson finished” Yes, you ask a question.

Human-Asking Robot Human What book wrote Lee? Human asks Yes, you ask a question.

Human-Asking Robot Human What book wrote Lee? Human asks I have no sure information.

Human-Asking Robot Human Try to guess. Human asks I have no sure information.

Human-Asking Robot Human Try to guess. Human asks Lee wrote book “Automata Theory” Observe that robot found similarity between Smith and Lee and generalized (incorrectly)

How we linked Behavior, Dialog and Learning The dialog/behavior has the following components: –(1) Eliza-like natural language dialogs based on pattern matching and limited parsing. Commercial products like Memoni, Dog.Com, Heart, Alice, and Doctor all use this technology, very successfully – for instance Alice program won the 2001 Turing competition. –This is a “conversational” part of the robot brain, based on pattern-matching, parsing and black-board principles. –It is also a kind of “operating system” of the robot, which supervises other subroutines.

(2) Subroutines with logical data base and natural language parsing (CHAT). –This is the logical part of the brain used to find connections between places, timings and all kind of logical and relational reasonings, such as answering questions about Japanese geography. (3) Use of generalization and analogy in dialog on many levels. –Random and intentional linking of spoken language, sound effects and facial gestures. –Use of Constructive Induction approach to help generalization, analogy reasoning and probabilistic generations in verbal and non-verbal dialog, like learning when to smile or turn the head off the partner. Behavior, Dialog and Learning

(4) Model of the robot, model of the user, scenario of the situation, history of the dialog, all used in the conversation. (5) Use of word spotting in speech recognition rather than single word or continuous speech recognition. (6) Avoidance of “I do not know”, “I do not understand” answers from the robot. –Our robot will have always something to say, in the worst case, over-generalized, with not valid analogies or even nonsensical and random. Behavior, Dialog and Learning

Generalization of the Ashenhurst- Curtis decomposition model

This kind of tables known from Rough Sets, Decision Trees, etc Data Mining

Decomposition is hierarchical At every step many decompositions exist

Constructive Induction: Technical Details U. Wong and M. Perkowski, A New Approach to Robot’s Imitation of Behaviors by Decomposition of Multiple-Valued Relations, Proc. 5 th Intern. Workshop on Boolean Problems, Freiberg, Germany, Sept , 2002, pp A. Mishchenko, B. Steinbach and M. Perkowski, An Algorithm for Bi-Decomposition of Logic Functions, Proc. DAC 2001, June 18-22, Las Vegas, pp A. Mishchenko, B. Steinbach and M. Perkowski, Bi- Decomposition of Multi-Valued Relations, Proc. 10 th IWLS, pp , Granlibakken, CA, June 12-15, IEEE Computer Society and ACM SIGDA.

Decision Trees, Ashenhurst/Curtis hierarchical decomposition and Bi-Decomposition algorithms are used in our software These methods create our subset of MVSIS system developed under Prof. Robert Brayton at University of California at Berkeley [2]. – The entire MVSIS system can be also used. The system generates robot’s behaviors (C program codes) from examples given by the users. This method is used for embedded system design, but we use it specifically for robot interaction. Constructive Induction

Additional Slides with Background

Ashenhurst Functional Decomposition Evaluates the data function and attempts to decompose into simpler functions. if A  B = , it is disjoint decomposition if A  B  , it is non-disjoint decomposition B - bound set A - free set F(X) = H( G(B), A ), X = A  B X

A Standard Map of function ‘z’ Bound Set Free Set a b \ c z Columns 0 and 1 and columns 0 and 2 are compatible column compatibility = 2 Explain the concept of generalized don’t cares

NEW Decomposition of Multi- Valued Relations if A  B = , it is disjoint decomposition if A  B  , it is non-disjoint decomposition F(X) = H( G(B), A ), X = A  B Relation A B X

Forming a CCG from a K-Map z Bound Set Free Set a b \ c Columns 0 and 1 and columns 0 and 2 are compatible column compatibility index = 2 C1C1 C2C2 C0C0 Column Compatibility Graph

Forming a CIG from a K-Map Columns 1 and 2 are incompatible chromatic number = 2 z a b \ c C1C1 C2C2 C0C0 Column Incompatibility Graph

A unified internal language is used to describe behaviors in which text generation and facial gestures are unified. This language is for learned behaviors. Expressions (programs) in this language are either created by humans or induced automatically from examples given by trainers. Constructive Induction

The integrated approach to robot vision and speech based dialogs

Open CV image processing software from Intel

Hidden Markov Model Based Face Recognition

Braitenberg Vehicles Two sensors two motors Many behaviors from simple rules Control can be a combinational mapping or automaton Combinational logic can be binary, multiple- valued, fuzzy and quantum. Automaton can be binary, multiple-valued, fuzzy, probabilistic, non-deterministic or quantum (entangled) This is a new concept in Machine Learning and Robotics

AND QUANTUM BREITENBERG FACES

Hadamard gate In standard computers probabilistic components are expensive and have aliasing. In quantum these are the cheapest gates and they are ideal random number generators.

Square Root of NOT Deterministic behaviors can be composed from probability waves. This does not happen outside quantum world. Square root controlled gate and its matrix Square root controlled hermitian and its matrix Two Square root gates composed to an inverter

Analysis of a Quantum Circuit. Matrices in Hilbert Space Analysis of a Quantum Circuit. Matrices in Hilbert Space

Kronecker Product for Quantum Circuit Analysis Parallel connection of gates requires Kronecker Product. Serial connection of gates (Previous slides) requires standard matrix multiplication.

Analysis of a Quantum Circuit Analysis of a Quantum Circuit

Conclusion. What did we learn (1) the more degrees of freedom the better the animation realism. (2) synchronization of spoken text and head (especially jaw) movements are important but difficult. (3) gestures and speech intonation of the head should be slightly exaggerated.

Conclusion. What did we learn(cont) (4) the sound should be laud to cover noises coming from motors and gears and for a better theatrical effect. (5) noise of servos can be also reduced by appropriate animation and synchronization. (6) best available ATR and TTS packages should be applied, especially those that use word spotting. (7) use puppet theatre experiences.

(8) because of a too slow learning, improved parameterized learning methods will be developed, but also based on constructive induction. (9) open question: funny versus beautiful. (10) either high quality voice recognition from headset or low quality in noisy room. YOU CANNOT HAVE BOTH WITH CURRENT ATR TOOLS. The bi-decomposer of relations and other useful software used in this project can be downloaded from Conclusion. What did we learn(cont)

Conclusion Monday, May 10, Intelligent Robotics Laboratory and Industrial Robotics Laboratory. Demo 10am- 2pm Thursday, May 13, Convention Center. Demo 10am- 2pm Sunday, June 6, PSU Balroom Smith Center. Competition and Demo 10am- 2pm Help needed. If you can program, interface PCs, know about networks, want to help with WWW Page, build robots, learn advanced theories, ……. You are welcome to the lab.