1. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 1 of 25 Primatech: Models and Future Directions GVU Brownbag 9/30/1999 by Jim Davies, Tim Keenan, Wasinee Rungsarityotin & Irfan Essa {jimmyd, tk, wasinee, irfan} @cc.gatech.edu

2. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 2 of 25 What is Primatech? Chantek: An orangutan who can sign Will be on display at Zoo Atlanta; visitors can interact with him He will be overloaded. Primatech simulates having a conversation with him

3. Schematic

4. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 4 of 25 Perception module 150 Signs to recognize (Starner et al. recognized 40) Controlled environment (any background, etc.) Novice users (few signs used) Supporting speech data

5. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 5 of 25 Graphics module Memory of sign animations Home positions? “Essence” with on the fly transformations Remember every transition? (22500)

6. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 6 of 25 Chantek and trainer Lyn Miles

7. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 7 of 25 Using a gorilla model

8. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 8 of 25 An animated sign for “like”

9. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 9 of 25 Ape language research Language trained apes know 140 signs, deaf children of 3 know 132 Chantek –Dr. Lyn Miles –150 signs (strict criterion) –Anthropological approach

10. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 10 of 25 Three cognitive models 1: Simple activation with Wordnet 2: Express words that correspond to desire satisfaction 3: Sentence verification

11. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 11 of 25 Model 1: Simple activation with Wordnet ACT-R uses spreading activation Relevance links only (from WordNet) distinguish words, signs, concepts Retrieve anything, let spreading activation from goal do the work

12. Model 1: Representation Instead of: (banana ISA fruit color yellow) (yellow ISA color) I used: (cup-concept ISA concept word cup-word sign cup-sign r1 drink-concept) (cup-sign ISA sign meaning cup-concept) (cup-word ISA sign meaning cup-concept)

13. What do people ask? (Collected by Tara Stoinski) 1. What do you like to eat? (5 people asked) 2. What do you like to do? (3) 3. Are you bored? (3) 4. What is it like to have fur? (1) 5. Do you like the animals that you live with? (1) 6. What is it like to live in a cage? (1) 7. Do you mind people staring at you all the time? (1) 8. Do you have a favorite keeper? (1) 9. Are you happy? (7) 10. What do you want that you don't have? (4) 11. Do you want to go back to the wild? (2) 12. Tell me about evolution (1)

14. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 14 of 25 Model 1: Results 1. What do you like to eat? response: food-eat-sign 3. Are you bored? response: you-sign 5. Do you like the animals that you live with? response: you-sign 9. Are you happy? response: you-sign 11. Do you want to go back to the wild? response: go-sign

15. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 15 of 25 Why so bad? definitions too dictionary-like –banana n 1: any of several tropical and subtropical treelike herbs of the genus Musa having a terminal crown of large entire leaves and usually bearing hanging clusters of elongated fruits –What about being yellow? Eating it? Deliciousness? Only 20% of words linked “You” “eat” and “go” were the only words in questions that were in memory.

16. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 16 of 25 Model 2: Express words that correspond to desire satisfaction ACT-R has a goal structure but no way to express desires that are always around but no specifically acted upon all the time. Desires: food, drink, sex, freedom, and stimulation-- represented as memory chunks Words were linked to desires

17. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 17 of 25 Model 2: Relationships (screwdriver-word-meaning ISA relation2 relation meaning-relation first screwdriver-word second screwdriver-concept) (dog-isa-animal ISA relation2 relation subclass-relation first dog-concept second animal-concept)

18. Model 2: Input Representation (goal-1-1 ISA goal-talk sentence-id 1 word what-word next-word goal-1-2) (goal-1-2 ISA goal-talk sentence-id 1 word do-word next-word goal-1-3) (goal-1-3 ISA goal-talk sentence-id 1 word you-word next-word goal-1-4) (goal-1-4 ISA goal-talk sentence-id 1 word like-word next-word unknown)

19. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 19 of 25 Model 2: Results Often responds appropriately when offered something desired Linguistically still impoverished

20. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 20 of 25 Model 3: Sentence verification Implemented a simple grammar Part of speech relation added Walked through input sentence, putting words in sentence template Checks the memory for a fact that has the same subject, verb, and object If found, express verb

21. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 21 of 25 Model 3: Results Could verify relations found in the memory (Apple is a fruit, Chantek likes meat, Banana is yellow, etc.) Combination of Models 2 and 3 might account for much of his conversation

22. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 22 of 25 Cognitive Model contributions Input sentence representation Desire representation in ACT-R New relation representation/ relations can become activated themselves First computer modeling of ape language

23. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 23 of 25 Future directions Perception: Apply machine learning to signs, installation design Graphics: Collision detection, closer modeling of the ape’s actual motion Cognitive Model: Modeling transcript data, integration of models 2 and 3

24. http://www.cc.gatech.edu/gvu/perception/projects/primatech/ 24 of 25 Thanks Graphics, Visualization & Usability seed grant 98-99 Tara Stoinski for question collection Victor Zordan for graphics help Lyn Miles for Chantek

25. More information We are looking for collaborators! http://www.cc.gatech.edu/gvu/perception/ projects/primatech/ Email me at jimmyd@cc.gatech.edu