1. WP 4 Language Emergence Britta Wrede (BIEL)
Gerhard Sagerer, Katharina Rohlfing, Karola Pitsch, Katrin Lohan, Lars Schillingmann, BIEL
Jun Tani, RIKEN
Stefano Nolfi, CNR
Angelo Cangelosi, Martin Peniak PLYM
Chrystopher Nehaniv, Kerstin Dautenhahn, Yo Sato, Joe Saunders, Frank Förster, Caroline Lyon, UH
Kerstin Fischer, Arne Zeschel, USD

2. Overview
Task 4.1
Generalization as a basis for emergence of symbolic systems (start: M7)
Task 4.2
Acoustic Packaging and the learning of words (start: M13)
Task 4.3
From single word lexicons to compositional languages (start: M13)
Task 4.4
Constructional grounding and primary scenes (start: M19)
Task 4.5
Evolutionary origins of action nd language compositionality (start: M31)
ITALK Year 1 Review Düsseldorf, 30 June 2009

3. Objectives & Goals 4.4 4.3 4.2 4.1 Speech Action
Grammatical Constructions
4.4
Lexicon Construction
Speech
4.3
Acoustic
Packages
4.2
Action
Action Hierarchy
4.1
ITALK Year 1 Review Düsseldorf, 30 June 2009

4. Objectives Year 1
Robotics experiments on the development of action categorisation as a basis for linguistic communicative capabilities
Analysis of the role of temporal synchrony between speech and action as a basis for the implementation of a module that fuses two modalities to segment actions
ITALK Year 1 Review Düsseldorf, 30 June 2009

5. Task 4.1 Generalization as a basis for emergence of symbolic systems (Start: M7)
Emergence of action structure through use of slow vs fast context units in a Multiple Timescale Recurrent Neural Network (MTRNN)
Fast context units are able to find primitives
Slow context units are able to sequence primitives without explicit learning
ITALK Year 1 Review Düsseldorf, 30 June 2009

6. Compositionality in Action Generation
How to acquire set of reusable behavior primitives to be combined to generate actions? a b c
Homunculus?? a b c
Reparatory of primitives b a a c b
Combined streams
ITALK Year 1 Review Düsseldorf, 30 June 2009

7. Method: Self-Organization of Functional Hierarchy
[Yamashita & Tani, 2008]
Behavioral Compositionality MTRNN
τ= 50.0
slow time constant
τ= 5.0
Fast time constant
Slow
Fast
time
Teach
ITALK Year 1 Review Düsseldorf, 30 June 2009

8. 4.1 Summary
Emergence of action structure through fast and slow context units
Slow context units are able to learn composition / sequencing of primitives
4.1 Outlook
Extend Tani’s Recurrent Networks, e.g. using (pre)trained networks on actions and teach words for action and object categories/properties
Evaluation of MTRNN on Motionese Corpus (input: hand trajectories)
Hypothesis about relation of action hierarchy with language hierarchy: verbs related to slow context units, objects to fast context units

9. Overview 4.4 4.3 4.2 4.1 Speech Action Grammatical Constructions
Lexicon Construction
Speech
4.3
Acoustic
Packages
4.2
Action
Action Hierarchy
4.1
ITALK Year 1 Review Düsseldorf, 30 June 2009

10. 4.2 Acoustic Packaging Background (Start: M13)
How to associate information in different modalities for language learning?
Synchrony [Zukow-Goldring, 1997] [Matatyaho, Mason & Gogate et al., 2007]
Synchronous object movement and verbal labeling enhances object learning
More low-level synchrony in ACI than in AAI [Rolf et al., 2009]
Acoustic Packaging [Brand et al, 2007]
Synchrony between language and events helps to divide sequence of events into units [Hirsh-Pasek & Golinkoff, 1996]
Speech segment determines perceived (end of) action

11. Acoustic Packaging [Brand et al., 2007]
Question: Does speech influence how action is structured by infants?
Experiment: 32 Infants of 7.5 – 11.5 months of age; Preferential Looking A B C
Wow! Do you see what she‘s doing? She‘s blixing!
Audio
Vision
Familia- rization
Preferred Sequence A B
Test: Split Screen
Non-packaged sequence perceived as new
Speech structures action !

12. Computational Model of Acoustic Packaging [Schillingmann et al
Computational Model of Acoustic Packaging [Schillingmann et al., 2009, best paper award at the ICDL09]
Long term goals
Temporal segmentation of actions
Generating appropriate feedback
Integration with imitation learning approaches
Evaluation
Does model reflect structural properties of tutoring behavior?
ITALK Year 1 Review Düsseldorf, 30 June 2009

13. Computational Model of Acoustic Packaging
Segmentation
Speech: by ASR (ESMERALDA)
Temporal Association
Acoustic Package created if segments overlap
Action: by motion history images
ITALK Year 1 Review Düsseldorf, 30 June 2009

14. Computational Model of Acoustic Packaging
ITALK Year 1 Review Düsseldorf, 30 June 2009

15. Computational Model of Acoustic Packaging
ITALK Year 1 Review Düsseldorf, 30 June 2009

16. Evaluation
Data
Videos from Motionese corpus (11 AAI, 11 ACI) and from babyface study (11 ARI)
Task: stacking cups
Analysis
Automatic detection of Acoustic Packages
Measurements:
number of Acoustic Packages (#AP)
mean number of motions per Acoustic Package (#motions / AP)
Hypothesis
ACI more structured than AAI
More #AP and less #motions / AP in ACI
ITALK Year 1 Review Düsseldorf, 30 June 2009

17. Results Sig. more Acoustic Packages in ACI and ARI
Sig. less Motions per Acoustic Packages in ACI and ARI
Hypothesis confirmed
Automatically detected Acoustic Packages find more structure in ACI and ARI
ITALK Year 1 Review Düsseldorf, 30 June 2009

18. 4.2 Outlook
Interaction
Designing feedback of iCub robot based on AP for user studies (Live-Demo!)
Hyp. 1: Signalling detected AP ends to user will increase contingency in interaction
Hyp. 2: APs are related to action structure and therefore signal understanding
ITALK Year 1 Review Düsseldorf, 30 June 2009

19. 4.2 Outlook Learning Use AP as units for Speech learning
Action learning
Comparison with other segmentation approaches (e.g. MTRNNs)
ITALK Year 1 Review Düsseldorf, 30 June 2009

20. Overview 4.4 4.3 4.2 4.1 Speech Action Grammatical Constructions
Lexicon Construction
Speech
4.3
Acoustic
Packages
4.2
Action
Action Hierarchy
4.1
ITALK Year 1 Review Düsseldorf, 30 June 2009

21. 4.3 From single word to compositional lexicons (Start: M13)
Relation to Task 1.4 – Hierarchical Actions with RNNPB
In T1.4 the main aim is to develop neural architectures capable of learning compositional and hierarchical actions
This is currently being investigated through the use of Tani’s Recurrent Neural Network : (i) RNNPB with Parametric Bias for hierarchical architecture and (ii) TRNN fully time recurrent for emergence hierarchy.
Future Work in Task 4.3
Extend Tani’s Recurrent Networks, e.g. using (pre)trained networks on actions and teach words for action and object categories/properties

22. 4.3 and 4.4 Grammar learning in children
Tomasello (2003):
From holophrases… lemme-see (=let me see)
…to pivot schemas… lemme Z (=let me Z)
…to item-based constructions… X let Y Z
…to abstract constructions SUBJ VERB OBJ COMP

23. 4.3 and 4.4 Grammar learning in children
Tomasello (2003):
From holophrases… lemme-see
…to pivot schemas… lemme Z
…to item-based constructions… X let Y Z
…to abstract constructions SUBJ VERB OBJ COMP
starting point: concrete, simplex

24. 4.3 and 4.4 Grammar learning in children
Tomasello (2003):
From holophrases… lemme-see
…to pivot schemas… lemme Z
…to item-based constructions… X let Y Z
…to abstract constructions SUBJ VERB OBJ COMP
starting point: concrete, simplex
end point: abstract, complex

25. 4.3 and 4.4 Grammar learning in children
Tomasello (2003):
From holophrases… lemme-see
…to pivot schemas… lemme Z
…to item-based constructions… X let Y Z
…to abstract constructions SUBJ VERB OBJ COMP
starting point: concrete, simplex
end point: abstract, complex
in between: increasing abstraction & complexity

26. Argument structure constructions
4.3 and 4.4 Grammar learning
Argument structure constructions
Elementary blueprints for predicating basic event types
intransitive He walks N V
intransitive motion He walks through the park N V OBL
transitive He walks the dog N V N
transitive motion He walks the dog through the N V N OBL park
Holistic constructions are associated with semantic frames
Formal constituents in syntax map to conceptual constituents in event structure

27. 4.3 and 4.4 Grammar learning Sem CAUSE-MOVE AGT THM LOC
(Adapted from Goldberg 1995)
Sem CAUSE-MOVE AGT THM LOC
Syn V SUBJ OBJ OBL

28. Semantics: CAUSED MOTION frame
4.3 and 4.4 Grammar learning
(Adapted from Goldberg 1995)
Semantics: CAUSED MOTION frame
Sem CAUSE-MOVE AGT THM LOC
Syn V SUBJ OBJ OBL

29. Semantics: CAUSED MOTION frame
4.3 and 4.4 Grammar learning
(Adapted from Goldberg 1995)
Semantics: CAUSED MOTION frame
Sem CAUSE-MOVE AGT THM LOC
Syn V SUBJ OBJ OBL
Syntax: Complex transitive complementation

30. Semantics: CAUSED MOTION frame
4.3 and 4.4 Grammar learning
(Adapted from Goldberg 1995)
Semantics: CAUSED MOTION frame
Sem CAUSE-MOVE AGT THM LOC
Syn V SUBJ OBJ OBL
Syntax: Complex transitive complementation
Correspondence links

31. 4.3 and 4.4 Planned Experiments
Associate holistic utterance with complex event structure
→ holophrase learning
Segment/decompose sequence into constitutive elements
→ word/morpheme learning
Identify recurrent functional relationships between constituent markings and meanings
→ construction learning

32. 4.3 and 4.4 Planned Experiments
Generalisation within construction (Replication Sugita & Tani, 2005) V N
Generalisation across 2 constructions V | V N V N | V N OBL push block kick ball to-the-box push block to-the-box
Generalisation across N+1 constructions (increase complexity) V N | V N OBL | V | V N N
Experiment with acquisition order
Generalisation across N+1 constructions with empirically motivated statistical biases
ITALK Year 1 Review Düsseldorf, 30 June 2009

33. 4.3 and 4.4 Empirically motivated statistical biases
Experimentally vary…
Statistical learning cues available to learner, e.g.
relative constructional frequencies in the input
lexical type frequencies per constructional slot
availability of ‘pathbreaking’ verbs for particular event types
amount of lexical overlap between constructions
→ input modeled on basis of WP 3.1 grammar classifications
Effect of combining the availability of linguistic input-statistical with tutor-provided paralinguistic cues
This work is complementary to the approach using the social learning architecture ROSSUM and predictive grammar inductions (Yo Sato‘s) as seen in the presentation of WP3.2.
The data used here will also be used by Herfordshire is in the human-humanoid tutoring over iterated interaction scenarios.
ITALK Year 1 Review Düsseldorf, 30 June 2009

34. WP4 Summary
Extension of the work at RIKEN on the use of generalisation as a basis for compositional linguistic communication
Multiple timescales recurrent neural network (MTRNN) for action and language learning experiments.
For experiments planned jointly by PLYM and USD on compositional action and language learning
Analysis of the phenomenon of synchrony between verbal utterances and action for an objective measurement of synchrony in multimodal behaviour
Integration of acoustic packaging modules on the iCub platform using the XCF integration framework
ITALK Year 1 Review Düsseldorf, 30 June 2009

35. WP4 Outlook
Interaction studies to analyse feedback produced wrt Acoustic Packages
Use MTRNN for joint learning of action and speech as a basis for construction learning
Data analysis and first modeling approaches of construction grammar
ITALK Year 1 Review Düsseldorf, 30 June 2009

36. Thank you! 4.4 4.3 4.2 4.1 Speech Action Grammatical Constructions
Lexicon Construction
Speech
4.3
Acoustic
Packages
4.2
Action
Action Hierarchy
4.1
ITALK Year 1 Review Düsseldorf, 30 June 2009

37. Thank you!
ITALK Year 1 Review Düsseldorf, 30 June 2009