1. Chapter 17. The Progress drive hypothesis Course: Robots Learning from Humans Yoon, Bo Sung Department of Economics Seoul National University http://bi.snu.ac.kr

2. Contents Introduction Progress driven learning Developmental mechanisms for early imitation Conclusion 2

3. Introduction © 2015, SNU CSE Biointelligence Lab., http://bi.snu.ac.kr 3 progress-driven learning could help to understand why children focus on specific imitative activities at a certain age how they progressively organize preferential interactions with particular entities present in their environment.

4. Progress Drive Learning System a progress drive hypothesis early imitation phenomena, including self-imitation and simple interpersonal co-ordination is an 1)intrinsic motivation system driving the infant into situations expected to result in 2) maximal learning progress progress-driven learning system a critic capable of producing internal rewards in order to guide the agent towards learning new skills agent to acts in order to be in situations in which its error in prediction decreases maximally fast. © 2015, SNU CSE Biointelligence Lab., http://bi.snu.ac.kr 4

5. Prediction and Meta-prediction © 2015, SNU CSE Biointelligence Lab., http://bi.snu.ac.kr 5  s : state  a : action  y : actual outcome y’ : expected outcome  e : actual error ( = y – y’) e’ : expected error  P(s, a) : prediction system  Meta P (s, a) : meta prediction system

6. Examples of Learning Systems  progress-driven learning system  In this system, agent acts in order to be in situations in which its error in prediction decreases maximally fast. → take action to maximize the expected process p’(t)  Mastery-driven systems  In this system, agent acts in order to be in situations in which its error in prediction is minimal. → take action to make “ e’ ” minimal (e = y – y’)  novelty-driven systems  In this system, agent chooses actions leading to situations in which its error in prediction is maximal. → take action to make “ e’ ” maximal (e = y – y’) © 2015, SNU CSE Biointelligence Lab., http://bi.snu.ac.kr 6

7. Screen Problem © 2015, SNU CSE Biointelligence Lab., http://bi.snu.ac.kr 7  s (state) = y (outcome) : 9-D vector including 9 binary value (0 or 1) corresponding to nine pixels  A (action) = 4-D vector including 4 binary value (0 or 1) corresponding 4 buttons  notation  s : state  a : action  y : actual outcome y’ : expected outcome  e : actual error ( = y – y’) e’ : expected error  P(s, a) : prediction system → y’  Meta P (s, a) : meta prediction → e’ system

8. Progress Niches The progress drive pushes the agent to discover and focus on situations which lead to maximal learning progress. These situations, neither too predictable nor too difficult to predict, are ‘progress niches’. © 2015, SNU CSE Biointelligence Lab., http://bi.snu.ac.kr 8

9. Screen Problem © 2015, SNU CSE Biointelligence Lab., http://bi.snu.ac.kr 9

10. Neonate imitation (0m) : Like-me stance infants will indeed create a discrimination between predictable couplings between modals(what is seen and what is felt) and unpredictable situations (all the other cases) they will focus on the first zone of their sensorimotor space that constitutes a ‘progress niche’. © 2015, SNU CSE Biointelligence Lab., http://bi.snu.ac.kr 10 Developmental Mechanisms for Early Imitation

11. Self-imitation(1 – 2m) : Circular Reaction During the first two months of their life, infants perform repeated body (sucking its fingers, shaking legs) children are structuring their own behaviour in order to make it more predictable and in this way form ‘circular reactions’ motion self-centred types of behaviour are ‘progress niches’. © 2015, SNU CSE Biointelligence Lab., http://bi.snu.ac.kr 11 Developmental Mechanisms for Early Imitation

12. Pseudo-imitation (2-4m) Infants become more attentive to the external world and particularly to people. Parents adapt their own responses so that interactions with the child follow the normal social rules that characterize communicative exchanges(ex. Taking turns )(Parental scaffolding). if an adult imitates an infant’s own actions, it can trigger continued activity in the infant. © 2015, SNU CSE Biointelligence Lab., http://bi.snu.ac.kr 12

13. developmental mechanisms for early imitation Interaction with Objects (5-7m) Ashifts again from people to objects. Children gain increased control over the manipulation of some objects on which they discover ‘affordances’ A progress-driven process can account for this discrimination between affordant objects and unmastered aspects of the environment. © 2015, SNU CSE Biointelligence Lab., http://bi.snu.ac.kr 13

14. Conclusion the development of imitative capabilities could be interpreted as a result of a progress driven process Early imitation can be seen as the result of a process by which an agent looks for ‘progress niches’ by picking up easy-to-predict aspects of its environment, by performing self-centred circular reactions by engaging in scaffolded social interactions By discovering particular affordances of certain objects. The existence of a progress drive could explain why certain types of imitative behaviour are produced by children at certain ages and stop being produced later on. how discrimination between actions oriented towards the self, towards others and towards the environment may occur. © 2015, SNU CSE Biointelligence Lab., http://bi.snu.ac.kr 14