1. Relational Symbol Grounding through Affordance Learning: An Overview of the ReGround Project
Laura Antanas1, Ozan Arkan Can2, Jesse Davis1, Luc De Raedt1, Amy Loutfi3, Andreas Persson3, Alessandro Saffiotti3, Emre Unal2, Deniz Yuret2, Pedro Zuidberg Dos Martires1
1KU Leuven, Belgium
2Koc University, Turkey
3Orebro University, Sweden

2. Motivation Pick the red apple next to the black mug
and cut it into slices
Among other things, this requires
Identifying which objects “apple” and “mug” refer to
Understanding the relation “next to”
Knowing that it is possible to cut an apple and how to cut it

3. ReGround Project’s Hypotheses
Pick the red apple next to the black mug
Cut it into slices
Language
Vision
Spatial
relation
Apple
affords
cutting
Four tenets of ReGround
Grounding requires integrating from multiple modalities
Identifying symbols is only the first step of grounding
Learning affordances is the second, underexplored step of grounding
Exploiting relationships and affordances will improve grounding
Distinguish what the task requires and what we want to do …
Learning affordances is helping the grounding … and is the central point of reground
-It would be good to have a good scenario and then to highlight the specific goals of reground (that is, learning and using affordances to help grounding)
Cut the apple (without mentioning the knife) – we need to know that an apple affords being cut with a knife (this is a kind of planning simple
Sequences of actions – take the knife, take the apple, cut … )
Grounding should integrate information from multiple modalities to consider the full environmental context to
Identify symbols and their referents
Account for the relationship between multiple symbols and multiple referents in the environment
Learn affordances, that is, what action possibilities for an object in an environment exist

4. Main Components Object anchoring from perceptual data
Natural language processing and grounding
Relational affordance learning
Planning

5. Object Anchoring Given: Image of world Do: Interpret image to
Goal: Create and maintain a consistent world model of perceived objects
Given: Image of world
Do: Interpret image to
Identify each object
Extract attributes about it
Assign it a symbolic id
This involves numerous steps. The Image has to be segmented to identify objects, the objects are then categorized using a convolutional neural network. Attributes could be things like shape, position or color. For example, color could be represented as a histogram in the HSV color space [hue, saturation, value]

6. Anchoring: Acquiring, Reacquiring, and Tracking [Perrson et al., Front. In Rob. 2017; Perrson et al. in prep] T
T+1
T+2
T+3
T+4
Anchor Apple-1
Anchor Skin-1
Anchor Skin-1
Anchor Skin-1
Anchor Apple-1
Standard
Anchoring
Permit
reasoning
The relational information also you to
Knowing that the relationship onTopOf holds after the hand is over the apple allows the system to reason about what might be below the hand. Add pop up.
Anchor Apple-1
Anchor Apple-1
Anchor Apple-1
Anchor Apple-1
Anchor Apple-1
With
Relations

7. Natural Language Processing and Grounding
Goal: Relate multiple symbols with multiple referents in the environment
Pick the red apple next to the black mug
Given:
Statement in natural language
Perceptual information from anchors
Do: Ground words to objects, properties, relations, and actions
This involves numerous steps. The Image has to be segmented to identify objects, the objects are then categorized using a convolutional neural network. Attributes could be things like shape, position or color. For example, color could be represented as a histogram in the HSV color space [hue, saturation, value]

8. GRID WORLD REPRESENTATION
Approach: Neural Language Grounder [Bisk et al., NAACL’16; Can & Yuret, in prep]
Locate
Predict
Location of referent
in the environment
Next To
Shift
Attention
Relation
And
Combine
Pick the red apple next to the black mug
And
Combine
Red
Apple
Mug
Detect
The example has been changed … is it needed ?
The system can cope with ambiguity – if there are two apples, they both will be marked
GRID WORLD REPRESENTATION

9. Relational Affordance Learning
Goal: Learn and reason about combinations of primitives (e.g., object-action, action-effect, etc.) that are possible in the environment
Given:
Facts about the objects in the world
Positive and negative examples of affordances
Learn: Rules that describe when affordances apply
0.99::Object(apple-3,apple)
0.99::Object(apple-2, pear)
0.98::Object(knife-1, knife)
0.8::is(knife, sharp)
0.9::is(pear, soft)
This involves numerous steps. The Image has to be segmented to identify objects, the objects are then categorized using a convolutional neural network. Attributes could be things like shape, position or color. For example, color could be represented as a histogram in the HSV color space [hue, saturation, value]
0.95::affords(knife, cut, pear)
0.02::affords(knife, cut, cup)

10. Probabilistic Relational Rule Learning
Three key innovations:
Learn rules from probabilistic examples and data [De Raedt et al., IJCAI’15]
Learn relational affordances [Antanas et al., ILP’17]
Learn relational rules in dynamic worlds [Nitti et al., ECAI’16]
0.95::affords(knife, cut, pear)
0.02::affords(knife, cut, cup)
0.75::affords(X, cut, Y) ← is(X,sharp), can(X, grasp), is(Y, fruit)
It is better to say that this is corresponding to a kind of STRIPS or PDDL like notion of affordance. It is going to be used for planning simple sequences of actions
Examples may be a bit too far from the other examples right now.
Pre & Post
Conditions
isSlicedt+1(Y) ← affordst(X, Cut, Y), cutt(X,Y)
Use to plan

11. Planning [Nitti et al., MLJ in press]
Given: Learned affordances, world state, and goal
Derive: A plan to accomplish the goal
Cutting an apple would mean
Picking up an apple on putting it on a cutting board
Finding and picking up the knife
Grasping the apple and cutting it

12. Evaluation Setup

13. Conclusions Pursuing an approach to grounding centered on
Understanding the full environment
Capturing relationships
Affordances
Three pronged approach:
Perceptual anchoring of objects
Symbol grounding for language
Learning affordances from language and percepts
See:

14. Questions?

15. Conclusions Pursuing an approach to grounding centered on
Understanding the full environment
Capturing relationships
Affordances
Three pronged approach:
Perceptual anchoring of objects
Symbol grounding for language
Learning affordances from language and percepts
See:

16. Object Anchoring
Goal: Create and maintain a consistent world model of perceived objects
Given:
Do:

17. Relational Affordance Learning
Goal: Learn and reason about combinations of primitives (e.g., object-action, action-effect, etc.) that are possible in the environment
Given:
Positive and negative examples of affordances
Facts about the world
Learn: Rules that describe when affordances are applicable
This involves numerous steps. The Image has to be segmented to identify objects, the objects are then categorized using a convolutional neural network. Attributes could be things like shape, position or color. For example, color could be represented as a histogram in the HSV color space [hue, saturation, value]

18. Experimental setup / Current State
See input

19. Relational Affordance Learning
Goal: Learn and reason about combinations of primitives (e.g., object-action, action-effect, etc.) that are possible in the environment
Given:
Positive and negative examples of affordances
Facts about the world
Learn: Rules that describe when affordances are applicable
This involves numerous steps. The Image has to be segmented to identify objects, the objects are then categorized using a convolutional neural network. Attributes could be things like shape, position or color. For example, color could be represented as a histogram in the HSV color space [hue, saturation, value]

20. Learning Relational Affordances
0.99::Object(apple-3,apple)
0.99::Object(apple-2, pear)
0.98::Object(knife-1, knife)
0.8::is(knife, sharp)
0.9::is(pear, soft)

21. Motivation and Context
CHISTA-ERA Project:
Attacker

22. Probabilistic Relational Rule Learning
Learn rules: p::target(…) ← body
p is a probability
target is a specific affordance
body is a conjunction of literals
Learning procedure: Learn one rule at a time
Each rule learned greedily
0.03::affords(X, cut, Y) ←
0.75::affords(X, cut, Y) ← is(X,sharp), allows(X, grasp), is(Y, fruit)

23. Probabilistic Relational Rule Learning
0.03::affords(X, cut, Y) ←
0.08::affords(X, cut, Y) ←
is(X,sharp)
0.08::affords(X, cut, Y) ←
is(X, dull)
0.08::affords(X, cut, Y) ←
is(X, sharp), allows(X, grasp)
0.08::affords(X, cut, Y) ←
is(X, sharp), is(Y, fruit)
0.75::affords(X, cut, Y) ← is(X,sharp), allows(X, grasp), is(Y, fruit)

24. Overview of the Envisioned Approach
Language grounding
Object
Anchoring
Language Grounding
Vision
Language
Relational
Language
Grounding
Environment
Relational
Anchoring
Might be cut ?? Too business like …
Domain
Knowledge
Plan
Relational
Language
Description
Relational
Perceptual
Description
Relational Affordance Learning

25. ReGround Project’s Hypotheses
Language
Vision
Spatial
relation
Pick the knife to the right of the apple
Cut it into slices
knife
affords
cutting
Grounding should integrate information from multiple modalities to consider the full environmental context to
Identify symbols and their referents
Account for the relationship between multiple symbols and multiple referents in the environment
Learn affordances, that is, what action possibilities for an object in an environment exist
Distinguish what the task requires and what we want to do …
Learning affordances is helping the grounding … and is the central point of reground
-It would be good to have a good scenario and then to highlight the specific goals of reground (that is, learning and using affordances to help grounding)
Cut the apple (without mentioning the knife) – we need to know that an apple affords being cut with a knife (this is a kind of planning simple
Sequences of actions – take the knife, take the apple, cut … )
The slide at the end with the four tenets Is preferred by some people