1. Perception & Imagination:
Grounding semantics of concrete and abstract concepts
CoNGA – Cognitive & Neuroscience Group, Antwerp
14th December
Giovanni Cassani
4A257E -> FFFFFF in 10 steps using HSV gradient

2. Multimodal distributional semantics
Bridging the gap between language and the world
Distributional Semantics : “You shall know a word by the company it keeps”
→ Similar concepts occur in similar contexts
BUT:
- perceptual info doesn’t appear in text (no orange carrots, trivial)
- how is the word-referent link established?
Lack of grounding…
…bring in info from images!
CoNGA – 14th December 2016: Perception & imagination: Grounding semantics of concrete & abstract concepts

3. The vector representation
A single framework to encode all information
Distributional Semantic Models (DSMs) encode each word as a numeric vector: each vector define the position of a word in a high-dimensional geometrical space (you can count or predict, doesn’t make a difference) computed in the same way from texts and images.
Fusion: grounded semantics
Mapping: establishing word-referents links
CoNGA – 14th December 2016: Perception & imagination: Grounding semantics of concrete & abstract concepts

4. Multimodal fusion Combining information to know the world better
From language data, it’s hard to know that a carrot is orange.
From visual data it’s hard to know it’s similar to other veggies.
Yet, people describe carrots as both orange and vegetables.
An accurate, human-like semantic space should do like a human.
Combining linguistic and visual vectors!
Word similarities: ask people to say how related two concepts are, then look at how close they are in a DSM. The higher the correlation, the better the DSM.
CoNGA – 14th December 2016: Perception & imagination: Grounding semantics of concrete & abstract concepts

5. Cross-modal mapping Learning to link
How do you know that the word carrot actually refers to the oblong, orange, crunchy vegetable you really don’t like eating?
If I give you a sentence with a new word, and a scene containing many unknown objects, chances are high you will map the new word to the right object.
Function learning and application!
You’ve seen a bunch carrots, you’ve heard a bunch of sentences about carrots, you learned a function that maps the visual carrot to the language carrot. You apply that function to eggplants, chickens, airplanes, ...
CoNGA – 14th December 2016: Perception & imagination: Grounding semantics of concrete & abstract concepts

6. DSMs and cognition A computational explanation of meaning
Put people in an fMRI scan and ask them to think about concepts.
Concept similarity computed on images predicts better concept similarity in the visual cortex.
Concept similarity computed on texts predicts better concept similarity in language processing areas.
Concept similarity computed fusing visual and linguistic representations predicts overall concept similarity better than any other model.
CoNGA – 14th December 2016: Perception & imagination: Grounding semantics of concrete & abstract concepts

7. Fun facts Little tiles to solve the big jigsaw
Image-to-text mapping works better: vision first, i.e. low-level first?
In pictures of deers, the surroundings are as important as the deers themselves to understand that all images depict the same concept (and something not far from roe deer).
Positive correlation between abstractness of a word and vector dispersion.
Visual data help with concrete words, but harm abstract nouns – language only is better! Yet, using perceptual info about concrete nouns, help deriving better representations for abstract verbs (but not abstract nouns).
Combining texts and images allows to better identify metaphors!
CoNGA – 14th December 2016: Perception & imagination: Grounding semantics of concrete & abstract concepts

8. Is it all grounded? Further questions about grounded semantics
Embodied cognition postulates that the perceptual system and the cognitive structures that support simulations underpin processing of both abstract and concrete concepts.
BUT:
- Sensory deprived subjects
- Deep dyslexia
- Age-of-acquisition effects
Is there a unifying mechanism? At what level does it operate? How does language interact with perceptual information?
CoNGA – 14th December 2016: Perception & imagination: Grounding semantics of concrete & abstract concepts

9. Thank you (?) Questions? Doubts? Useful references:
Baroni, Grounding distributional semantics in the visual world. Language & Linguistic Compass, 10(1), 3-13.
Bruni & al, Multimodal Distributional Semantics. Journal of Artificial Intelligence Research, 49, 1-47.
Anderson & al, Reading visually embodied meaning from the brain: visually grounded computational models decode visual-object mental imagery induced by written text. NeuroImage –22.
Pulvermüller, How neurons make meaning: brain mechanisms for embodied and abstract-symbolic semantics." Trends in cognitive sciences 17(9)
Doubts?
Questions?
Thank you (?)