Perception & Imagination: Grounding semantics of concrete and abstract concepts CoNGA – Cognitive & Neuroscience Group, Antwerp 14th December Giovanni Cassani 4A257E -> FFFFFF in 10 steps using HSV gradient

Multimodal distributional semantics Bridging the gap between language and the world Distributional Semantics : “You shall know a word by the company it keeps” → Similar concepts occur in similar contexts BUT: - perceptual info doesn’t appear in text (no orange carrots, trivial) - how is the word-referent link established? Lack of grounding… …bring in info from images! CoNGA – 14th December 2016: Perception & imagination: Grounding semantics of concrete & abstract concepts

The vector representation A single framework to encode all information Distributional Semantic Models (DSMs) encode each word as a numeric vector: each vector define the position of a word in a high-dimensional geometrical space (you can count or predict, doesn’t make a difference) computed in the same way from texts and images. Fusion: grounded semantics Mapping: establishing word-referents links CoNGA – 14th December 2016: Perception & imagination: Grounding semantics of concrete & abstract concepts

Multimodal fusion Combining information to know the world better From language data, it’s hard to know that a carrot is orange. From visual data it’s hard to know it’s similar to other veggies. Yet, people describe carrots as both orange and vegetables. An accurate, human-like semantic space should do like a human. Combining linguistic and visual vectors! Word similarities: ask people to say how related two concepts are, then look at how close they are in a DSM. The higher the correlation, the better the DSM. CoNGA – 14th December 2016: Perception & imagination: Grounding semantics of concrete & abstract concepts

Cross-modal mapping Learning to link How do you know that the word carrot actually refers to the oblong, orange, crunchy vegetable you really don’t like eating? If I give you a sentence with a new word, and a scene containing many unknown objects, chances are high you will map the new word to the right object. Function learning and application! You’ve seen a bunch carrots, you’ve heard a bunch of sentences about carrots, you learned a function that maps the visual carrot to the language carrot. You apply that function to eggplants, chickens, airplanes, ... CoNGA – 14th December 2016: Perception & imagination: Grounding semantics of concrete & abstract concepts

DSMs and cognition A computational explanation of meaning Put people in an fMRI scan and ask them to think about concepts. Concept similarity computed on images predicts better concept similarity in the visual cortex. Concept similarity computed on texts predicts better concept similarity in language processing areas. Concept similarity computed fusing visual and linguistic representations predicts overall concept similarity better than any other model. CoNGA – 14th December 2016: Perception & imagination: Grounding semantics of concrete & abstract concepts

Fun facts Little tiles to solve the big jigsaw Image-to-text mapping works better: vision first, i.e. low-level first? In pictures of deers, the surroundings are as important as the deers themselves to understand that all images depict the same concept (and something not far from roe deer). Positive correlation between abstractness of a word and vector dispersion. Visual data help with concrete words, but harm abstract nouns – language only is better! Yet, using perceptual info about concrete nouns, help deriving better representations for abstract verbs (but not abstract nouns). Combining texts and images allows to better identify metaphors! CoNGA – 14th December 2016: Perception & imagination: Grounding semantics of concrete & abstract concepts

Is it all grounded? Further questions about grounded semantics Embodied cognition postulates that the perceptual system and the cognitive structures that support simulations underpin processing of both abstract and concrete concepts. BUT: - Sensory deprived subjects - Deep dyslexia - Age-of-acquisition effects Is there a unifying mechanism? At what level does it operate? How does language interact with perceptual information? CoNGA – 14th December 2016: Perception & imagination: Grounding semantics of concrete & abstract concepts

Thank you (?) Questions? Doubts? Useful references: Baroni, 2016. Grounding distributional semantics in the visual world. Language & Linguistic Compass, 10(1), 3-13. Bruni & al, 2014. Multimodal Distributional Semantics. Journal of Artificial Intelligence Research, 49, 1-47. Anderson & al, 2015. Reading visually embodied meaning from the brain: visually grounded computational models decode visual-object mental imagery induced by written text. NeuroImage 120. 309–22. Pulvermüller, 2013. How neurons make meaning: brain mechanisms for embodied and abstract-symbolic semantics." Trends in cognitive sciences 17(9). 458-470. Doubts? Questions? Thank you (?)