1 Viewing Vision-Language Integration as a Double-Grounding case Katerina Pastra Department of Computer Science, Natural Language Processing Group, University of Sheffield, U.K. Language Technology Group, Institute for Language and Speech Processing (ILSP), Athens, Greece

2 Vision-Language Integration  What is computational V-L integration? (definition)  How is it achieved? (state of the art, trends, needs)  Why is it needed in AI agents? (explanatory/theoretical ground)  How far can we go? (implementation suggestions, the VLEMA prototype)  Content integration vs. technical integration à Small part within cognitive architectures à Small part of the integration story still, lack of an AI study of V-L integration 

3 The notion of integration State of the art V-L integration prototypes: a)Deal with blocksworlds or miniworlds (scalability issues) b)Work with already abstracted/analysed visual input c)Rely on integration resources for V-L association Descriptive Definition  Computational V-L integration is a process of associating visual and corresponding linguistic representations  It may take the form of one of four integration processes according to the integration purpose to be served

4 Classification of V-L integration prototypes System typeIntegration Process Performance Enhancement Medium x analysis  Medium y analysis (NL  IU, or NL  IU) Medium Translation Source medium analysis  Target medium gen. (image  language or image  language) Multimedia Generation Abstracted data  Multimedia generation (tabular data or knowledge representation) Situated Dialogue Multimedia analysis  Medium/multimedia gen. (NL analysis and shared visual scene  action/MM)

5 Why do agents need V-L integration ?  Inherent characteristics of integrated media: Does each one of them lack something that the other can compensate for?  Gains for an agent in communication: Are agents with V-L integration abilities more intelligent? Why do we need to know ?  to decide on the significance of such a mechanism for an artificial agent  to get the theoretical ground needed for research that is currently done mostly ad hoc and in isolation within different AI sub-areas

6 Inherent Characteristics  Images: - reference object: physical or mental - lack inherent means of indicating focus/salience (cf. indexical-deictic mechanisms in vision theories) - lack inherent means of indicating type:token distinctions (i.e. level of abstraction)  Language: - reference object: mental - has subtle mechanisms for indicating level of abstraction - has mechanisms for controlling attendance to details, focus etc. - lacks direct access to the physical world (cf. indexicals)

7 From Symbol Grounding… From the Symbol Grounding debate we get the following : - Language lacks direct access to the physical world - Language needs such access to express intentionality - Symbol grounding is a process of associating symbols/language with percepts (visual percepts) - Symbol grounding provides language direct access to the physical world - An agent must perform symbol grounding on its own to be intrinsically intentional (must go beyond instantiation of associations to inference)

Visual Perception Representations Linguistic Representations Association Direct Access Grounding

9 Shifting the focus from symbols… Relying on the inherent characteristics of images, one may argue that : - Images lack controlled access to mental aspects of the world - Images need such access to express intentionality - Image grounding is a process of associating images with language - Image grounding provides images controlled access to the mental world - An agent must perform image-grounding on its own to be intrinsically intentional

Visual Representations Linguistic Representations Association Direct Access Uncontrolled Access Grounding

11 From Symbol Grounding to Double Grounding The Double-Grounding Theory: - Double-grounding is a process of associating symbolic with iconic representations - Double-grounding provides language a direct access to the physical world, and at the same time it provides vision a controlled access to mental aspects of the world - Vision-language integration is a case of double-grounding - V-L integration compensates for features images and language inherently lack on their own – it is necessary for expressing and understanding intentionality in V-L MM situations

12  V-L integration abilities are needed for an agent to be intentional in MM situations  Exploring how V-L integration can be achieved computationally, one realises that this research issue involves not only the perceptual and linguistic modules of a cognitive architecture, but also the learning and reasoning ones.  The corresponding AI communities need, therefore, to join forces for addressing the challenges in endowing agents with their own V-L integration abilities Viewing integration as a double-grounding case

13 The AI quest for V-L Integration In relying on human created data, state of the art V-L integration systems avoid core integration challenges and therefore fail to perform real integration Can we do better? How far can we go ??? Challenging current practices means that a prototype should:  work with real visual scenes  analyse its visual data automatically  associate images and language automatically Is it feasible to develop such a prototype ???

14 An optimistic answer VLEMA: A Vision-Language intEgration MechAnism  Input: automatically re-constructed static scenes in 3D (VRML format) from RESOLV (robot-surveyor)  Integration task: Medium Translation from images (3D sitting rooms) to text (what and where in EN)  Domain: estates surveillance  Horizontal prototype  Implemented in shell programming and ProLog