1. KRISTINA Consortium Presented by: Mónica Domínguez (UPF-TALN)
Towards a Multimedia Knowledge-Based Agent with Social Competence and Human Interaction Capabilities
KRISTINA Consortium
Presented by: Mónica Domínguez (UPF-TALN)

2. Contents Introduction General Architecture The Agent’s Knowledge
Embodied Conversational Agents Shortcomings
KRISTINA’s Goal
KRISTINA’s Characteristics
General Architecture
The Agent’s Knowledge
Multimodal Knowledge Representation
Multimedia Web Retrieval
Multimodal Interaction
KB Dialogue Management
Multimodal Communication Analysis
Multimodal Communication Generation
Conclusions
06/06/16
MARMI Workshop

3. 1. Introduction Embodied Conversational Agents Shortcomings
KRISTINA’s Goal
KRISTINA’s Characteristics

4. 1.1. Embodied Conversational Agents (ECAs)
Shortcomings of the state-of-the-art ECAs
Do not take into account cultural idiosyncrasies
Degree of politeness, directness, appearance of the agent …
Are usually not multilingual
Follow, as a rule, predefined dialogue strategies
Are, most often, not expressive enough for interaction in social and medical care settings
06/06/16
MARMI Workshop

5. 1.2. KRISTINA’S Goal Migrants Elderly Intermediate Literacy Routines
06/06/16
MARMI Workshop

6. 1.3. KRISTINA’S Characteristics
Multimodal understanding and interpretation of users’ concerns
Ontology based reasoning techniques for dialogue planning
Multimedia web content retrieval techniques for informed communication
Multimodal and culturally adapted expressive communication
06/06/16
MARMI Workshop

7. 2. General Architecture

8. KRISTINA 2nd Plenary Meeting, Thessaloniki
Knowledge
Base
Targeted IR
SSI
ASR
Facial Expr. Recognition
Gesture
Analysis
Emotion Recognition
Language Analysis
Multimodal Fusion
Interaction Manager (VSM)
Dialogue Manager
Idle Behavior Manager
Multimodal
Fission
&
Discourse Planning
Facial Expr. Generation
Generation
Language Synthesis
Recognition KB
Management
Generation DM
KRISTINA 2nd Plenary Meeting, Thessaloniki

9. 3. The Agent’s Knowledge Multimodal Knowledge Representation
Multimedia Web-Based Content Retrieval

10. 3.1. Multimodal Knowledge Representation
Ontology’s Functional Design: to support the dialogue and to represent the relevant basic care and healthcare background information
Models for the representation, integration and interpretation of verbal and non-verbal communication
Background knowledge specific models: user profile ontologies, ontologies for modeling routines and habits related to users’ culture
Existing ontologies for health-care information
06/06/16
MARMI Workshop

11. 3.2. Multimodal Web Retrieval
Extraction of background content from the web using:
Crawling and scraping
Distillation of content elements from web resources and social media (using MetaMap, 4 Babelfy and Unitex6)
Social media topic detection for identification of health-related topics
Indexing, search and retrieval of multimodal information from web
Domain-specific query formulation for querying the indexed repository and triggering the ontology population mechanism
User interaction query formulation that copes with the derivation of knowledge-driven interpretations from users' input in order to compile the appropriate reactions
06/06/16
MARMI Workshop

12. KRISTINA’s Knowledge KB Websites Social media 06/06/16 MARMI Workshop
Topic detection
Topic and event detection
Indexing and Retrieval
Indexing
Indexed repository
Passage retrieval
Search and retrieval (texts)
Query rewriting
Content extraction
Named Entities and concepts extraction
Relation extraction
Query formulation
Domain-specific query formulation
User interaction query formulation
Web crawling & scraping
Web and social media crawling
Content scraping
Ontology Annotation
Ontology mapping
KB population
Knowledge Integration
Inference and reasoning
Response generation KB
KRISTINA Ontologies
Websites
Social media
LA and DM analysis
06/06/16
MARMI Workshop

13. 4. Multimodal Interaction
KB Dialogue Management
Multimodal Communication Analysis
Multimodal Communication Generation

14. 4.1. KB Dialogue Management (I)
OwlSpeak as basic platform
Divided dialogue state
Domain state (handled by KB)
User interaction state: incl. cultural idiosyncrasies and emotions of the user (handled by DM) DM
06/06/16
MARMI Workshop

15. 4.1. KB Dialogue Management (II)
Turn taking and idle behavior
Visual Scene Maker as basic platform
Policy-based turn taking decision making
Is it appropriate to interrupt the user?
How to react upon user’s attempt to barge in?
Idle behavior
Mimicking the user’s affective state / gestures / facial expresion
06/06/16
MARMI Workshop

16. KRISTINA 2nd Plenary Meeting, Thessaloniki
Knowledge
Base
Targeted IR
SSI
ASR
Facial Expr. Recognition
Gesture
Analysis
Emotion Recognition
Language Analysis
Multimodal Fusion
Interaction Manager (VSM)
Dialogue Manager
Idle Behavior Manager
Multimodal
Fission
&
Discourse Planning
Facial Expr. Generation
Generation
Language Synthesis
Recognition KB
Management
Generation DM
KRISTINA 2nd Plenary Meeting, Thessaloniki

17. 4.1. KB Dialogue Management (III)
06/06/16
MARMI Workshop

18. 4.2. Multimodal Communication Analysis (Verbal I)
Speech recognition
Use of Vocapia’s ASR (with statistical acoustic and language modeling) focusing on spoken material
Speech prosody analysis
Investigation of the correlation between prosody and Information Structure and discourse
Language Analysis
Surface-syntactic (LSTM-based) parsing
Deep-syntactic parsing
Frame semantics-based parsing
Projection of frame-based structures onto ontologies
06/06/16
MARMI Workshop

19. 4.2. Multimodal Communication Analysis (Verbal II)
SPEECH
Feature Extraction
Decoder
Ortographic Transcription
ASR
Communicative and Semantic Parser
Disambiguation
Dependency Parser LA
Ontology
FN to OWL KB
06/06/16
MARMI Workshop

20. 4.2. Multimodal Communication Analysis (Non-verbal)
Facial Expression
Facial action units (AUs) from the Facial Action Coding System as representation
AU Classification using SIFT-based features
Gestures
Multi-step video frame analysis
Face detection / skin color detection / edge detection / hand detection / hand tracking
Event-based Mode Fusion and projection into the valence-arousal space
06/06/16
MARMI Workshop

21. 4.3. Multimodal Communication GenerationKB
OWL
Dialogue Manager
Mode Selection
Text Generation (verbal)
Expressive characters (non-verbal)
KRISTINA Agent DB
CoNLL history
Prosody Prediction (verbal)
Speech synthesis (verbal)
06/06/16
MARMI Workshop

22. 4.3. Multimodal Verbal Communication Generation (I)
Text generation
LANGUAGE DEPENDANT
OWL triples
FrameNet Structure
Semantic Structure
Deep-Syntactic Structure
Surface-Syntactic Structure
Topological Structure
Morphological Structure
Text
Language-specific
Predicate-Argument
Rough sentence structure
Rough syntactic labels
- Fine-grained syntactic labels - Grammatical words
Word order
Agreement resolution
Inflectional forms
Contractions and elisions
Final forms
06/06/16
MARMI Workshop

23. 4.3. Multimodal Verbal Communication Generation (II)
Expressive Prosody Generation
Information / discourse structure – prosody correlation
06/06/16
MARMI Workshop

24. 4.3. Multimodal Non-Verbal Communication Generation
Projection of the continuous valence-arousal space onto 2D gestures and facial expressions of the avatar
Realistic lip-synchronization using a reduced set of commands for expressions (RSCE)
3D realization of facial expressions and lip-synchronization
06/06/16
MARMI Workshop

25. 5. Conclusions

26. 5. Conclusions
The KRISTINA technologies facilitate the conduct of socially competent emotive multilingual conversations with individuals in need of advice and support in the context of basic care and healthcare
The functionality of the individual modules will incrementally grow over the next two years
Test scenarios designed by experts in basic and health care will ensure a practice-oriented validation of the agent
06/06/16
MARMI Workshop

27. Thank You
@MonikaUPF
@projectKRISTINA