1. Dr. Dimitrios Tzovaras Senior Researcher C’ ITI-CERTH
A Multimodal (Gesture+Speech) Interface for 3D Model Search and Retrieval Integrated in a Virtual Assembly Application
Dr. Dimitrios Tzovaras
Senior Researcher C’
ITI-CERTH

2. Project Objective
Development of a multimodal interface for content-based search of 3D objects
The interface will:
Combine gesture-speech to aid the application
Support searching for similar content
Integration into a virtual assembly application
Challenge: Query by example using a 3D sketch drawn by the user
2/17/2019

3. R&D areas Gesture and speech recognition
3D model content-based search and retrieval
Multimodal user interface for 3D S&R based on sketched objects
Speech driven manipulation of the S&R interface
Implementation of the speech driven S&R platform using sketches
Integration of a multimodal interface in a virtual assembly application environment
2/17/2019

4. Workflow
2/17/2019

5. Tasks T1. 3D model search by example (from a DB of objects)
T2. Gesture recognition
T3. Speech recognition
T4. Virtual assembly
T5. Gesture - Speech integration
T6. 3D search - Virtual assembly integration
T7. Gesture – Speech driven virtual assembly – 3D search integration
T8. Sketch-based descriptor extraction and integration to the 3D search platform
2/17/2019

6. T1. 3D model search Search by example
Volume-based approach (RIT, SIT, Radon, 3D Trace)
Basic tool already provided by ITI-CERTH
A large database of 3D models is available
2/17/2019

7. 2/17/2019

8. 2/17/2019

9. T2. Gesture Recognition
Gesture will be the basic input modality for the system
Challenge: Specific gestures will be recognized and used so as to perform specific tasks (to be provided by FTRD)
For the manipulation of objects in the virtual assembly environment
For the generation of the “example” object to be used for content-based search (3D sketch)
2/17/2019

10. T3. Speech Recognition-Synthesis
Speech will be another input modality for the system:
Specific speech commands will be recognized and used so as to perform specific tasks (interface commands, accept/refuse, etc.) in the virtual assembly environment
Speech synthesis will also be used to support a 3D virtual character (help agent) to guide-verify the steps during the interaction procedure (provided by LIS-INPG)
2/17/2019

11. T4. Virtual assembly Supports assembling 3D objects from their parts
A special authoring environment is used for modeling the assembly sequence
Basic tool already provided by ITI-CERTH
2/17/2019

12. 2/17/2019

13. 2/17/2019

14. T1. 3D model search
2/17/2019

15. T5. Gesture-Speech integration
Gesture and speech recognition will be interlinked and integrated according to two criteria:
Resolution of possible ambiguities of each modality using the alternative one
Cover all the interaction field by handling complementary issues (e.g. speech for handling the application interface and gesture for manipulating the scene objects)
2/17/2019

16. T6. 3D Search – Virtual Assembly Integration
Integration of the 3D search engine to the virtual assembly application
The user will be able to:
Use a query model to find similar objects in the database
Export assembled parts-scenes and search for similar objects
Verify specific tasks using the talking head
2/17/2019

17. T7. Gesture – Speech driven virtual assembly – 3D search
Integration of the Gesture-Speech module into the Virtual Assembly-3D search application
Usability tests
Usability improvements
System evaluation according to a specific scenario, which will be defined during the first week of the project
2/17/2019

18. Sketches will be defined as the trajectory of the hand
T8. Sketch-based descriptor extraction and integration to the 3D search platform
Sketches will be defined as the trajectory of the hand
The sketch could perform specific predefined actions like stretching, shrinking, scaling, etc. on template models like spheres, toruses, etc.
The processed objects will be used as initial query models in the 3D search tool
2/17/2019

19. Two possible implementation directions:
T8. Sketch-based descriptor extraction and integration to the 3D search platform
Two possible implementation directions:
Approach 1: Perform simple deformation operations on primitive models
Approach 2: Use specific simple sketches to define primitives and their properties (i.e size, relative length, etc.)
One of these directions will be chosen according to the feasibility study, which will be performed during the first two weeks.
2/17/2019

20. T8. Sketch-based descriptor extraction and integration to the 3D search platform
2/17/2019

21. T8. Sketch-based descriptor extraction and integration to the 3D search platform
Previous work:
M Oliveira et. Al., “Modeling Solids and Surfaces with Sketches:an Empirical Evaluation”, ISEP/INESC, Portugal, 2001.
J. Pereira et. al., “Towards Calligraphic Interfaces: Sketching 3D Scenes with Gestures and Context Icons”, ISEP/INESC, Portugal, 2004.
J. Mitani et. al., 3D Sketch: “Sketch-Based Model Reconstruction and Rendering”, University of Tokio, Japan, 2000.
D. Xiao, “Sketch-based Instancing of Parameterized 3D Models”, PhD Thesis, Zhejiang University, 2002.
2/17/2019

22. Workplan
2/17/2019

23. Team members Dimitrios Tzovaras, ITI-CERTH
Konstantinos Moustakas, ITI-CERTH
Olivier Bernier, FTRD
Jean Emmanuel Viallet, FTRD
Sebastien Carbini, FTRD
Stephan Raidt, INPG
Matei Mancas, FPMS
Maria Dimiccoli, UPC
Enver Yagci, BUMM
Serdar Balci, BUMM
Eloisa Ibanez Leon, TUM
2/17/2019

24. Team members Informatics and Telematics Institute:
Dimitrios Tzovaras: Senior Researcher. Research interests include virtual reality, image processing, 3D content based search and retrieval, multimodal interfaces,coding.
Konstantinos Moustakas: PhD candidate. Research interests include 3D content based search and retrieval, rigid and deformable body simulation, virtual reality, multimodal interfaces.
2/17/2019

25. Team members France Telecom R&D:
Olivier Bernier: Senior Scientist. Research interest include computer vision, statistical learning and their application to human machine interfaces.
Jean Emmanuel Viallet: Senior Expert. Research interest include vision, interface and image analysis, multimodality.
Sebastien Carbini: PhD candidate. Research interests include speech recognition, gesture recognition, speech-gesture interfaces.
2/17/2019

26. Team members INP-Grenoble, Institute of Speech and Communication:
Stephan Raidt: PhD candidate. Research interests include speech synthesis, mutual attention, face-to-face conversation, eye gaze, deixis.
Faculte Polytechnique de Mons:
Matei Mancas: PhD candidate: Research interests include image processing, audiovisual systems.
2/17/2019

27. Team members Universitat Polytechnica de Catalunya:
Maria Dimiccoli: PhD candidate. Research interest include image processing, differential and morphological image analysis.
Bogazici University:
Enver Yagci: Graduate student. Research interests include medical data analysis and visualization, robotics.
Serdar Balci, Post graduate student. Research interests include signal/image processing of biomedical signals.
Technical University of Madrid:
Eloisa Ibanez Leon: MSc student. Research interest include speech-gesture analysis, language processing
2/17/2019

28. Team members Lannion Mons Grenoble Madrid Istanbul Barcelona
Thessaloniki
2/17/2019

29. Teams
The participants will be split into teams so as to better manage the work and to improve cooperation within each team.
The tasks are interrelated and the two teams will be in close cooperation.
Team 1 (Image-3D processing): K. Moustakas, M. Mancas, M. Dimiccoli, E. Yagci, S. Balci
Team 2 (Gesture-Speech processing): O. Bernier, J. Viallet, S. Carbini, S. Raidt, E. Leon
2/17/2019

30. Tasks per team Team 1 T1 – 3D search: Team 1
T4 – Virtual Assembly: Team 1
T6 – 3D search-Virtual Assembly integration: Team 1
Team 2
T2 – Gesture recognition: Team 2
T3 – Speech recognition: Team 2
T5 – Gesture-Speech integration: Team 2
Teams 1&2
T7 – Gesture-speech & 3D search- Assembly integration: Team 1&2
T8 – Sketch based 3D search-Assembly: Teams 1&2
2/17/2019

31. INFORMATICS & TELEMATICS INSTITUTE Konstantinos Moustakas
THANK YOU!
INFORMATICS & TELEMATICS INSTITUTE
1st km. Thermi-Panorama Road
PO BOX 361, THERMI THESSALONIKI, GREECE
TEL: FAX:
Dr. Dimitrios Tzovaras
Konstantinos Moustakas
2/17/2019