1. S Carbini, O. Bernier, J.E. Viallet,
A Multimodal (Gesture+Speech) Interface for 3D Model Search and Retrieval Integrated in a Virtual Assembly Application
K. Moustakas, D. Tzovaras,
S Carbini, O. Bernier, J.E. Viallet,
S. Raidt,
M. Mancas,
M. Dimiccoli,
E. Yagci, S. Balci,
E.I. Leon

2. Initial Objectives
Development of a multimodal interface for content-based search of 3D objects
The interface should:
Combine gesture-speech to aid the application
Support searching for similar content
Integration into a virtual assembly application
Challenge: Query by example using a 3D sketch drawn by the user
5/9/2019

3. Virtual assembly Supports assembling 3D objects from their parts
A special authoring environment is used for modeling the assembly sequence
5/9/2019

4. 3D model search Search by example
Volume-based approach (RIT, SIT, Radon, 3D Trace)
A large database of 3D models is available
5/9/2019

5. Gesture is be the basic input modality for the system
Gesture Recognition
Gesture is be the basic input modality for the system
For the manipulation of objects in the virtual assembly environment
Translation
Rotation
Scaling
For the generation of the “example” object to be used for content-based search (3D sketch)
5/9/2019

6. Speech Recognition-Synthesis
Speech is the other input modality for the system:
Specific speech commands will be recognized and used so as to initiate and terminate specific tasks.
Translate
Rotate
Terminate
Sketch
etc.
5/9/2019

7. 5/9/2019

8. Sketches will be defined as the trajectory of the hand
Sketch-based descriptor extraction and integration to the 3D search platform
Sketches will be defined as the trajectory of the hand
The sketch could perform specific predefined actions like stretching, shrinking, scaling, etc. on template models like spheres, toruses, etc.
The processed objects will be used as initial query models in the 3D search tool
5/9/2019

9. Two possible implementation directions:
Sketch-based descriptor extraction and integration to the 3D search platform
Two possible implementation directions:
Approach 1: Perform simple deformation operations on primitive models
Approach 2: Use specific simple sketches to define primitives and their properties (i.e size, relative length, etc.)
One of these directions will be chosen according to the feasibility study, which will be performed during the first two weeks.
5/9/2019

10. Sketch-based descriptor extraction and integration to the 3D search platform
5/9/2019

11. Combination Initial decision:
To combine if possible approach 1 and 2 in a framework which will consist of the following steps:
Primitive object generation using sketching.
Deformation of the primitive objects.
Manipulation of the objects in the scene for the generation of more complex structures.
Deformation of the final structures.
5/9/2019

12. Primitive deformation
Control Points
Affecting only the closest points (geodesic distance)
5/9/2019

13. Sketch recognition (4 approaches!)
Fast schemes:
Method 1: Representation in polar coordinates – Contour-based processing
Method 2: Image based recognition scheme
Method 3: Vector based recognition
Robust scheme:
Method 4: Least-squares fitting and non-linear optimization
5/9/2019

14. Method 1: Representation in polar coordinates
5/9/2019

15. Methods 2-3: Image-Vector based recognition schemes
Calculates the difference between the area of the sketch and its bounding box
Classifies to rectangle, circle, triangle accordingly
Method 3:
Creates a “histogram” of the directions of the vectors for consequtive points.
5/9/2019

16. Method 4: Least-squares fitting and non-linear optimization
5/9/2019

17. Method 4: Least-squares fitting and non-linear optimization
5/9/2019

18. Important side-implementations
5/9/2019

19. Sketch-based 3D search
5/9/2019

20. Sketch-based 3D search
5/9/2019

21. VIDEO
5/9/2019

22. INFORMATICS & TELEMATICS INSTITUTE Konstantinos Moustakas
THANK YOU!
INFORMATICS & TELEMATICS INSTITUTE
1st km. Thermi-Panorama Road
PO BOX 361, THERMI THESSALONIKI, GREECE
TEL: FAX:
Dr. Dimitrios Tzovaras
Konstantinos Moustakas
5/9/2019