1. SoLSTiCe Similarity of locally structured data in computer vision Université-Jean Monnet (Saint-Etienne) LIRIS (Lyon) (1/02/2014 -2018) Elisa Fromont, Kick-off meeting, 14/02/2014

2. Présentation du consortium

3. Aim: design new models and tools for representing and managing images and videos Targeted applications: classification, recognition or indexing (in a context of occlusions and non rigid objects in 2D (+ t), 3D and 3D+t media) Proposal: explore locally structured data (LSD) = visual features + discrete structures to model local (spatio-temporal) relationships 3 main tasks: 1.[Extracting LSD from images and videos:] extract relevant visual features and structure them w.r.t. spatial and temporal relationships. 2.[Measuring the similarity of LSD:] design relevant similarity measures for comparing LSD, and efficient algorithms for computing these measures. 3.[Mining LSD:] characterize LSD by means of frequently (or infrequently) occurring patterns (itemsets, sequences or graphs) and use them to create discriminative features for solving computer vision tasks. Main ideas

4. The project : 4 tasks interconnected 1.[Task 0] will be dedicated to the project management; 2.[Task 1] will design LSD for describing images and videos, and will design tools for extracting these LSD; 3.[Task 2] will design kernels, similarity measures and matching algorithms for comparing LSD; 4.[Task 3] will design mining algorithms for extracting relevant patterns in LSD; 5.[Task 4] will be dedicated to the design and use of demo platforms to test (and demonstrate) on computer vision benchmarks and new datasets the models and tools designed in Tasks 1 to 3.

5. Livrables (1/2) Tâche 1 From images and Vidéos to LSD (LaHC) D1.1Research report describing new descriptors for images D1.2.2Survey of state-of-the-art approaches for structuring visual words by means of strings, trees or graphs D1.2.2Research report describing new LSD for images and 3D objects D1.3Research report on extensions of LSD of subtask 2.2 for videos, and evaluation Tâche 2: Mesuring the similarity of LSD (LIRIS) D 2.1.1Research report describing new matching algorithms D2.1.2Design of an open-source library of graph matching algorithms D2.2Research report on new kernel for combinatorial maps D2.3Research report on metric learning or deep learning on locally structured data

6. Livrables (2/2) Tâche 3: Mining LSD D3.1.1Research report on mining LSD in images and videos D3.1.2Research report on new algorithms to mine LSD in images and/or videos D3.2Research report on using frequent substructures to find relevant features for image classification D3.3.1Research report on mining approximate patterns in plane graph D3.3.2Research report on finding relevant spatio temporal patterns in videos Tâche 4: Demonstrations in computer vision D4.1.1Creation of the Solstice platform D4.1.2Activity recognition module for software platform LIRIS-VISION Tracking D4.1.3Demo in the Solstice platform D4.1.4Activity recognition module for robotics platform LIRIS-VOIR D4.2.1Object recognition module for software platform LIRIS-VISION D4.2.2Object recognition demo for the Solstice platform D4.2.3Object recognition module for robotics platform LIRIS-VOIR

7. Planning

8. Valorisation/Impact scientific communications submitted to major conferences and journals (CVPR, ECCV, ICCV, ICPR, AVSS, KDD, ICML, ECML, PKDD, ICPR, etc.) and journals (IEEE-T-PAMI, PR, IJCV, CVIU, MLJ, JMLR, etc.) in image processing, pattern recognition, combinatorial optimization, machine learning, and data mining. open source platforms developed in task 4 (and task 2) workshops co-located with major conferences in order to share ongoing research. design educational and recreational demos targeting a non specialist public to be presented during popular events such as “la fête de la science”.

9. Use of resources LaHC (140000 euros): – Staff (100000 euros) Ph.D Student: 36 months on « New matching strategies for data mining applied to computer vision problems » (tasks 2 and 3 + 1 and 4) co-supervised with liris – Travels – Other expenses: master thesis grants + hardware LIRIS (134000 euros) – Staff (100000 euros): Ph.D Student: 36 months on « Analysis of complex scenes with structured models » (tasks 1 and 2 + 3) co-supervised with LaHC – Travels – Other expenses: master thesis grants + hardware

10. Points to discuss Website (Jean Monnet) Include some more members (Taygun, Romain?) How to spend the money for the second thesis (Remi, Marc, Damien ?) Demos? Next meetings