1. INRIA, Nice. December 7 th -8 th 2006 Evaluation protocol Evaluation process

2. INRIA, Nice. December 7 th -8 th 2006 Evaluation protocol ScheduleAgreement Results submission File format transfer tool Feedback

3. INRIA, Nice. December 7 th -8 th 2006 Results collection April Jan July Oct Participants test Corpus diffusion First evaluation Seminar Dec June Results analysis Evaluation review CYCLE 1 Corpus, metrics & Gt review Results analysis Evaluation review Data Collection CYCLE 2 Seminar Results collection Participants test Corpus diffusion Etiseo evaluation Schedule

4. INRIA, Nice. December 7 th -8 th 2006 ETISEO Project ScheduleAgreement Results submission File format transfer tool Feedback

5. INRIA, Nice. December 7 th -8 th 2006 Agreement - Actors : The participant & the coordinator (Silogic) - To participate to the ETISEO project evaluation cycle, the participant will receive specific resources containing video sequences and meta-data files. The agreement describes the terms of the exploitation of the resources, accepted by the participant. The coordinator will transmit the resources to the participant after the signature of the agreement.

6. INRIA, Nice. December 7 th -8 th 2006 Agreement - The participant agrees : - To use the resources only for research and development of video surveillance evaluation purpose. - Not to use the resources for commercial purposes. - Not to diffused the resources to others outside the participant ’ s research group. - To carry out the tests of its algorithms on the resources. - To send to the coordinator the results of the tests, in due time, in a suitable XML format using the defined ontology.

7. INRIA, Nice. December 7 th -8 th 2006 Agreement - The participant is authorised : - To make scientific publications based on the exploitation of the resources, under the condition of mentioning the ETISEO project. - Small part of the resources can be displayed in a research environment, under the condition of mentioning the ETISEO project.

8. INRIA, Nice. December 7 th -8 th 2006 Agreement - The participant authorises : - The coordinator to exploit the results in comparison with the ETISEO ground-truth to compute the evaluation. - The coordinator to diffused the evaluation results in an anonymous way. - The coordinator to diffuse the evaluation results to INRIA-ORION team, which is the ETISEO scientific leader, for the evaluation analysis. - The coordinator has to send back to the participant the results of the evaluation concerning its own results.

9. INRIA, Nice. December 7 th -8 th 2006 ETISEO Project ScheduleAgreement Results submission File format transfer tool Feedback

10. INRIA, Nice. December 7 th -8 th 2006 Results format -1 Result Submission Format : Based on PETS-metrics XML schema, with the addition of events. Some attributes are not processed (segmentation = shape). Documentation on this format is available on the web site also with examples.

11. INRIA, Nice. December 7 th -8 th 2006 Results format -1 Result Submission Format : Extended PETS metrics developper details algorithm details run date camera used results frame tracked_target info_2D segmentation row_seg object info_3D parent target event_target physical_object contextual_object speed Must occur once and only once in the parent element. Must occur at least once. Optional but can occur once and only once in the parent element. Optional to occur but can occur numerously.

12. INRIA, Nice. December 7 th -8 th 2006 Results format -2 Based on VIPER using ETISEO attributes.

13. INRIA, Nice. December 7 th -8 th 2006 Results format -2 Id Start time Type (ex: person, vehicle…) Subtype (ex: car, van…) Info2D Info3D Links Speed GT Nature GT Status GT Occluded (partially or completely) GT Connected with Physical object GT : specific to annotation purpose

14. INRIA, Nice. December 7 th -8 th 2006 Results format -2 Id Name Start and end time Status Involved objects (physical object of interest and contextual object) Event

15. INRIA, Nice. December 7 th -8 th 2006 Results content Submission content : - For each sequence you processed, its xml results file using the naming convention: sequence-name.xml - The questionnaire describing how your system proceeded on all sequences.questionnaire

16. INRIA, Nice. December 7 th -8 th 2006 Results submission Submission method : - by email (<10Mo) to: results-etiseo@silogic.frresults-etiseo@silogic.fr - on ftp www.silogic.fr/etiseo.www.silogic.fr/etiseo Connection parameters will be provide to all participants. Send also a notification to results-etiseo@silogic.fr.results-etiseo@silogic.fr Warning : other participants can have access to your results. - by mail to: Silogic, ETISEO project. Attention of Luc Barthelemy 6, rue Camboulives BP 1133 31036 Toulouse Cedex 1 France

17. INRIA, Nice. December 7 th -8 th 2006 Results submission Submission method : In any case, you received an acknowledgement to your submission.

18. INRIA, Nice. December 7 th -8 th 2006 ETISEO Project ScheduleAgreement Results submission File format transfer tool Evaluation feed-back

19. INRIA, Nice. December 7 th -8 th 2006 Evaluation feedback Evaluation feedback : Metrics results are sent back to participants: res fileres file Example: xls file1, xls file2, xls file3xls file1xls file2xls file3 Multiple submission is allowed. Statistics on metrics for each sequence available on the web-site.web-site Global feedback to all participants.feedback