1. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 ETISEO Annotation rules Data structure Annotation tool and format Ground truth creation rules Reference data generation

2. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Data structure - context Id Type (ex: zone) Name (ex: exit_zone) Polygon Context

3. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Data structure - object Id Start time Type (ex: person, vehicle…) Subtype (ex: car, van…) Info2D Info3D Links Speed GT Nature GT Status GT Occluded (partially or completely) GT Connected with Physical object GT : specific to annotation purpose

4. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Data structure - event Id Name Start and end time Status Involved objects (physical object of interest and contextual object) Event

5. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 ETISEO Annotation rules Data structure Annotation tool and format Ground truth creation rules Reference data generation

6. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Annotation tool and format Database - Excel file Context annotation - Viper-GT tool and format Annotation delivered with video: Ground truth - Viper-GT tool and format Reference data (computed from GT) – Specific tool and Viper-GT format Annotation delivered at the end of Etiseo:

7. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Annotation tool and format

8. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Annotation tool and format Viper-GT format example

9. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 ETISEO Annotation rules Data structure Annotation tool and format Ground truth creation rules Reference data generation

10. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation rules Rules for objects:  Contextual object  Static physical object of interest  Occlusion  Artefacts  Sudden illumination changes  Held object  Group - Crowd  Crowded video clip  Multi-view annotation Rules for events

11. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for objects Contextual object Annotated during its period of movement

12. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation rules Rules for objects:  Contextual object  Static physical object of interest  Occlusion  Artefacts  Sudden illumination changes  Held object  Group - Crowd  Crowded video clip  Multi-view annotation Rules for events

13. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for objects Static physical object of interest Any physical object of interest is annotated on the whole video clip (ex: a parked vehicle) First frame and last frame of a video clip

14. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation rules Rules for objects:  Contextual object  Static physical object of interest  Occlusion  Artefacts  Sudden illumination changes  Held object  Group - Crowd  Crowded video clip  Multi-view annotation Rules for events

15. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for objects Occlusions – general rules Apparition of an object: the BBox increase with object apparition including only its visible part, All BBox are limited by the image boundary, Partial and complete occlusion will be differentiate in the evaluation (attribute “occluded”).

16. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for objects Static occlusions (by a contextual object) Partial occlusion: full BBox is drawn including hidden part,

17. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for objects Static occlusions (by a contextual object) Complete occlusion - two cases:  all “escapes” of the contextual object are visible: full BBox is drawn all the time,

18. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for objects  all “escapes” of the contextual object are not visible: the object is no more annotated when completely occluded

19. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for objects Partial or complete occlusion: in both cases, the full BBox is drawn including hidden part. Dynamic occlusion (by another object)

20. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for objects Split object due to occlusion A unique BBox including all sub part is drawn.

21. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation rules Rules for objects:  Contextual object  Static physical object of interest  Occlusion  Artefacts  Sudden illumination changes  Held object  Group - Crowd  Crowded video clip  Multi-view annotation Rules for events

22. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for objects Artefacts Only the part constituting the objects forms the BBox, artefacts (shadow, reflection…) are ignored for instant. Annotated in video data set 2

23. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation rules Rules for objects:  Contextual object  Static physical object of interest  Occlusion  Artefacts  Sudden illumination changes  Held object  Group - Crowd  Crowded video clip  Multi-view annotation Rules for events

24. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for objects Sudden illumination changes (tagged frames) Static object Object in movement: position interpolation

25. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation rules Rules for objects:  Contextual object  Static physical object of interest  Occlusion  Artefacts  Sudden illumination changes  Held object  Group - Crowd  Crowded video clip  Multi-view annotation Rules for events

26. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for objects Held object Example: bag-person, person-bike, door-vehicle The holder and the held object are both annotated (if visible), The carried object is connected to the holder.

27. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for objects Specific case Person getting in/out a vehicle: annotated only when visible

28. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation rules Rules for objects:  Contextual object  Static physical object of interest  Occlusion  Artefacts  Sudden illumination changes  Held object  Group - Crowd  Crowded video clip  Multi-view annotation Rules for events

29. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for objects Group - Crowd Group: multiple bounding boxes (persons + group) Crowd : unique Bbox when individual annotation is impossible.

30. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation rules Rules for objects:  Contextual object  Static physical object of interest  Occlusion  Artefacts  Sudden illumination changes  Held object  Group - Crowd  Crowded video clip  Multi-view annotation Rules for events

31. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for objects Crowded video clip Only object involved in event of interest are annotated. No false detections evaluation

32. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation rules Rules for objects:  Contextual object  Static physical object of interest  Occlusion  Artefacts  Sudden illumination changes  Held object  Group - Crowd  Crowded video clip  Multi-view annotation Rules for events

33. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for objects Multi-view annotation

34. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation rules Rules for objects:  Contextual object  Static physical object of interest  Occlusion  Artefacts  Sudden illumination changes  Held object  Group - Crowd  Crowded video clip  Multi-view annotation Rules for events

35. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for events Id Name Start and end time Status Involved objects:  Id of the physical object of interest in GT files  Id of the contextual object in context information files Event – data structure

36. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for events Choice of event for each video clip in the event ontology => defined in the database xls file (provided with video) Sequence NameETI-VS1-RD-15 ……… Content Physical objects typePerson, vehicle Events of interestgets_in, gets_out, parked

37. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for events Automatic annotation of events with quantitative constraints, example:  Related to kinematics properties: stopped, running…  Related to spatial properties: close_to, inside_zone… Manual annotation of other events: holding, pushing, waiting, gets_in/out,…

38. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Ground truth creation – Rules for events Example

39. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 ETISEO Annotation rules Data structure Annotation tool and format Ground truth creation rules Reference data generation

40. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 Reference data generation Filtering on object size Object speed computation Group generation Multi-view and 3D reconstruction Events with quantitative constraints generation

41. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 ETISEO Annotation rules Documentation: ETI-Data structure and output format ETI-Video Annotation Rules

42. INRETS, Villeneuve d’Ascq, December 15 th -16 th 2005 ETISEO Annotation rules Questions …