1. Trinity College Dublin PixelGT: A new Ground Truth specification for video surveillance Dr. Kenneth Dawson-Howe, Graphics, Vision and Visualisation Group (GV2), School of Computer Science & Statistics, Trinity College Dublin

2. Trinity College Dublin Contents  Introduction  Review  Specification  Annotation  Conclusions Introduction Review Specification Annotation Conclusions

3. Trinity College Dublin Available video sequences  Video sequences with no ground truth  Video sequences with moving object pixel masks  Video sequences with moving object bounding boxes  Video sequences with labeled events Introduction Review Specification Annotation Conclusions

4. Trinity College Dublin Available video sequences Introduction Review Specification Annotation Conclusions

5. Trinity College Dublin Video sequences with moving object pixel masks  Video sequences with moving object and moving shadow pixel masks.  Artificial video sequences with moving object pixel masks.  Video sequences with labeled pixels. Introduction Review Specification Annotation Conclusions CamVid dataset VSSN dataset PETS/LIMU dataset ATON dataset

6. Trinity College Dublin Video sequences with moving object bounding boxes Introduction Review Specification Annotation Conclusions ETISEO dataset CAVIAR dataset

7. Trinity College Dublin Video sequences with labeled events Introduction Review Specification Annotation Conclusions PETS 2006 dataset

8. Trinity College Dublin Motivation  Existing ground truth requires a choice:  Binary pixel masks (with no specific object identification) OR  Object identification (with bounding boxes only and no shadows).  Relatively little pixel accurate ground truth is available.  No existing means of dealing with transparency and reflections. Introduction Review Specification Annotation Conclusions

9. Trinity College Dublin Ground truth Specifcation  Pixel based ground truth  Bit organisation  Meta data  Gradient profiles  Bounding boxes & events Introduction Review Specification Annotation Conclusions

10. Trinity College Dublin Pixel based ground truth  Object class (8 bits)  Object instance (9 bits)  Shadow level (4 bits)  Transparency level (3 bits) Introduction Review Specification Annotation Conclusions

11. Trinity College Dublin Bit organisation Introduction Review Specification Annotation Conclusions L = Label class I = ID S = Shadow T = Transparency 1 = LSB 8 = MSB

12. Trinity College Dublin Meta data  Based on CVML  Includes:  File names  Bit organisation  Meaning of object labels  Gradient profile Introduction Review Specification Annotation Conclusions

13. Trinity College Dublin Gradient profiles  Width of edges?  Required for assessing performance? Introduction Review Specification Annotation Conclusions

14. Trinity College Dublin Bounding boxes and events  Bounding boxes for moving objects  Implicitedly included  Can be explicitly specified  Aids comparison of techniques  Events  Some implicitedly included  Can be explicitly specified  Aids comparison of techniques Introduction Review Specification Annotation Conclusions

15. Trinity College Dublin Creation of ground truth  Difficult to label all pixels in all frames  Short videos?  Not all frames / pixels?  Method  Background annotation  Object and shadow boundary specification  Object class and ID annotation  Transparency, Meta-data, Bounding boxes, Events and Gradient Profile specification  Propagation of ground truth Introduction Review Specification Annotation Conclusions

16. Trinity College Dublin Specifying object class & ID Introduction Review Specification Annotation Conclusions

17. Trinity College Dublin Background annotation  For static cameras  Annotate all background pixels  Facilitates automatic event detection Introduction Review Specification Annotation Conclusions

18. Trinity College Dublin Specifying object and shadow boundaries  Identify boundary edges of objects / shadows  Once shadow boundary is complete shadow level is computed automatically. Introduction Review Specification Annotation Conclusions

19. Trinity College Dublin Specifying object class & ID  Once boundary is complete  Select object class and (optionally) ID  Click on the object Introduction Review Specification Annotation Conclusions

20. Trinity College Dublin Transparency, Meta-data, Bounding boxes, Events and Gradient Profile specification  Transparency not yet addressed  Meta data – specified in CVML  Bounding boxes & events – extract automatically from pixel based ground truth  Gradient profile – extract automatically from each image Introduction Review Specification Annotation Conclusions

21. Trinity College Dublin Propagation of ground truth  Propagation only required for moving objects  Working with every frame results in very little movement between frames (0-5 pixels)  Search for the closest similar contour in the new frame where the pixels on the object side of the contour appear similar…  Need to identify & correct 1.when new objects enter the scene. 2.when objects reappear from behind occlusions. 3.when internal holes appear within an object. 4.when errors occur in the propagation Introduction Review Specification Annotation Conclusions

22. Trinity College Dublin The positive spin…  PixelGT is a new form of ground truth which will facilitate  Pixel-level assessment of trackers  Semi-automatic generation of object location, bounding boxes and event descriptions.  Comparison of trackers which assess performance using different types of ground truth Introduction Review Specification Annotation Conclusions

23. Trinity College Dublin Questions  Do we really need this detailed ground truth ?  Should we change anything about the specification ?  Other questions ? Introduction Review Specification Annotation Conclusions