Trinity College Dublin PixelGT: A new Ground Truth specification for video surveillance Dr. Kenneth Dawson-Howe, Graphics, Vision and Visualisation Group (GV2), School of Computer Science & Statistics, Trinity College Dublin

Trinity College Dublin Contents  Introduction  Review  Specification  Annotation  Conclusions Introduction Review Specification Annotation Conclusions

Trinity College Dublin Available video sequences  Video sequences with no ground truth  Video sequences with moving object pixel masks  Video sequences with moving object bounding boxes  Video sequences with labeled events Introduction Review Specification Annotation Conclusions

Trinity College Dublin Available video sequences Introduction Review Specification Annotation Conclusions

Trinity College Dublin Video sequences with moving object pixel masks  Video sequences with moving object and moving shadow pixel masks.  Artificial video sequences with moving object pixel masks.  Video sequences with labeled pixels. Introduction Review Specification Annotation Conclusions CamVid dataset VSSN dataset PETS/LIMU dataset ATON dataset

Trinity College Dublin Video sequences with moving object bounding boxes Introduction Review Specification Annotation Conclusions ETISEO dataset CAVIAR dataset

Trinity College Dublin Video sequences with labeled events Introduction Review Specification Annotation Conclusions PETS 2006 dataset

Trinity College Dublin Motivation  Existing ground truth requires a choice:  Binary pixel masks (with no specific object identification) OR  Object identification (with bounding boxes only and no shadows).  Relatively little pixel accurate ground truth is available.  No existing means of dealing with transparency and reflections. Introduction Review Specification Annotation Conclusions

Trinity College Dublin Ground truth Specifcation  Pixel based ground truth  Bit organisation  Meta data  Gradient profiles  Bounding boxes & events Introduction Review Specification Annotation Conclusions

Trinity College Dublin Pixel based ground truth  Object class (8 bits)  Object instance (9 bits)  Shadow level (4 bits)  Transparency level (3 bits) Introduction Review Specification Annotation Conclusions

Trinity College Dublin Bit organisation Introduction Review Specification Annotation Conclusions L = Label class I = ID S = Shadow T = Transparency 1 = LSB 8 = MSB

Trinity College Dublin Meta data  Based on CVML  Includes:  File names  Bit organisation  Meaning of object labels  Gradient profile Introduction Review Specification Annotation Conclusions

Trinity College Dublin Gradient profiles  Width of edges?  Required for assessing performance? Introduction Review Specification Annotation Conclusions

Trinity College Dublin Bounding boxes and events  Bounding boxes for moving objects  Implicitedly included  Can be explicitly specified  Aids comparison of techniques  Events  Some implicitedly included  Can be explicitly specified  Aids comparison of techniques Introduction Review Specification Annotation Conclusions

Trinity College Dublin Creation of ground truth  Difficult to label all pixels in all frames  Short videos?  Not all frames / pixels?  Method  Background annotation  Object and shadow boundary specification  Object class and ID annotation  Transparency, Meta-data, Bounding boxes, Events and Gradient Profile specification  Propagation of ground truth Introduction Review Specification Annotation Conclusions

Trinity College Dublin Specifying object class & ID Introduction Review Specification Annotation Conclusions

Trinity College Dublin Background annotation  For static cameras  Annotate all background pixels  Facilitates automatic event detection Introduction Review Specification Annotation Conclusions

Trinity College Dublin Specifying object and shadow boundaries  Identify boundary edges of objects / shadows  Once shadow boundary is complete shadow level is computed automatically. Introduction Review Specification Annotation Conclusions

Trinity College Dublin Specifying object class & ID  Once boundary is complete  Select object class and (optionally) ID  Click on the object Introduction Review Specification Annotation Conclusions

Trinity College Dublin Transparency, Meta-data, Bounding boxes, Events and Gradient Profile specification  Transparency not yet addressed  Meta data – specified in CVML  Bounding boxes & events – extract automatically from pixel based ground truth  Gradient profile – extract automatically from each image Introduction Review Specification Annotation Conclusions

Trinity College Dublin Propagation of ground truth  Propagation only required for moving objects  Working with every frame results in very little movement between frames (0-5 pixels)  Search for the closest similar contour in the new frame where the pixels on the object side of the contour appear similar…  Need to identify & correct 1.when new objects enter the scene. 2.when objects reappear from behind occlusions. 3.when internal holes appear within an object. 4.when errors occur in the propagation Introduction Review Specification Annotation Conclusions

Trinity College Dublin The positive spin…  PixelGT is a new form of ground truth which will facilitate  Pixel-level assessment of trackers  Semi-automatic generation of object location, bounding boxes and event descriptions.  Comparison of trackers which assess performance using different types of ground truth Introduction Review Specification Annotation Conclusions

Trinity College Dublin Questions  Do we really need this detailed ground truth ?  Should we change anything about the specification ?  Other questions ? Introduction Review Specification Annotation Conclusions