Volodymyr Bobyr Supervised by Aayushjungbahadur Rana Week 8 Volodymyr Bobyr Supervised by Aayushjungbahadur Rana

Goals Goal Completed ✓ X Optimize the Data Loader Object & Action Segmentations and Centroid Results Incorporate Mean Average Precision metrics Rough Comparison to Challenge Results Full (temporal tube) Comparison to Challenge Results X

Quick Info Managed to bring training time back to 20 minutes Trained actions, objects, and centroids separately Planning to train in a sequence: Objects -> Actions -> Relations -> Centroids Order subject to change Got good checkpoints for objects & actions Actions Objects

Segmentation to B-Boxes Process: Segmentation Threshold: (0, 1) labels Segmentation to Blobs: connected segments are extracted as ‘blobs’ Blob Filtering: dispose of blobs with area < blob threshold (currently: 25) Bounding Box Generation: corners of the blobs Likely Problems: A single blob may represent multiple objects Centroids will be used to separate instances

Mean Average Precision Used as main metric in: ACM Multimedia 2019 Grand Challenge Process: Generate bounding boxes from segmentations Compare each generated bounding box to ground truth bounding box If IoU > threshold (0.5), the bounding box is a true positive If a bounding box wasn’t marked as true positive, it’s a false positive Mean-AP is the average over all classes, frames, and samples

Mean Average Precision If true positives == false positives == 0 for a class: Two different ways: If there weren’t any labels for that class, mark it as 1 Bumps up mean-ap by a lot because most channels will return a 1 If there weren’t any labels for that class, ignore this box If there were labels, mark it as 0 Results: 1st Way 2nd Way Objects 0.9363 0.2407 Actions 0.9576 0.3148

Approach Shift Combine actions and relations as they share most context Split the actions/relations into unidirectional and bidirectional Bidirectional: Actions/Relations that have no need for subject/object separation Ex: ‘next to’ Predicted between 0 and 1 Unidirectional: Actions/Relations where subjects & objects have to be clearly defined Ex: ‘holding’, ‘behind’, etc. Predicted between -1 and 1 (-1: object, 1: subject) When predicting some actions, the counterpart can often be assumed (post-processing) IE: (A, ‘behind’, B) implies (B, ‘in front’, A)