1. Volodymyr Bobyr Supervised by Aayushjungbahadur Rana
Week 8
Volodymyr Bobyr
Supervised by Aayushjungbahadur Rana

2. Goals Goal Completed ✓ X Optimize the Data Loader
Object & Action Segmentations and Centroid Results
Incorporate Mean Average Precision metrics
Rough Comparison to Challenge Results
Full (temporal tube) Comparison to Challenge Results X

3. Quick Info Managed to bring training time back to 20 minutes
Trained actions, objects, and centroids separately
Planning to train in a sequence: Objects -> Actions -> Relations -> Centroids
Order subject to change
Got good checkpoints for objects & actions
Actions
Objects

4. Segmentation to B-Boxes
Process:
Segmentation Threshold: (0, 1) labels
Segmentation to Blobs: connected segments are extracted as ‘blobs’
Blob Filtering: dispose of blobs with area < blob threshold
(currently: 25)
Bounding Box Generation: corners of the blobs
Likely Problems:
A single blob may represent multiple objects
Centroids will be used to separate instances

5. Mean Average Precision
Used as main metric in: ACM Multimedia 2019 Grand Challenge
Process:
Generate bounding boxes from segmentations
Compare each generated bounding box to ground truth bounding box
If IoU > threshold (0.5), the bounding box is a true positive
If a bounding box wasn’t marked as true positive, it’s a false positive
Mean-AP is the average over all classes, frames, and samples

6. Mean Average Precision
If true positives == false positives == 0 for a class:
Two different ways:
If there weren’t any labels for that class, mark it as 1
Bumps up mean-ap by a lot because most channels will return a 1
If there weren’t any labels for that class, ignore this box
If there were labels, mark it as 0
Results:
1st Way
2nd Way
Objects
0.9363
0.2407
Actions
0.9576
0.3148

7. Approach Shift
Combine actions and relations as they share most context
Split the actions/relations into unidirectional and bidirectional
Bidirectional:
Actions/Relations that have no need for subject/object separation
Ex: ‘next to’
Predicted between 0 and 1
Unidirectional:
Actions/Relations where subjects & objects have to be clearly defined
Ex: ‘holding’, ‘behind’, etc.
Predicted between -1 and 1
(-1: object, 1: subject)
When predicting some actions, the counterpart can often be assumed (post-processing)
IE: (A, ‘behind’, B) implies (B, ‘in front’, A)