1. Volodymyr Bobyr Supervised by Aayushjungbahadur Rana
Week 7
Volodymyr Bobyr
Supervised by Aayushjungbahadur Rana

2. Goals Goal Completed ✓ Running In Queue Centroids Output
Centroids Benchmark
Combined Action & Relation Results
Running
Split Action & Relation Results
In Queue

3. Centroids Purpose: object tracking & distinction
Label: Gaussian point applied at the center of object bounding boxes
Output: Binary label for each pixel (0-1)
Threshold is applied at testing
Attempts:
Loss – MSE:
noisy, pattern-less output
Inconsistent among first, middle, last frames
Improvements:
Switched to BinaryLoss
Increased centroid loss weight (2:1)
Label

4. Actions & Relations Actions: Relations: Nuances: # Classes: 42
Ex: ‘hold’, ‘look_at’, ‘bite’
Relations:
# Classes: 8
Ex: ‘next_to’, ‘in_front’, ‘behind’
#𝑟𝑒𝑙𝑎𝑡𝑖𝑜𝑛𝑠≈ # 𝑜𝑏𝑗𝑒𝑐𝑡𝑠 2
Nuances:
A lot more relational data
Different feature relationship between actions and relations

5. Relations for 4 objects in frame 468
Problems
Heavy relation/action imbalance
Solution: class weights
Possible difference in features between relations/actions
Solution: split output
Consequence: +10m parameters
Actions & Relations increase train time x4.5
Possible Explanation: #bboxes≈2 ∗ #𝑜𝑏𝑗 2 ≈ #𝑜𝑏𝑗 2
All the bboxes have to be augmented
Consequence: 16 𝑚𝑖𝑛 𝑒𝑝𝑜𝑐ℎ  72 𝑚𝑖𝑛 𝑒𝑝𝑜𝑐ℎ
Partial Solution: optimization

6. Combined Model – Experimental Results
Object Segmentation:
Similar results to when done without action & relation segmentation
CCE expected cap: 0.60
IOU expected cap: 0.36
Action & Relation segmentation
Accuracy going up
Unexpected behavior in IoU

7. Combined Model – Experimental Results
Centroid Detection

8. Proposed Model Structures
Split Simple
Combined Simple
Object Segmentation
Object Segmentation
Centroids
Encoder Decoder
Centroids
Encoder Decoder
Action Segmentation
Action & Relation Segmentation
+10m parameters
Relation Segmentation
Tradeoff: Split Features vs Model Complexity

9. Proposed Model Structures
Split Complex
Centroids
Object Segmentation
Encoder Decoder
Action Segmentation
Relation Segmentation
Motivation:
Object-centric action & relation detection
Centroids from segmentation

10. Conclusion Current Challenges: Long training times: 2 days / model
Many models to train
Long training times: 2 days / model
Next Week:
Training models
Post-processing