1. Visual Attributes in Video
Marielle Morris
May 26, 2017

2. Project Goals Attributes: descriptive labels
Ex. a trotting horse, a man with a pointy nose
Identify and track attributes in videos
Focus on time-dependent traits
Different than action localization

3. Project Outline Create large-scale video attribute dataset
Ideally build on existing, pre-annotated set
Amazon Mechanical Turk to tag attributes
Localize primary object in each frame
Maintain in “spatio-temporal tube” across clip
Train attribute prediction model
Predict attribute dynamic in relation to object tube

4. Dataset Survey Name Clips Annotation Attributes Notes DAVIS 150
Obj segmentation —
Humans, animals, objects, vehicles
USAA
100
“Weak”
Per-video
Attributes are not per-frame
SegTrack v2 14
Small dataset
FBMS-59 59
Background sub
Not very diverse, small dataset
BVSD
Fragmented ground-truth annotations
CMU 30
Outlines
Few frames annotated, limited motion
UMASS 38
Bad annotations
VSB100
Every 20th frame annotated
McGillFaces 60
Pose
All Faces
Gesture 16
Gestures
All Hand Gestures
Youtube-Faces
3425
Bounding box
CMU Panoptic 65
Skeleton/Pose
All action sequences
WWW Crowd
10000
Action/scene
Poor attribute selections
Youtube-Objects
126
Object is not in every frame, every 10th
CRP 7
Action sequences, high-res, every 10th
Youtube-BB
380000
Single object, 23 object types

5. Current Choice: Youtube-BB
380,000 video segments of 15-20s
Single object bounding boxes at 1 fps
23 object types

6. Attribute Labeling: Single Frame
Choose three of the following attributes:
White
Sniffing
Preparing to jump
Small
Crouching

7. Attribute Annotation: Reel
Choose three of the following attributes:
White
Sniffing
Preparing to jump
Small
Crouching

8. Timeline: Week Proof of concept with Amazon Mechanical Turk
Crowd-source results of single frame vs. reel
Method: COCO Attributes (Patterson & Hays)
Economic Labeling Algorithm for annotation
Formulate plan for attribute generation

9. Timeline: Month Prepare YT-BB dataset
Create Human Intelligence Task for labeling
Building off COCO Attributes Github
Minor modification: adding image reel to GUI