1. Motion Segmentation at Any Speed
Shrinivas Pundlik and Stan Birchfield
Department of Electrical and Computer Engineering
Clemson University Clemson, SC USA

2. The problem of motion segmentation
Carve an image according to motion vectors
Gestalt theory:
Focus on well-organized patterns rather than disparate parts
“Grouping” - the key idea behind visual perception
But motion is inherently differential!
common fate

3. Previous approaches Extraction of motion layers
Multi-body factorization
Eigenvector based
Wang and Adelson 1994,
Ayer and Sawhney 1995,
Xiao and Shah 2005
Ke and Kanade 2001,
Vidal and Sastry 2003
Shi and Malik 1998
Object level grouping
Functional
Rank Constraint
Sivic, Schaffalitzky and
Zisserman 2004
Cremers and Soatto 2005
Rothganger, Lazebnik,
Schmid and Ponce 2004

4. Traditional approach Two unanswered questions:
time
two
frames
spatiotemporal
volume
Two unanswered questions:
What are the limitations of processing a block of frames?
How to integrate information over time?

5. Batch processing  dependent upon speed x fast medium slow threshold t
time
window
 dependent upon speed

6. Incremental processingx
fast
medium
slow
crawling
threshold t
 independent of speed dependent only upon the amount of information

7. Algorithm overview Detect and track Kanade-Lucas-Tomasi feature points
Accumulate groups using region growing (neighbors from Delaunay triangulation)
Retain consistent groups
Maintain groups over time

8. Region growing Between two frames, Repeat
Randomly select seed feature
Fit motion model to neighbors
Repeat until group does not change:
Discard all features except the one near the centroid
Grow group by recursively including neighboring features with similar motion
Update the motion model
Until all features have been considered

9. Region growing for a single group
Choice of seed
heavily influences
resulting group

10. Finding consistent groups
Consistency check: Features that are always grouped together, no matter the seed point
seed point
seed point a a a b b b c c c d d d a b c d a b c d a b c d a 1 1 a 2 1 1 1 1 1 1 = 1 1 + b b 2 2 c 1 c 1 2 d 1 d 2 1
In practice, we use 7 seed points

11. Single consistent group
seed point 1
seed point 2
seed point 3
consistent group

12. Multiple consistent groups
seed point 1
seed point 2
seed point 3
only 3 groups
in initial results
4 groups
in final result
consistent groups

13. Maintaining groups over time
Find new groups (when new objects enter scene)
Split existing groups (when configuration changes)
Add new features to existing groups (when new information available)

14. Finding new groups group1 group2 group1 group2 ungrouped features
find consistent groups

15. Splitting existing groups
if lost features >
x % of original features
frame k
frame k + n
lost features 2 2 2
try to regroup (find consistent groups again) 1 1 1
track
features 3 3 4 4 7 4 7 5 8 6
either all
are regrouped
or multiple
groups
are found 6 5 9 10
newly added
features 8 3 7 6 9 5 10 8 3 7 6 9 5 10

16. Adding new features new (ungrouped) features 1 2 3 group 1
(with motion
model 1)
group 2
(with motion
model 2)
Feature 2 is neighbor to multiple groups:
Compare feature motion with all group motion models
Add if similar to one and dissimilar from the rest
Feature 1 is neighbor to only one group:
Compare feature motion with group motion model
Add if similar
Feature 1 is neighbor to only one group:
Compare feature motion with group motion model
Add if similar
Feature 3 is neighbor to only one group:
Compare feature motion with group motion model
Add if similar

17. Experimental results 8 64
185
279
395
468
497
520
statue sequence

18. Number of groups is determined automatically and dynamically
Experimental results
Number of groups is determined
automatically and dynamically

19. mobile-calendar sequence
Experimental results
mobile-calendar sequence 14 70
100
car-map sequence 11 20 35
free-throw sequence 10 15 20

20. Videos
Videos available at

21. Insensitivity to speed
normal 64
185
395
480
½ frames
dropped 32 93
197
240
double
frames
128
370
790
960

22. Insensitivity to parameter
normal 4 8 12 64
threshold 4 8 12 64
threshold
x 2 4 8 12 64

23. Future application: Mobile robot obstacle avoidance
Speed of algorithm: 20 ms per image frame
(plus feature tracking, which is real time)
 Can apply algorithm to real-time problems

24. Conclusion Motion is inherently differential Proposed algorithm
Motion segmentation should take this into account
Proposed algorithm
segments based upon available evidence, independently of object speed
incrementally processes video
contains one primary parameter, namely the amount of evidence needed to split a group
works in real time
automatically computes the number of objects dynamically
Future work: dense segmentation