Object Tracking Based on Appearance and Depth Information

outline NN tracker How to make use of top-view information

Challenges changing appearance patterns of both the object and the scene; object-to-object and object-to-scene occlusions; camera motion loss of information from 3D world on a 2D image, —scene illumination changes —complex object motion, —real-time processing requirements. 3

NN Tracker A tracking-by-detection framework , combines nearest-neighbor classification of bags of features A framework that handles occlusion, background clutter, scale and appearance change Steve Gu, Ying Zheng, and Carlo Tomasi, "Efficient Visual Object Tracking with Online Nearest Neighbor Classifier". (ACCV2010), Queenstown, New Zealand, November 8-12, 2010

-----The main advantages of tracking by detection come from the flexibility and adaptability of its underlying representation of appearance.

Tracking-by-detection framework

Tracking result with dense-sift

Information from Top-view Position Distance

Initialize output : Input : obj & bg Model RGB Image target boundingbox pointsCloud para of motion output : obj & bg Model TVI Convexhull TransMatrix Position/Scale prediction

From RGB to TV

Predict the position in TVI Given: position in previous TVI Use transM Nearest neighbor strategy

TVI to RGB See RES1 Why don’t we just use this prediction to find obj ? It can’t recover from the fault of robot motion

Tracking Appearance model dense-sift feature pathchsize :32, gridsize:8 Motion model predicted position/ expected window size See RES2

How to handle occlusion Small occlusion: threshold for matched feature large occlusion if report ‘losing target’, then discard current frame, keep current state, take another image….

How to handle object move or false position prediction Means we can’t tracking the object in predict location. Then search the whole image Still can’t find matched object? Then - stop .or. - rotate to find object, if founded, take it as the first frame and proceeding further motion.

Main Challenge How many matched features means target is found? Too large : miss the target Too small : false positive From testing : 20 now

Next step to try…. Only evaluate these convexhull Possibility from position prediction