Transfer Learning of Object Classes: From Cartoons to Photographs NIPS Workshop Inductive Transfer: 10 Years Later Geremy Heitz Gal Elidan Daphne Koller December 9 th, 2005

Localization vs. Recognition Traditional question: “Is there an object of type X in this image?” Airplane? NO Human? YES Dog? YES Our question: “Where in this image is the object of type X?” MAN DOG The man is walking the dog

Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results

Shape Model Set of landmarks Piecewise-linear contour between neighbors Features of individual landmarks Features of pairs of landmarks tail nose

Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results

“Registering” the Model to an Image Requires assigning each landmark to a pixel location ? ?

Localization Are local cues enough? Need to jointly consider all cues (features) “Correct” pixel is often not the best match! Markov Random Field Potentials = Functions of local and global features Inference using max-product Registration = Most Likely Assignment L nose L tail L under L cockpit

Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results

Bootstrap from simple instances where outlining is easy = cartoons / drawings Learning Challenge Hand LabelHidden Variables Costly, and time-consuming Where to start? Local optima problem no confusing background outline (shape) is easily recovered using snake ? ? ? ? ?

Learning from Cartoon Drawings Registration Shape Learning Shape and Appearance Learning +

Phase I: Learning from Cartoons Extract high resolution contour using snake Create shape-based model from training contours Pairwise merging of models Selection of landmarks Registration Pyramid Final Shape Model

Training Set Selection high score low score Phase II: Learning from Images Correspond initial model to training images Select best correspondences as training instances Learn final shape- and appearance-based model Cartoon Phase Model Natural Image Model Transfer

Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results

Localization Results sample training cartoons sample registration

Transfer of Object Shape Transfer of shape speeds up learning Benefit of shape transfer # images in phase II Average overlap transfer no transfer

Learning Appearance No Appearance FG/BG Appearance Average overlap Shape template shape + appearance # images in phase II

Training Instance Selection AUTOPICKED AUTO PICKED HAND Average overlap # images in phase II

Summary and Future Work  Flexible probabilistic shape model  Effective registration to images  Transfer  Shape from cartoons  Appearance from real images Develop a better appearance model Investigate self-training issues Transfer from one class to another

Thanks!

Cartoon vs. Hand Segmentation Number of Training Instances Mean Overlap Score Learned from Drawings Hand Constructed Human Inter-Observer cartoon hand segmented Learning shape from cartoons is competitive with hand segmentation!

Landmark Features Shape Template Patch Appearance (Foreground/Background) Location

Prediction False positive rate True positive rate objectrecognition car side86% cougar86% airplane86% buddha84% bass76% rooster73%  Comparable to constellation w/ 5 instances (Fei Fei et. Al)  Leading (discriminative) methods require many instances