Separating Style and Content with Bilinear Models Joshua B Separating Style and Content with Bilinear Models Joshua B. Tenenbaum, William T. Freeman Computer Examples Barun Singh 25 Feb, 2002

PHILOSOPHY & REPRESENTATION Data contains two components: style and content Want to represent them separately Symmetric Bilinear Model: y : observed data a : style vector b : content vector I, j : components of style and content W : matrix of basis vectors (e.g., “eigenfaces”) Y : (SK) x C A : (SK) x J b : J x C Asymmetric Bilinear Model: A : matrix of style-specific basis vectors More flexible model Easier to deal with

PROBLEMS TO BE SOLVED Given a labeled training set of observations in multiple styles and content classes, extrapolate a new style to unobserved content classes Fit asymmetric model (find A and b for known styles and contents) using SVD Find style matrix that best explains data for incomplete style (i.e., minimizes E given below) Extrapolate using the estimated style matrix OLC used to solve overfitting problem Parameters involved: l l = 0 : Purely asymmetric model l =  : Purely symmetric model

PROBLEMS TO BE SOLVED Given a labeled training set of observations in multiple styles and content classes, classify content observed in a new style Fit asymmetric model Use separable mixture model (SMM) with EM algorithm to determine style matrix for new style Parameters: model dimensionality J, model variance s2, max number of EM iterations tmax Select content class c that maximizes Pr(s’,c|y)

PROBLEMS TO BE SOLVED Given a labeled training set of observations in multiple styles and content classes, translate from new content observed only in new styles into known styles or content classes Fit symmetric model (find W, a, and b for known styles and contents) using iterated SVD procedure Given a single image in a new style and content type, iterate to find the style and content vectors for the new image (given an initial guess for the new content vector):

TOY EXAMPLE - intro Image made of 4 pixels, each of which are either white or red. Style represents if the top or bottom rows are red or white Content represents if the left or right columns are red or white. SYMMETRIC MODEL Basis Images ( W ) Content Vectors ( b ) Style Vectors ( a ) Output Images ( y )

TOY EXAMPLE - intro ASYMMETRIC MODEL Content Vectors ( b ) *Note: Images drawn as blocks, but represented as vectors, not matrices Content Vectors ( b ) Style-specific Basis Images ( A ) Output Images ( y )

? TOY EXAMPLE - extrapolation Fitting the asymmetric model Extrapolate Content Vectors ( b ) Style-specific Basis Images ( A ) ? Extrapolate

FONTS EXAMPLE - extrapolation Training Set Incomplete Style Content (Letter) Style (Font)

FONTS EXAMPLE - extrapolation Asymmetric Model 10 20 30 40 50 60 model dimension 10 20 30 40 50 60 model dimension Symmetric Model Sym. W/ Asym. Prior (dim = 60) vs. Actual

TOY EXAMPLE - classification 1: Fit asymmetric model to training set Content Vectors ( b ) Basis Images ( A ) Style-specific

TOY EXAMPLE - classification 2: Use Separable Mixture Model w/ EM to classify Content Vectors ( b ) Basis Images ( A ) Style-specific Actual Resulting Images s 2 = 0.5 s 2 = 0.6 s 2 = 0.35

FACES EXAMPLE - translation Content : faces Style: ligting

finito