Separating Style and Content with Bilinear Models Joshua B. Tenenbaum, William T. Freeman Computer Examples Barun Singh 25 Feb, 2002

PHILOSOPHY & REPRESENTATION Data contains two components: style and content Want to represent them separately Symmetric Bilinear Model: y : observed data a : style vector b : content vector I, j : components of style and content W : matrix of basis vectors (e.g., “eigenfaces”) Y : (SK) x C A : (SK) x J b : J x C Asymmetric Bilinear Model: A : matrix of style-specific basis vectors More flexible model Easier to deal with

PROBLEMS TO BE SOLVED Given a labeled training set of observations in multiple styles and content classes,  Fit asymmetric model (find A and b for known styles and contents) using SVD  Find style matrix that best explains data for incomplete style (i.e., minimizes E given below)  Extrapolate using the estimated style matrix  OLC used to solve overfitting problem  Parameters involved:  = 0 : Purely asymmetric model  =  : Purely symmetric model  extrapolate a new style to unobserved content classes

PROBLEMS TO BE SOLVED Given a labeled training set of observations in multiple styles and content classes,  Use separable mixture model (SMM) with EM algorithm to determine style matrix for new style  Parameters: model dimensionality J, model variance  2, max number of EM iterations t max  classify content observed in a new style  Fit asymmetric model  Select content class c that maximizes Pr(s’,c| y )

PROBLEMS TO BE SOLVED Given a labeled training set of observations in multiple styles and content classes,  translate from new content observed only in new styles into known styles or content classes  Fit symmetric model (find W, a, and b for known styles and contents) using iterated SVD procedure  Given a single image in a new style and content type, iterate to find the style and content vectors for the new image (given an initial guess for the new content vector):

TOY EXAMPLE - intro Image made of 4 pixels, each of which are either white or red. Style represents if the top or bottom rows are red or white Content represents if the left or right columns are red or white. SYMMETRIC MODEL Basis Images ( W ) Content Vectors ( b ) Style Vectors ( a ) Output Images ( y )

TOY EXAMPLE - intro ASYMMETRIC MODEL *Note: Images drawn as blocks, but represented as vectors, not matrices Content Vectors ( b ) Style-specific Basis Images ( A ) Output Images ( y )

TOY EXAMPLE - extrapolation ? Fitting the asymmetric model Content Vectors ( b ) Style-specific Basis Images ( A ) Extrapolate

FONTS EXAMPLE - extrapolation Training Set Incomplete Style Content (Letter) Style (Font)

FONTS EXAMPLE - extrapolation Asymmetric Model model dimension model dimension Symmetric Model Sym. W/ Asym. Prior (dim = 60) vs. Actual

TOY EXAMPLE - classification 1: Fit asymmetric model to training set Content Vectors ( b ) Style-specific Basis Images ( A )

TOY EXAMPLE - classification 2: Use Separable Mixture Model w/ EM to classify  2 = 0.5  2 = 0.6  2 = 0.35 Content Vectors ( b ) Style-specific Basis Images ( A ) Actual Resulting Images

FACES EXAMPLE - translation Content : faces Style: ligting

finito