1. Separating Style and Content with Bilinear Models Joshua B
Separating Style and Content with Bilinear Models Joshua B. Tenenbaum, William T. Freeman Computer Examples
Barun Singh
25 Feb, 2002

2. PHILOSOPHY & REPRESENTATION
Data contains two components: style and content
Want to represent them separately
Symmetric Bilinear Model:
y : observed data
a : style vector
b : content vector
I, j : components of style and content
W : matrix of basis vectors (e.g., “eigenfaces”)
Y : (SK) x C
A : (SK) x J
b : J x C
Asymmetric Bilinear Model:
A : matrix of style-specific basis vectors
More flexible model
Easier to deal with

3. PROBLEMS TO BE SOLVED
Given a labeled training set of observations in multiple styles and content classes,
extrapolate a new style to unobserved content classes
Fit asymmetric model (find A and b for known styles and contents) using SVD
Find style matrix that best explains data for incomplete style (i.e., minimizes E given below)
Extrapolate using the estimated style matrix
OLC used to solve overfitting problem
Parameters involved: l
l = 0 : Purely asymmetric model
l =  : Purely symmetric model

4. PROBLEMS TO BE SOLVED
Given a labeled training set of observations in multiple styles and content classes,
classify content observed in a new style
Fit asymmetric model
Use separable mixture model (SMM) with EM algorithm to determine style matrix for new style
Parameters: model dimensionality J, model variance s2, max number of EM iterations tmax
Select content class c that maximizes Pr(s’,c|y)

5. PROBLEMS TO BE SOLVED
Given a labeled training set of observations in multiple styles and content classes,
translate from new content observed only in new styles into known styles or content classes
Fit symmetric model (find W, a, and b for known styles and contents) using iterated SVD procedure
Given a single image in a new style and content type, iterate to find the style and content vectors for the new image (given an initial guess for the new content vector):

6. TOY EXAMPLE - intro
Image made of 4 pixels, each of which are either white or red.
Style represents if the top or bottom rows are red or white
Content represents if the left or right columns are red or white.
SYMMETRIC MODEL
Basis Images ( W )
Content Vectors ( b )
Style Vectors ( a )
Output Images ( y )

7. TOY EXAMPLE - intro ASYMMETRIC MODEL Content Vectors ( b )
*Note: Images drawn as blocks, but represented as vectors, not matrices
Content Vectors ( b )
Style-specific Basis Images ( A )
Output Images ( y )

8. ? TOY EXAMPLE - extrapolation Fitting the asymmetric model Extrapolate
Content Vectors ( b )
Style-specific Basis Images ( A ) ?
Extrapolate

9. FONTS EXAMPLE - extrapolation
Training Set Incomplete Style
Content (Letter)
Style (Font)

10. FONTS EXAMPLE - extrapolation
Asymmetric Model
model dimension
model dimension
Symmetric Model
Sym. W/ Asym. Prior
(dim = 60) vs. Actual

11. TOY EXAMPLE - classification
1: Fit asymmetric model to training set
Content Vectors ( b )
Basis Images ( A )
Style-specific

12. TOY EXAMPLE - classification
2: Use Separable Mixture Model w/ EM to classify
Content Vectors ( b )
Basis Images ( A )
Style-specific
Actual
Resulting Images
s 2 = 0.5
s 2 = 0.6
s 2 = 0.35

13. FACES EXAMPLE - translation
Content : faces
Style: ligting

14. finito