1. Style/Content separation Evgeniy Bart, Dan Levi April 13, 2003

2. Artistic styles Photograph

3. Artistic styles Impressionist

4. Artistic styles Expressionist

5. Artistic styles Pointillist

6. Photographic styles *Pictures by Aya Aner-Wolf

7. Fonts ABCDE ABCDE ABCDE ABCDE Style Content

8. Faces *Images from FERET database

9. Tasks Extrapolation: Extrapolation of familiar style to new content

10. Tasks Extrapolation: Extrapolation of familiar content to new style

11. Tasks Translation:

12. Task specification by analogy : : ? Image analogies, Hertzmann et al.

13. Wei&Levoy Ashikhmin

14. Region growing Somewhat similar to quilting *Picture from presentation by Tal and Zeev

15. Region growing

16. Combining the two … : :

17. … hierarchically : :

18. : :

19. : :

20. Selecting best match

21. Arbitrating ? yes no Use Wei&Levoy value Use Ashikhmin value

22. Arbitrating ? yes no Use Wei&Levoy value Use Ashikhmin value

23. Arbitrating ? yes no Use Wei&Levoy value Use Ashikhmin value

24. : : Results – artistic filters

25. : :

26. : :

27. Super-resolution : Training 1

28. Super-resolution : Training 2

29. Super-resolution : Training 3

30. Super-resolution Results :

31. Super-resolution Results :

32. A lonely pine is standing In the North where high winds blow. He sleeps; and the whitest blanket wraps him in ice and snow. He dreams - dreams of a palm-tree that far in an Orient land Languishes, lonely and drooping, Upon the burning sand. H. Heine, translated by L. Untermeyer

33. Texture by numbers : :

34. Parameters annEpsilon: [float] = 1.000000 ashLastLevel: [bool] = false biasPenalty: [float] = 0.000000 cheesyBoundaries: [bool] = true coherenceEps: [float] = 5.000000 coherencePow: [float] = 2.000000 createSrcLocHisto: [bool] = false decayWeight: [double] = 0.000000 filterColorspace: [enum] = {Lab, Luv, RGB, XYZ} filterMM: [string] = (none!) filterModeMask: [string] = (none!) filterProcedure: [enum] = {Copy, Synthesize} filteredFeatureType: [enum] = {Difference, Raw} filteredPyramidType: [enum] = {Gaussian, Laplacian} finalSourceFac: [float] = -1.000000 gainPenalty: [float] = 0.000000 heurAnnEpsilon: [float] = 1.000000 heurMaxTSVQDepth [int] = 7 histogramEq: [bool] = false levelWeighting: [float] = 1.000000 matchBtoA: [bool] = false matchGrayHistogram: [bool] = false matchMeanVariance: [bool] = false maxTSVQDepth: [int] = 20 maxTSVQError: [float] = 0.000000 modeMaskWeight: [float] = 0.010000 neighborhoodWidth: [int] = 5 pyramidType: [enum] = {Gaussian, Laplacian, Steerable} samplerEpsilon: [float] = 0.100000 searchMethod: [enum] = {ANN, Ash, HeurANN, HeurTSVQ, Image, MLP, TSVQ, TSVQR, Vector} sourceColorspace: [enum] = {Lab, Luv, RGB, XYZ} srcWeight: [float] = 1.000000 targetMM: [string] = (none!) targetModeMask: [string] = (none!) useBias: [bool] = false useFilter: [bool] = true useFilterModeMask: [bool] = false useGain: [bool] = false useInterface: [bool] = true useRandomStart: [bool] = true useSigmoidalDecay: [bool] = false useSplineWeights: [bool] = true useTargetModeMask: [bool] = false useYIQ: [bool] = false numHiddenNeurons: [int] = 20 numLevels: [int] = 2 numPasses: [int] = 1 numTSVQBacktracks: [int] = 8 onePixelSource: [bool] = false oneway: [bool] = false pyramidHeight: [int] = 4

35. 3D rotation : :

36. : :

37. : :

38. : :

39. : :

40. What went wrong? There is some structure, but not simple correspondence Need more knowledge about objects

41. Rectangular parallelepipeds (cuboids)

42. Representation by 3D point coordinates Linear classes, Vetter&Poggio

43. May combine linearly += + =

44. Only 3 dimensions ++= = Call it linear class

45. Linear operators Linear operator L

46. Example: rotation ++= =

47. Rotation If Then

48. Example: projection ++= =

49. Projection If Then

50. Example: projection + rotation ++= =

51. Rotation + projection If Then But also We may work entirely in 2D domain!

52. + + Working in 2D : : + +

53. Results : :

54. : :

55. Can we apply this idea to faces? Linear class assumption: Object may be represented as linear combination of other (similar) objects ++ = Use raw images as basis Reconstruction quality will be poor PCA reconstruction is much better Can we use PCA?

56. + + Using linear classes : : + +

57. Eigenfaces do not correspond Cannot use the same coefficients

58. So far: Solving Specific Tasks (Image analogies), Linear Classes Learn Interaction Generalize To New Examples Goal : General Style/Content Framework

59. .............

60. Motivation : Linear Models Faces Images Form A linear subspace Illumination variations (of same face) can be modeled by a low-dimensional linear space (Hallinan ‘94) Eigenvector Basis Reconstruct faces Model: Linear In Style And In Content eigenfaces (Turk, Pentland ‘91)

61. Bilinear Models (Tenenbaum, Freeman 2000,97) is bilinear if : Linear in x Linear in y y constant

62. Bilinear Forms : Example x,y ∈ Real f(x,y) = xy Bilinear Forms To Model Style And Content 2 Models : Symmetric, And Asymmetric

63. K Image in Style s Content c Style vector I J Content vector IXJ Interaction Matrix Symmetric Bilinear Model

64. K Image in Style s Content c J Content vector Symmetric Bilinear Model Pixel Style vector J Style Matrix AsAs JXK

65. Symmetric Bilinear Model : basis vectors

66. Toy Example: Symmetric Model Images: Style: Color Content: Shape (0,0,1,1) (2,2,2,0) (0,3,0,3) Content: (0,0,1,1) (1,0,1,0) (0,1,1,1) Style: 123123

67. Face Example - Symmetric Style: Pose, Content: Person

68. K Image in Style s Content c Style vector I J Content vector IXJ Interaction Matrix Asymmetric Bilinear Model WksWks

69. K Image in Style s Content c J Content vector Asymmetric Bilinear Model Pixel Style vector J Style Matrix AsAs JXK A s : Content  Images

70. K Image in Style s Content c J Content vector Asymmetric Bilinear Model J

71. ……… J K Image in Style s Content c A style specific basis Mixed by content coefficients

72. Content: (0,0,1,1) (1,0,1,0) (0,1,1,1) Toy Example: Asymmetric Model Style:

73. Face Example - Asymmetric Style: Pose, Content: Person

74. Training Person Illumination Style Content Image Matrix:

75. Training – Model Fitting Problem : Given {y sc (t) } t = 1..T find model parameters Error Minimization: Asymmetric: Closed SVD solution or Quasi-Newton methods  A s, b c Free parameter: J – content vector dimension. Symmetric: Iterative solution using SVD  a s, W k, b c

76. Content: person Style: illumination Translation - Faces Asymmetric Model Cannot handle translation!

77. Problem: C = 23 (faces), S = 3 (illuminations) Training : Fit a symmetric model using iterative SVD with I = S, J = C  a s, W k, b c Generalization: find a s`, b c ` that minimize E* =  k | y k s`c` - a s` W k b c` | 2 Alternating Iterative Linear Solution Translation : Produce a s` W k b c, a s W k b c` for each s and c. Translation – Symmetric Model

78. Translation - Results

79. Extrapolation Content: Letter, Style: Font Main Problem : Image Representation Linear combinations of letters should look like a letter.

80. A Displacement Vector Warp Map

81. Coulomb Warp Map For unique mapping: physical model of electrostatic forces. Linear combination of letters looks like a letter

82. Extrapolation Scheme Fit An Asymmetric bilinear model S = 5 training fonts (styles), C = 62 characters (content), K=2888 data dim. closed-form SVD  A s, b c C={c 1,…,c M } letters in a new style s’  find best fitting A s’ Minimize : E* =  c ║y s’c - A s’ b c ║  ∂E*/ ∂ A s’ = 0  Set J High(~60) Overfitting on Test Data (173,280 degrees of freedom!)

83. Constraint: Close To Symmetric A OLC =  s α s A s (Optimal Linear Combination) α s - style parameters (symmetric) Minimize: E* =  c ║y s’c - A s’ b c ║ + λ║ A s’ - A OLC ║ ∂E*/ ∂ A s’ = 0 Extrapolate missing letters by: y s’c = A s’ b c

84. Results

85. Symmetric vs. Aymmetric Can reduce dimensionality of factors Learns the structure of factor interactions : handles translation More Flexible Too flexible: overfitting Cannot handle translation Can be overcome by combining both

86. Bilinear Models (Tenenbaum, Freeman 2000,97) General framework for two factor problems Explicit parameterized representations of each factor and their interaction Natural generalization for extrapolation and translation tasks Fast algorithms (SVD) Pros: Cons: Assumes Linearity In Each Factor Find Clever Input Representations Decompose To Sub Problems

87. Example-Based Style Synthesis ( Ido Drori Hezi Yeshurun Daniel Cohen-Or 03)

88. Algorithm Outline 1.Divide Image To Overlapping Tiles

89. Algorithm Outline 1.Divide Image To Overlapping Tiles 2. Find Best Match In Each Scene

90. Algorithm Outline 1.Divide Image To Overlapping Tiles 2. Find Best Match In Each Scene 3.Synthesize tiles By Bilinear Model

91. Algorithm Outline 1.Divide Image To Overlapping Tiles 2. Find Best Match In Each Scene 3.Synthesize tiles By Bilinear Model 4. Image Quilting

92. Algorithm Outline 1.Decompose Image To Tiles 2. Find Best Match In Each Scene 3.Synthesize tiles By Bilinear Model 4. Image Quilting 5. Image Analogies

93. Create Gaussian Pyramids For Examples And Input Images Apply Algorithm To Each Level From Coarse To Fine

94. Finding Best Matching Fragment Similar GeometryAgreeing Boundaries V search = (,,,,, ) Gradient Laplacian Luminance For Each Training Scene : Create V In every Position And Orientation Search For Nearest Neighbor