1. Primal Sketch Integrating Structure and Texture Ying Nian Wu UCLA Department of Statistics Keck Meeting April 28, 2006 Guo, Zhu, Wu (ICCV, 2003; GMBV, 2004; CVIU, 2006)

2. sketchable image input image + sketch graph synthesized textures texture regions synthesized image = A Generative Model for Natural Images

3. Outline Sparse coding Markov random field Primal sketch model Sketch pursuit algorithm

4. Sparse Coding Olshausen and Field (1996)

5. input image 500 bases 800 bases matching pursuit Matching Pursuit Mallat and Zhang (1993)

6. Symbolic representation of 300 bases Reconstructed image Primak sketch

7. Markov Property: MRF model = Gibbs distribution One example of neighborhood Markov Random Fields Besag (1974) Geman and Geman (1984) Cross and Jain (1983)

8. MRF model & Image ensemble Image ensemble (Wu, Zhu & Liu, 2000) MRF model (Zhu, Wu & Mumford, 1997)

9. Feature statistics : histograms of filter responses (Heeger and Bergen, 1995) Filtering – convolution original image Ifilter responses J of the “dy” filter a set of filters F

10. Histogram of Filter Responses

11. Average histogram error

14. 800 bases 50*70 patch A sample of image ensemble with 5*13=65 parameters

15. observed image sampled image from image ensemble Primak sketch

16. Sparse Coding vs. MRF MRF models target high complexity patterns. Sparse Coding models target low complexity patterns. const: related to the dictionary p*: fitted MRF q: any distribution

17. Primal Sketch Model Image pixels = Sketchable & Non-sketchable Sketchable: sparse coding using image primitives Non-sketchable: feature statistics/Markov random fields Integration: Non-sketchable interpolates sketchable Non-sketchable recycles failed sketch detections

18. Sketches Elder and Zucker, 1998

19. Sketch Graph Vertices: 1,2,3 – corners 4,5,6,7 – end points 8,9,10 – junctions, etc Sketch graph

20. Image Primitives a) Geometric b) Photometric

21. Geometric Photometric sketch image Sketch Graph Model

22. Sketchs = Gabor clusters Alignment across spatial and frequency domains

23. Non-alignment Pool marginal histograms Gabor filters Sketches Alignment Sketch Graph Textures Integrating structure and texture

24. Model fitting First: Sketch pursuit aided by Gabors Second: Non-sketchable texturing Sketchability test

25. Sketch pursuit objective Approximated model

26. Sketch Pursuit Phase I input imageEdge/ridge mapedge/ridge strength Proposals: a set of sketches as candidates. Select the sketches in the order of likelihood gain.

27. Sketch Pursuit Phase II Refinement Evolve the sketch graph by graph operators. Initialization Refinement

28. Graph Operators operatorsgraph changeillustration G 1, G 1 ’create / remove a stroke G 2, G 2 ’grow / shrink a stroke G 3, G 3 ’connect / disconnect vertices G 4, G 4 ’ extend one stroke and cross / disconnect and combine G 5, G 5 ’ extend two strokes and cross / disconnect and combine

29. operatorsgraph changeillustration G 6, G 6 ’ combine two stroke / break a stroke G 7, G 7 ’ combine two parallel strokes / split one into two parallel G 8, G 8 ’ merge two vertices / split a vertex G 9, G 9 ’create / remove a blob G 10, G 10 ’ switch between a stroke(s) and a blob

30. Graph Editing Phase I Phase II A B AB

31. Phase II Algorithm 1.Randomly choose a local sub-graph (S 0 ) 2.Try all 10 pair of graph operators 1~ 5 steps, to generate a set of new graph candidates (S 4,S 2,S 3 ) 3.Compare all new graph candidates 4.Select the one with the largest posterior gain (e.g. S 4 ), accept the new graph. If no positive gain, no update. 5.Repeat 1~4 until no update S0S0 S1S1 S2S2 S3S3 G1G1 S4S4 G3G3 G4G4

33. synthesized texturestexture regions K-mean clustering Histograms in 7x7 window 7 filters x 7 bins

34. Primal Sketch Model Result input image reconstructed image sketch graph sketchable image

35. input imagesketch graph reconstructed imagesketchable image

36. input imagereconstructed imagesketch graph

37. input imagesketch graphreconstructed image

45. Lossy Image Coding sketchable image sketch graph codes for the vertices: 152*2*9 = 2,736 bits codes for the strokes: 275*2*4.7 = 2, 585 bits codes for the profiles: 275*(2.4*8+1.4*2) = 6,050 bits Total codes for structures (18,185 pixels) 11,371 bits = 1,421 bytes

46. synthesized textures texture regions codes for the region boundaries: 3659*3 = 10,977 bits codes for the texture histograms: 7*5*13*4.5 = 2,048 bits Total codes for textures (41,815 pixels) 13,025 bits = 1,628 bytes Total codes for whole image (72,000 pixels), 3,049 bytes

48. Scaling

50. Wu, Zhu, Bahrami, Li (2006)