1. ShadowDraw Real-Time User Guidance for Freehand Drawing Larry Zitnick, Michael Cohen Microsoft Research Yong Jae Lee U. of Texas at Austin

2. Drawing Subject 1 Subject 2 We need help..

3. Tracing Difficult to find photos Limits creativity

4. Drawing a face Learn a set of rules Requires significant training

5. Our Idea: ShadowDraw ShadowDraw

6. Creating shadows = Shadow Shadow gives gist of many images simultaneously + Collection of images…

7. Creating shadows What if the only input is the user’s drawing? Shadow Drawing Requires partial matching and dynamic updates

8. Speed Real-time updating of shadows is critical 0.1 seconds – “feels right” 0.5 seconds – “useful” 2 seconds – “useless”

9. Overview User Drawing For each sub-window Histogram ( ID x d x x d y ) Top 100 Fine Alignment Spatial Scoring Shadow Verify Query time (online) Database ImageEdges For each sub-window Sketch, Img_ID, x, y …. Sketch, Img_ID, x, y Min Hash Database (offline)

10. Database

11. Efficient (sub-linear) retrieval Stores image index and offset Inverted file structure: [Zitnick, ECCV 2010] Edge descriptor: Position Orientation Linear length Database ImageEdges For each sub-window Sketch, Img_ID, x, y …. Sketch, Img_ID, x, y Min Hash Database (offline)

12. Matching Second stage: Refine alignment and compute blending weights First stage: Use inverted file structure to find candidate set User Drawing For each sub-window Histogram ( ID x d x x d y ) Top 100 Fine Alignment Spatial Scoring Shadow Verify Query time (online)

13. Fine alignment Hough transform (3D)Break into three 1D problems: ΔYΔY ΔXΔX ΔXΔX ΔYΔY ΔS (scale) ΔSΔS User Drawing For each sub-window Histogram ( ID x d x x d y ) Top 100 Fine Alignment Spatial Scoring Shadow Verify

14. Blending weights = Shadows are the composite of many images * * + Pen Strokes Top matchesWeights

15. Blending weights = * * + Shadows are the composite of many images Top matches Weights Pen Strokes

16. Rendering + Pen Strokes Pen PositionShadow * + = Output Higher contrast near pen position Main focus is user’s drawing

17. Related work Interactive drawing interfaces – Teddy [Igarashi et al., 1999] – Fluid Sketches [Arvo and Novins, 2000] – 3D drawing system [Igarashi and Hughes, 2000] – iCanDraw [Dixon et al., 2010] Drawing studies – Where do people draw lines? [Cole et al., 2008]

18. User studies 30,000 images, 20 categories 16 drawers (8 men, 8 women), 8 evaluators 5 objects (shoe, face, bicycle, butterfly, rabbit)

19. User studies Good drawers With ShadowDraw Without ShadowDraw

20. User studies Bad drawers With ShadowDraw Without ShadowDraw

21. User studies Average drawers With ShadowDraw Without ShadowDraw

22. User studies Average drawers With ShadowDraw Without ShadowDraw

23. User studies Significant improvement for “Average” group

24. User studies Improvement for most object categories Rabbit is control variable

25. User studies After training all users improved: Poor skillGood skill Subject’s personal style is maintained!

26. Future work Matching against drawings and not photographs Temporal strokes Color, shading, etc. Drawing “priors” Not future work: Changing people’s strokes

27. Live demo