ShadowDraw Real-Time User Guidance for Freehand Drawing Larry Zitnick, Michael Cohen Microsoft Research Yong Jae Lee U. of Texas at Austin
Drawing Subject 1 Subject 2 We need help..
Tracing Difficult to find photos Limits creativity
Drawing a face Learn a set of rules Requires significant training
Our Idea: ShadowDraw ShadowDraw
Creating shadows = Shadow Shadow gives gist of many images simultaneously + Collection of images…
Creating shadows What if the only input is the user’s drawing? Shadow Drawing Requires partial matching and dynamic updates
Speed Real-time updating of shadows is critical 0.1 seconds – “feels right” 0.5 seconds – “useful” 2 seconds – “useless”
Overview User Drawing For each sub-window Histogram ( ID x d x x d y ) Top 100 Fine Alignment Spatial Scoring Shadow Verify Query time (online) Database ImageEdges For each sub-window Sketch, Img_ID, x, y …. Sketch, Img_ID, x, y Min Hash Database (offline)
Database
Efficient (sub-linear) retrieval Stores image index and offset Inverted file structure: [Zitnick, ECCV 2010] Edge descriptor: Position Orientation Linear length Database ImageEdges For each sub-window Sketch, Img_ID, x, y …. Sketch, Img_ID, x, y Min Hash Database (offline)
Matching Second stage: Refine alignment and compute blending weights First stage: Use inverted file structure to find candidate set User Drawing For each sub-window Histogram ( ID x d x x d y ) Top 100 Fine Alignment Spatial Scoring Shadow Verify Query time (online)
Fine alignment Hough transform (3D)Break into three 1D problems: ΔYΔY ΔXΔX ΔXΔX ΔYΔY ΔS (scale) ΔSΔS User Drawing For each sub-window Histogram ( ID x d x x d y ) Top 100 Fine Alignment Spatial Scoring Shadow Verify
Blending weights = Shadows are the composite of many images * * + Pen Strokes Top matchesWeights
Blending weights = * * + Shadows are the composite of many images Top matches Weights Pen Strokes
Rendering + Pen Strokes Pen PositionShadow * + = Output Higher contrast near pen position Main focus is user’s drawing
Related work Interactive drawing interfaces – Teddy [Igarashi et al., 1999] – Fluid Sketches [Arvo and Novins, 2000] – 3D drawing system [Igarashi and Hughes, 2000] – iCanDraw [Dixon et al., 2010] Drawing studies – Where do people draw lines? [Cole et al., 2008]
User studies 30,000 images, 20 categories 16 drawers (8 men, 8 women), 8 evaluators 5 objects (shoe, face, bicycle, butterfly, rabbit)
User studies Good drawers With ShadowDraw Without ShadowDraw
User studies Bad drawers With ShadowDraw Without ShadowDraw
User studies Average drawers With ShadowDraw Without ShadowDraw
User studies Average drawers With ShadowDraw Without ShadowDraw
User studies Significant improvement for “Average” group
User studies Improvement for most object categories Rabbit is control variable
User studies After training all users improved: Poor skillGood skill Subject’s personal style is maintained!
Future work Matching against drawings and not photographs Temporal strokes Color, shading, etc. Drawing “priors” Not future work: Changing people’s strokes
Live demo