Orientable Textures for Image- Based Pen-And-Ink Illustration Michael P. Salisbury Michael T. Wong John F. Hughes David A. Salesin SIGGRAPH 1997 Andrea Rowan January 23, 2001

Outline Introduction to Image-Based Rendering Difference Image Algorithm –Foundations –Interactive System –Rendering Results Problems Future Work

Introduction Geometry-Based Systems Reads in 3-D geometry of scene Slow for complex objects Faster for walkthrough of scene / manipulation of objects Image-Based Systems Reads in a 2-D grayscale image Same speed for complex objects Slower for walkthrough of scene / manipulation of objects vertices: faces: v v v v v v v This paper’s algorithm uses an image- based system!

Algorithm - Foundations Target (Tone) Image –Defines the tones at every point in the grayscale image –0.0 (white) to 1.0 (black)

Algorithm - Foundations Direction Field –Defines the desired orientation of strokes at each region of the illustration

Algorithm - Foundations Stroke Example Set –Set of strokes that will be used to fill in tone areas –One stroke randomly chosen from the set each time

Interactive System Editing tone - user can lighten or darken reference image

Interactive System Editing direction - user can modify the direction image –comb - changes “direction” to match motion of cursor –blending tool - smooth between regions of different direction –region-filling tools Source toolConstant Direction FillInterpolated Fill

Interactive System Applying stroke –Vertical vector of stroke sample matches direction vector in direction image –Strokes placed dynamically (extra strokes for diverging field, strokes bent with direction) Direction imageStatic strokesDynamic strokes

Rendering Previous Steps were User-controlled Rendering is entirely automated importance - the fraction of darkness that has not been added to a section of the image. –Density of strokes is directly related to darkness in tone image.

Rendering - Steps Making illustration match tone image –Illustration is b/w, and tone image is grayscale –Divide screen space into regions Size of region depends on color in tone image (Larger regions for lighter areas of tone image) Regions Varying Region Size Constant Region Size

Rendering - Steps Making illustration match tone image –When adding a stroke, add blurred stroke to region –Difference image = tone image value - blurred version of illustration value –Importance image = current difference value/initial difference value

Rendering - Steps Example: Consider a pixel with initial tone value of 0.2, initial illustration value (as for all regions) is 0. At the start: difference image = tone - blurred illustration = = 0.2 importance image = current difference/initial difference = 0.2/0.2 = 1. Want this to approach zero or some min. threshold Add a stroke, which blurred, adds.15 to the value difference image = = 0.05 importance image =.05/.2 = Importance value decreases with each stroke.

Rendering - Steps Drawing next Stroke in the Right Place –When a stroke is drawn, pixels in the area of the stroke lose their importance –Quadtree keeps track of most important pixel or region of pixels –Next stroke is drawn at most important pixel

Rendering - Steps When to Stop the Illustration –Importance values get closer to zero with each stroke –Exact match of zero is difficult –Illustration stops when some minimum importance value is attained for each pixel

Rendering - Approximations Assumption 1: Blurred version of multiple strokes is same as sum of blurred versions of independent strokes difference image = tone - blurred illustration OR difference image = tone - (blurred illustration old + blurred stroke new ) difference image = difference image old - blurred stroke new

Rendering - Approximations Assumption 1: Cont’d –Depends on strokes not overlapping –Points where strokes cross will be counted as darkened twice –Illustration is in black & white, so two strokes crossed is the same darkness as one stroke –Solution is hacked with lightening factor stored in a “darkness-look-up-table” –Example: If region is 50% gray (0.5), 90% of pixels drawn are visible Reduce darkness of blurred strokes to 90% before adding blurred value to illustration

Rendering - Approximations Assumption 2: Simplified filtered image of stroke for computations –Render control hull as blurry line –Width = 2h/t h = stroke thickness (mm) t = desired tone value (0.0=white to 1.0=black) Clamped from 1 to 10 mm.

Drawing a Stroke Orienting and Bending –Control hull - frame around each stroke –Broken into parts and mapped to direction image PiPi P i+1 P i+2 P i+3 Stroke Direction imageIllustration PiPi P i+1 P i+2 P i+3

Drawing a Stroke Clipping –When direction field changes rapidly –When stroke crosses into a region that is already dark enough

Output Enhancements Variable Width –Pen and ink pressure can vary from start of stroke to end –Each stroke has 3 widths Start width Middle width End width –Different ratios if drawing hair vs. shadows

Output Enhancements “Wiggles” –Artists don’t draw with rulers! –For strokes > 5 mm –Add control points to control hull –Random perturbation within range (-0.15 to 0.15 mm)

Results Stroke density adjusts for different sized images

Results Performance (SGI 180MHz R500 processor)

Other Work Michael Kowalski et al., “Art-Based Rendering of Fur, Grass, and Trees,” SIGGRAPH 1999 Use off-screen grayscale rendering of scene as reference image Convert 2-D screen position to 3-D space for interactive geometry-based system

Accomplishments Textures appear attached to objects (this is difficult in image-based rendering) Good algorithm for stroke density in image- based rendering

Problems High degree of user-tweaking (direction image) –Direct user interaction with illustration causes performance hit Poor performance (every pixel is looked at)

Future Work User interaction with pen-and-ink illustration, not direction image Support of coherent textures (bricks, fabrics, etc.)