NPR - 2D to 3D, painting and rendering Daniel Teece Walt Disney Feature Animation Daniel Teece Walt Disney Feature Animation

IntroductionIntroduction Coming up… Background 2D and 3D brush models Animation problems and approaches Examples of NPR used in film production Sable Coming up… Background 2D and 3D brush models Animation problems and approaches Examples of NPR used in film production Sable

NPR Characteristics What defines an NPR system? “Natural Media Emulation” 2D / 3D Interactivity / Artistic direction Brushes, strokes and marks Complications - visibility … animation! What defines an NPR system? “Natural Media Emulation” 2D / 3D Interactivity / Artistic direction Brushes, strokes and marks Complications - visibility … animation!

NPR Cross-section 3D2D automatic interactive image processors stroke renderers 2D paint systems 3D paint / renderers

NPR Cross-section 3D2D automatic interactive Litwinowicz97 Hertzmann00 Hertzmann98 Meier96 Haeberli90 DeepCanvas Teece98 Kalnins02

2D Strokes 2D paint strokes and reference images Images in, images out Stamp-based, procedural or physically-based Haeberli’s “Paint by Numbers” [Haeberli90] Hertzmann’s subsequent improvements [Hertzmann98] 2D paint strokes and reference images Images in, images out Stamp-based, procedural or physically-based Haeberli’s “Paint by Numbers” [Haeberli90] Hertzmann’s subsequent improvements [Hertzmann98]

Animating Strokes Finding frame to frame coherence Stroke placement methods cause temporal artifacts Locking strokes to canvas improves coherence but can look strange Strokes should move smoothly, matching motion of subject Finding frame to frame coherence Stroke placement methods cause temporal artifacts Locking strokes to canvas improves coherence but can look strange Strokes should move smoothly, matching motion of subject

Image Space Stroke Animation Optical flow techniques “Processing images and video for an impressionist effect” [Litwinowicz97] “Painterly rendering for video and interaction” [Hertzmann00] Strokes moved, added or removed based on frame-to-frame optical flow data Strokes retain two-dimensional nature Optical flow techniques “Processing images and video for an impressionist effect” [Litwinowicz97] “Painterly rendering for video and interaction” [Hertzmann00] Strokes moved, added or removed based on frame-to-frame optical flow data Strokes retain two-dimensional nature

Brush Models - 3D Strokes and Surfaces Pixel (texture maps) Geometry (points & matrices) Procedure (paths & width vectors) From [Katanics,Lappas99] Strokes and Surfaces Pixel (texture maps) Geometry (points & matrices) Procedure (paths & width vectors) From [Katanics,Lappas99]

Painterly Rendering for Animation [Meier96] A 3D stroke renderer Particles placed on geometry, based on surface area Strokes placed at particle positions, with parameters driven by reference renders Depth sorting and layering for visibility Smooth stroke animation, with a painterly appearance A 3D stroke renderer Particles placed on geometry, based on surface area Strokes placed at particle positions, with parameters driven by reference renders Depth sorting and layering for visibility Smooth stroke animation, with a painterly appearance

3D Expressive Paint A 3D interactive painting system Strokes and marks in object space Developed for animating painterly imagery Fully interactive Two types of stroke - 3D geometry and screen aligned A 3D interactive painting system Strokes and marks in object space Developed for animating painterly imagery Fully interactive Two types of stroke - 3D geometry and screen aligned

3D Expressive Paint 3D Strokes are painted interactively by ray-casting in to the stencil scene

3D Expressive Paint Paint order / depth order (“clump buffer”) Silhouettes - coherence & character [Kalnins02] Brush model - lighting, textures etc. Limited animation support (e.g. deforming surfaces) Border or transition strokes between surfaces (back to 2D/3D relationship) Paint order / depth order (“clump buffer”) Silhouettes - coherence & character [Kalnins02] Brush model - lighting, textures etc. Limited animation support (e.g. deforming surfaces) Border or transition strokes between surfaces (back to 2D/3D relationship)

DeepCanvasDeepCanvas A 3D painting and rendering system Painters interface uses Digpaint (in-house 2D paint system) BG Painters use a set of reference layers prepared by TD (grey render, outlines …) Strokes are stored in stroke script files, and mapped on to 3D geometry Renderer ‘re-paints’ scene from new angle A 3D painting and rendering system Painters interface uses Digpaint (in-house 2D paint system) BG Painters use a set of reference layers prepared by TD (grey render, outlines …) Strokes are stored in stroke script files, and mapped on to 3D geometry Renderer ‘re-paints’ scene from new angle

DeepCanvasDeepCanvas A 2D/3D painting and rendering system From [Katanics,Lappas99]

DeepCanvasDeepCanvas Simplified stroke representation - stroke spine and width vectors Simplified stroke representation - stroke spine and width vectors From [Katanics,Lappas99]

SableSable A 2D/3D stroke renderer Library of specifically scanned & processed strokes Operates as a pipeline component with Inka Stroke paths can be any Inka line type Workflow designed to be driven by traditional animation A 2D/3D stroke renderer Library of specifically scanned & processed strokes Operates as a pipeline component with Inka Stroke paths can be any Inka line type Workflow designed to be driven by traditional animation

SableSable Requirements: Match look of original artwork (stylized, flat, few strokes, black background) Retain performance seen in pencil animation Animate strokes smoothly & efficiently Requirements: Match look of original artwork (stylized, flat, few strokes, black background) Retain performance seen in pencil animation Animate strokes smoothly & efficiently

SableSable First attempt - DeepCanvas brush Utility to convert ink lines to stroke scripts All pre-written software (except converter) Problems achieving distinctive look DeepCanvas procedural brush sensitive to segment distribution First attempt - DeepCanvas brush Utility to convert ink lines to stroke scripts All pre-written software (except converter) Problems achieving distinctive look DeepCanvas procedural brush sensitive to segment distribution

SableSable Animated non-mapped DeepCanvas strokes © Disney

SableSable Brush model Texture-based, extracted from example set Library of textures and shapes Stroke geometry is grown along spine curve Sharp twists are problematic … splitting Brush model Texture-based, extracted from example set Library of textures and shapes Stroke geometry is grown along spine curve Sharp twists are problematic … splitting

SableSable © Disney

SableSable Animation Rigged curve set animated to match traditional animation Animation Rigged curve set animated to match traditional animation © Disney

SableSable Results © Disney

SableSable Summary Animate strokes smoothly & efficiently Able to match appearance of strokes Integration with other tools in pipeline Animators able to work with a pencil Summary Animate strokes smoothly & efficiently Able to match appearance of strokes Integration with other tools in pipeline Animators able to work with a pencil

ConclusionsConclusions Combining animation and NPR can be approached with 2D or 3D methods With 2D, frame-to-frame coherence can be difficult to maintain With 3D, some operations may detract from the painterly look Brush models vary, but are important Interaction and workflow are also important Combining animation and NPR can be approached with 2D or 3D methods With 2D, frame-to-frame coherence can be difficult to maintain With 3D, some operations may detract from the painterly look Brush models vary, but are important Interaction and workflow are also important

AcknowledgementsAcknowledgements Mike Gabriel George Katanics Jack Brooks Mike Gabriel George Katanics Jack Brooks