Walk-Through Illustrations: Frame-Coherent Pen-and-Ink Style in a Game Engine Bert Greudenberg Maic Masuch Thomas Strothotte

It Looks All Crappy Cartman: Aw, screw it. It probably isn't all that good anyway. Kyle: Cartman! What are you talking about?! You LOVE Terrance and Phillip! Cartman: Yeah, but the animation's all crappy — it probably can't sustain itself over ninety minutes.

Who wants crappy animation? Tired of the gothic space dungeons? Tired of the gothic space dungeons? Tired of “realistic” animation that fools you for five seconds? Tired of “realistic” animation that fools you for five seconds? You need a change You need a change A non-photorealistic change! A non-photorealistic change!

3D Game Engines 3D game engines have made huge advances 3D game engines have made huge advances Engines primarily still focus on photorealistic images Engines primarily still focus on photorealistic images Game images are technically non photorealistic Game images are technically non photorealistic

NPR Here to Stay Research has turned to NPR of still images Research has turned to NPR of still images Great advances in Toon, Painterly, and Pen & Ink (i.e. South Park) Great advances in Toon, Painterly, and Pen & Ink (i.e. South Park) However, animating NPR or real-time rendering 2 problems occur. However, animating NPR or real-time rendering 2 problems occur.

What’s the Problem? Maintenance of frame-to-frame coherence Maintenance of frame-to-frame coherence –Hidden Surface Removal Objects appear and disappear correctly Objects appear and disappear correctly Ability of a 3D engine to render NP images in real time Ability of a 3D engine to render NP images in real time

Solution! Create an engine that facilitates a hybrid rendering pipeline Create an engine that facilitates a hybrid rendering pipeline Allows specification of the rendering style on a per object basis Allows specification of the rendering style on a per object basis More to follow

Computer Games and NPR No games achieves true photorealism No games achieves true photorealism Rendering style can be called photorealistic Rendering style can be called photorealistic –Shapes, spatial relationships, surface details

NPR Video Games Very few games incorporate NPR elements Very few games incorporate NPR elements Examples: Examples: –Cartoon Rendering –Pen & Ink Broken Arrow Broken Arrow

Cartoon Rendering Shading Model in 2D cartoon Shading Model in 2D cartoon –Games only use 2D data. –Visually appealing, but not that interactive Monkey Island III Monkey Island III Broken Sword Broken Sword

Monkey Island III

Pen and Ink Rendering NPRQuake by Gleicher NPRQuake by Gleicher Intercepts the calls to graphics libraries Intercepts the calls to graphics libraries Replaces them with different drawing primitives Replaces them with different drawing primitives Uses core GLQuake engine and maintains gameplay Uses core GLQuake engine and maintains gameplay

NPRQuake Sketch

NPRQuake Blue-Print

NPRQuake Brush-Stroke

Problems with NPRQuake Achieves non-photorealistic look Achieves non-photorealistic look Style elements do not preserve frame coherence Style elements do not preserve frame coherence –Outlines of objects vary from frame to frame –Results in lively but disturbing experience

How to Fix It? Few systems deal with frame to frame coherence. Few systems deal with frame to frame coherence. Many systems can create Pen & Ink illustrations Many systems can create Pen & Ink illustrations –But, probability is used to model the line deviations of hand drawings Not suitable for real time rendering in animations Not suitable for real time rendering in animations

Non-Photorealistic Visualization In Game Engine NP imaging allows the user to simplify an image by omitting details to control the viewer’s attention NP imaging allows the user to simplify an image by omitting details to control the viewer’s attention –Emphasize, Deemphasize certain elements of a scene Aim for pen-and-ink illustrations Aim for pen-and-ink illustrations –Silhouettes and creases are clearly drawn –Surface detail added through hatching

Hybrid Rendering Pipeline Able to specify rendering style of each object Able to specify rendering style of each object –Gain additional freedom in expression for the visualization Photorealistic Images – use normal texture mapping and shading Photorealistic Images – use normal texture mapping and shading NP Images – consist of modified shading, an outline, and NP textures (ink maps or hatch maps) NP Images – consist of modified shading, an outline, and NP textures (ink maps or hatch maps) Hybrid – combine photorealistic rendering and NP drawing styles Hybrid – combine photorealistic rendering and NP drawing styles

NP Elements in 3D Game Engine Properties of 3D game engine Properties of 3D game engine –Provide player with highly immersive experience Enhace speed, realism, and image quality Enhace speed, realism, and image quality –Preprocess as much data as possible –Let hardware handle most of the per-frame load Mip-Mapping, anti-aliasing Mip-Mapping, anti-aliasing

NP Elements in 3D Game Engine Non-Photorealism uses image space elements Non-Photorealism uses image space elements –Pixels generated on the screen are not a strict projection of the 3D scene objects –A triangle outline rendered as lines has a constant width independent of the actual distance of the triangle to the viewer

Outlining Silhouettes Silhouettes –Drawn to enhance the visual separation of objects from each other and background –Silhouette edges need to be determined every frame –Non-Convex edges are never drawn (second) –Sharp Edges are always drawn (third) –Only smooth convex edges are drawn (fourth) –Result with hidden lines removed

Hatching Textures are supposed to become smaller with increasing distance Textures are supposed to become smaller with increasing distance Hatching Hatching –shade by drawing or etching fine parallel or perpendicular lines –Pen & Ink hatching textures cause serious moire´e patterns –In motion this produces even worse artifacts than still images Prevents frame coherence Prevents frame coherence

How to Fix This? Common antidote Common antidote –Mip-mapping –Unfortunately, normal filtering process applied to large white areas and thin black lines of hatching texture cause lines to vanish –Visible structure is lost

Hatch Maps Normal mip-mapping Normal mip-mapping –Lines are black and well separated at the top level –Become lighter and closer at lower levels Hatch Map Hatch Map –Equally space black hatch lines for all mip-map levels –Only very lowest levels are gray values used to maintain tone

Hatch Maps Unfortunately, all over tone is not continuous Unfortunately, all over tone is not continuous Abrupt switch between hatch-map levels Abrupt switch between hatch-map levels Use Tri-linear Filtering Use Tri-linear Filtering

Ink Maps Hatching is a mechanical process Hatching is a mechanical process Hybrid Pipeline Hybrid Pipeline –Photorealistic: Mip-Mapping –Non-Photorealistic: Ink Map Adding details to surfaces is artistically challenging Adding details to surfaces is artistically challenging Construct ink maps Construct ink maps –Smooth animation achieved with carefully crafted ink maps –Frame Coherence is maintained using tri-linear filtering

What They Achieved Part of a medieval palace created for the UNREAL engine Part of a medieval palace created for the UNREAL engine

What they Achieved

Specifications 3D engine runs on: 3D engine runs on: – 750 MHz CPU –GeForce2 ultra graphics card Even on unoptimized models, engine runs at interactive speeds, i.e. 25 fps Even on unoptimized models, engine runs at interactive speeds, i.e. 25 fps

And Now….Zelda!!! The Legend of Zelda: The Wind Wakers The Legend of Zelda: The Wind Wakers Right now, this is the pinnacle of NPR in game engines. Right now, this is the pinnacle of NPR in game engines. –First game to max out hardware of Nintendo Gamecube –Combines great use of interactive 3D and NPR

Screenshots

More

Zelda Video!! Thank You Any Questions?