java.sun.com/javaone/sf | 2004 JavaOne SM Conference | Session TS Advanced Game Development with the Mobile 3D Graphics API Tomi Aarnio, Kari Pulli Nokia Research Center

| 2004 JavaOne SM Conference | Session TS Gain an insight to the Mobile 3D Graphics API and take your games to the next level

| 2004 JavaOne SM Conference | Session TS Prerequisites Fundamentals of 3D graphics OpenGL ® or some other modern 3D API Java™ technology, preferably MIDP

| 2004 JavaOne SM Conference | Session TS Objectives Get a quick overview of the whole API Gain a deeper understanding of selected topics Learn practical tricks not found in the spec

| 2004 JavaOne SM Conference | Session TS Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

| 2004 JavaOne SM Conference | Session TS Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

| 2004 JavaOne SM Conference | Session TS Mobile 3D Graphics API (M3G) Also known as JSR-184 Designed for mobile devices ─Primarily CLDC / MIDP ─But also CDC Mobile 3D Graphics API Java Virtual Machine (CLDC 1.1) MIDP Midlets Graphics Hardware

| 2004 JavaOne SM Conference | Session TS Overcome the performance barrier Benchmarked on an ARM9 processor Native (C/C++) vs. Java on mobiles

| 2004 JavaOne SM Conference | Session TS Leverage hardware acceleration Benefits for developers ─Mask hardware differences (3D HW, FPU, DSP,…) ─Exploit any new hardware automatically Benefits for hardware vendors ─Gives a concrete target which features to offer

| 2004 JavaOne SM Conference | Session TS Speed up development cycles Separate content from code ─Quick prototyping without programming ─Artists create looks and behavior ─Export all in a binary file format ─Load the file, populate scene graph Commonly needed functionality built in ─High-level functionality such as scene graph, keyframe animation

| 2004 JavaOne SM Conference | Session TS Why a new standard? OpenGL (ES) is too low-level ─Lots of Java code needed for simple things Java 3D™ API is too bloated ─A hundred times larger than M3G ─Does not fit together with MIDP ─Tried and failed, but… Now knew what we wanted! ─Basic Java 3D™ ideas: nodes, scene graph ─Add file format, keyframe animation ─Remain compatible with OpenGL ES

| 2004 JavaOne SM Conference | Session TS Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

| 2004 JavaOne SM Conference | Session TS Key classes World Graphics3D Loader 3D graphics context Performs all rendering Scene graph root node Can load individual objects and entire scene graphs (M3G and PNG files)

| 2004 JavaOne SM Conference | Session TS Graphics3D Contains global state ─Target surface, viewport, depth buffer ─Camera, light sources ─Rendering quality hints Each renderable object has its own local state ─Geometry and appearance (material, textures, etc.) ─Transformation relative to parent or world

| 2004 JavaOne SM Conference | Session TS Graphics3D: Rendering modes Retained mode ─Render a scene graph, rooted by the World ─Take the Camera and Lights from the World Immediate mode ─Render a branch or an individual node at a time ─Explicitly give the Camera and Lights to Graphics3D

| 2004 JavaOne SM Conference | Session TS Graphics3D: How-To-Use Bind a target to it, render, release the target Tip: Never mess with a bound target Tip: Graphics3D is a singleton (threads!) void paint(Graphics g) { myGraphics3D.bindTarget(g); myGraphics3D.render(world); myGraphics3D.releaseTarget(); }

| 2004 JavaOne SM Conference | Session TS Graphics3D: Rendering targets GraphicsCanvas Image CustomItem Graphics3DImage2D World Can render to textures also M3G MIDP

| 2004 JavaOne SM Conference | Session TS Everything is synchronous ─A method returns only when it’s done ─No separate thread for renderer or loader There are no callbacks ─No abstract methods, interfaces, or events ─Tip: Do not even try to override any methods Scene update is decoupled from rendering ─ render : draws the scene, no side-effects ─ animate : updates the scene to the given time ─ align : applies auto-alignments, e.g., billboards Things to keep in mind

| 2004 JavaOne SM Conference | Session TS “Hello, World” import javax.microedition.midlet.MIDlet; import javax.microedition.lcdui.Display; import javax.microedition.lcdui.game.GameCanvas; import javax.microedition.m3g.*; public class Player extends MIDlet { public void pauseApp() {} public void destroyApp(boolean b) {} public void startApp() { PlayerCanvas canvas = new PlayerCanvas(true); Display.getDisplay(this).setCurrent(canvas); try { canvas.run(); } catch (Exception e) {} notifyDestroyed(); } A simplified animation player

| 2004 JavaOne SM Conference | Session TS “Hello, World” class PlayerCanvas extends GameCanvas { PlayerCanvas(boolean suppress){super(suppress);} public void run() throws Exception { Graphics3D g3d = Graphics3D.getInstance(); World w = (World) Loader.load("/file.m3g")[0]; long start, elapsed, time = 0; while (getKeyStates() == 0) { start = System.currentTimeMillis(); g3d.bindTarget(getGraphics()); try { w.animate(time); g3d.render(w); } finally { g3d.releaseTarget(); } flushGraphics(); elapsed = System.currentTimeMillis()-start; time += (elapsed < 100) ? 100 : (int)elapsed; if (elapsed < 100) Thread.sleep(100-elapsed); }

| 2004 JavaOne SM Conference | Session TS Demo Animation playback 21

| 2004 JavaOne SM Conference | Session TS Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

| 2004 JavaOne SM Conference | Session TS Renderable Objects Mesh Sprite3D Made of triangle strips Base class for meshes (two subclasses) 2D image placed in 3D space Good for labels, etc.

| 2004 JavaOne SM Conference | Session TS Mesh VertexBuffercoordinates normals colors texcoords IndexBufferAppearance Composed of submeshes Common buffer of vertex data Submesh has triangle strip indices to vertices One Appearance for each submesh

| 2004 JavaOne SM Conference | Session TS Appearance components CompositingMode Material colors for lighting Can track per-vertex colors PolygonMode Fog Texture2D Material Blending, depth buffering Alpha testing, masking Winding, culling, shading Perspective correction hint Fades colors based on distance Linear and exponential mode Texture matrix, blending, filtering Multitexturing: One Texture2D for each unit

| 2004 JavaOne SM Conference | Session TS Sprite3D Scaled mode for billboards, trees, etc. Unscaled mode for text labels, icons, etc. Image2D 2D image with a position in 3D space Sprite3DAppearance Image2D CompositingMode Fog

| 2004 JavaOne SM Conference | Session TS Rendering tricks Use layers to impose rendering order ─Appearance contains a layer index (integer) ─Defines a global ordering for submeshes and sprites ─Tip: Disable z-buffering for sky boxes, use layers Tip: Maximize triangle strip length ─Even if it requires adding degenerate triangles ─Better benefits from vertex caching

| 2004 JavaOne SM Conference | Session TS Example: Rotating cube // Corners of a cube as (X,Y,Z) triplets static short[] cubeVertices = { -1, -1, 1, 1, -1, 1, -1, 1, 1, 1, 1, 1, -1, -1, -1, 1, -1, -1, -1, 1, -1, 1, 1, -1 }; // A color for each corner as an (R,G,B) triplet static byte[] cubeColors = { (byte) 255, (byte) 0, (byte) 0, // red … (byte) 0, (byte) 255, (byte) 0, // green }; // Define the cube as a single triangle strip static int[] indices = { 6,7,4,5,1,7,3,6,2,4,0,1,2,3 }; static int[] stripLen = { 14 }; Define raw data for a cube

| 2004 JavaOne SM Conference | Session TS Example: Rotating cube // Fill in VertexArrays from short[] and byte[] int numVertices = vertices.length/3; VertexArray pos = new VertexArray(numVertices, 3, 2); VertexArray col = new VertexArray(numVertices, 3, 1); pos.set(0, numVertices, cubeVertices); col.set(0, numVertices, cubeColors); // Attach the VertexArrays to a VertexBuffer // Note the scale (1.0) and bias (0,0,0) vertices = new VertexBuffer(); vertices.setPositions(pos, 1.0f, null); vertices.setColors(col); // Fill in the triangle strip triangles = new TriangleStripArray(cubeIndices, stripLen); // Create a Mesh with default Appearance cube = new Mesh(vertices, triangles, new Appearance()); Copy raw data into Objects

| 2004 JavaOne SM Conference | Session TS Example: Rotating cube Camera cam = new Camera(); // 60-degree field of view, screen aspect ratio // Near clipping plane at 1.0, far plane at 1000 float aspect = (float) getWidth() / (float) getHeight(); cam.setPerspective(60.0f, aspect, 1.0f, f); // Place the camera at z=5.0 in world space // View vector is along the negative Z axis transform = new Transform(); transform.postTranslate(0.0f, 0.0f, 5.0f); g3d = Graphics3D.getInstance(); g3d.setCamera(cam, transform); Set up a Camera

| 2004 JavaOne SM Conference | Session TS Example: Rotating cube g3d = Graphics3D.getInstance(); g3d.bindTarget(getGraphics()); try { g3d.clear(null); g3d.render(cube, transform); } finally { g3d.releaseTarget(); } xAngle += 1; yAngle += 2; zAngle += 3; transform.setIdentity(); transform.postRotate(xAngle, 1.0f, 0.0f, 0.0f); transform.postRotate(yAngle, 0.0f, 1.0f, 0.0f); transform.postRotate(zAngle, 0.0f, 0.0f, 1.0f); Clear the buffers, render, animate

| 2004 JavaOne SM Conference | Session TS Demo Rotating cube 32

| 2004 JavaOne SM Conference | Session TS Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

| 2004 JavaOne SM Conference | Session TS Add bitmap graphics 3D graphics is mostly vector graphics ─Mathematical shape definitions ─Define camera, lights, simulate ─“Computer graphics look”, pretty boring Bitmap graphics adds visual richness ─Background images ─Texture maps on surfaces ─Sprites

| 2004 JavaOne SM Conference | Session TS Texturing tricks Tip: Use light maps for lighting effects ─Usually faster than per-vertex lighting ─Use luminance textures, not RGB ─Multitexturing is your friend Tip: Use the perspective correction hint ─Almost always faster than adding more vertices ─But first check if the implementation supports it!

| 2004 JavaOne SM Conference | Session TS More bitmap tricks Tip: Use background images ─Can be scaled, tiled and scrolled very flexibly ─Generally much faster than sky boxes or similar Tip: Use sprites as impostors, particles ─Generally (much) faster than textured quads ─Unscaled mode is (much) faster than scaled Tip: Load your images with the Loader ─ Loader.load(“/img.png”) outputs Image2D ─Going through MIDP Image is a waste of memory

| 2004 JavaOne SM Conference | Session TS Demo Backgrounds and texturing 37

| 2004 JavaOne SM Conference | Session TS Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

| 2004 JavaOne SM Conference | Session TS The scene graph SkinnedMesh Group Mesh Sprite Light World GroupCamera GroupMorphingMesh Actually, it’s just a tree Not allowed!

| 2004 JavaOne SM Conference | Session TS VertexArray Background Image2D VertexBuffer IndexBuffer Appearance Image2D Texture2D MeshGroup Mesh GroupLightCamera World How to find an object in the scene UserID defaults to 0 UserIDs not necessarily unique find(4) Diagram courtesy of Sean Ellis, Superscape Mesh 4

| 2004 JavaOne SM Conference | Session TS From this node to the parent node ─Ignored for the root node Composed of four parts ─Translation T ─Orientation R ─Non-uniform scale S ─Generic 3x4 matrix M Composite: C = T R S M Node transformations Group Mesh Sprite C CC CC World C

| 2004 JavaOne SM Conference | Session TS Node transformations Tip: Keep the transformations simple ─Favor the T R S components over M ─Avoid non-uniform scales in S Tip: How to scale along an arbitrary axis ─There is no direct support (C = T R N -1 S N M) ─Method 1: R’ = R N -1 and M’ = N M ─Method 2: Use an extra Group node Tip: How to rotate about an arbitrary point ─Again, no direct support (C = T P -1 R P S M) ─Method 1: Use an extra Group node ─Method 2: If S is identity, combine into T and M

| 2004 JavaOne SM Conference | Session TS Example: Set up a hierarchy private Group group1, group2; private Mesh cube1, cube2, cube3; … cube1 = new Mesh(cubeVB, triangles, appearance); cube2 = new Mesh(cubeVB, triangles, appearance); cube3 = new Mesh(cubeVB, triangles, appearance); group1 = new Group(); group2 = new Group(); group1.addChild(cube1); group1.addChild(cube2); group1.addChild(group2); group2.addChild(cube3); group2 group1 cube1 cube3 cube2

| 2004 JavaOne SM Conference | Session TS Animate and render the hierarchy g3d.render(group1, null); … cube1.setOrientation( yAngle, 0.0f, 1.0f, 0.0f ); cube1.setTranslation( 0.0f, 2.0f, 0.0f ); cube2.setOrientation(-yAngle, 0.0f, 1.0f, 0.0f ); cube2.setTranslation( 0.0f,-2.0f, 0.0f ); group2.setOrientation(-yAngle, 0.0f, 1.0f, 0.0f ); group2.setTranslation( -2.0f, 0.0f, 0.0f ); cube3.setOrientation( zAngle, 0.0f, 0.0f, 1.0f ); cube3.setTranslation( -2.0f, 0.0f, 0.0f ); cube3.setScale( 0.5f, 0.5f, 0.5f );

| 2004 JavaOne SM Conference | Session TS Demo Hierarchic transformations 45

| 2004 JavaOne SM Conference | Session TS Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

| 2004 JavaOne SM Conference | Session TS MorphingMesh A base mesh and several morph targets Result = weighted sum of morph deltas Can morph any vertex attribute ─Vertex positions ─Colors ─Normals ─Texture coordinates For vertex morphing animation

| 2004 JavaOne SM Conference | Session TS MorphingMesh BaseTarget 1 eyes closed Target 2 mouth closed Animate eyes and mouth independently Example: Animating a rabbit’s face

| 2004 JavaOne SM Conference | Session TS SkinnedMesh Articulated characters without cracks at joints Stretch a mesh over a hierarchic “skeleton” ─The skeleton consists of scene graph nodes ─Each node (“bone”) defines a transformation ─Each vertex is linked to one or more bones

| 2004 JavaOne SM Conference | Session TS SkinnedMesh Neutral pose, bones at rest Weighted skinning

| 2004 JavaOne SM Conference | Session TS SkinnedMesh Bone B rotated 90 degrees Weighted skinning

| 2004 JavaOne SM Conference | Session TS SkinnedMesh No skinning Local skinning one bone per vertex Smooth skinning two bones per vertex Example: Animating an arm

| 2004 JavaOne SM Conference | Session TS Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

| 2004 JavaOne SM Conference | Session TS Animation: Relevant classes KeyframeSequence AnimationController AnimationTrack A link between sequence, controller and target Object3D Base class for all objects that can be animated Controls the playback of one or more animations Storage for keyframes Defines interpolation mode

| 2004 JavaOne SM Conference | Session TS KeyframeSequence Keyframe is a time and the value of a property at that time Can store any number of keyframes Several keyframe interpolation modes Can be open or closed (looping) sequence time t v KeyframeSequence Diagram courtesy of Sean Ellis, Superscape

| 2004 JavaOne SM Conference | Session TS AnimationController Can control several keyframed animations together Determines relationship between world time and sequence time 0dsequence time world time t 0 t s 0dsequence time 0d AnimationController Diagram courtesy of Sean Ellis, Superscape

| 2004 JavaOne SM Conference | Session TS Animation Relates animation controller, keyframe sequence, and object property together. 0dsequence time Identifies animated property on this object Call to animate(worldTime) s v Calculate sequence time from world time Look up value at this sequence time Apply value to animated property Diagram courtesy of Sean Ellis, Superscape Object3D AnimationTrack AnimationController KeyframeSequence

| 2004 JavaOne SM Conference | Session TS Animation Tip: You can read back the animated values ─Use the animation engine for anything you want ─Much faster than doing interpolation in Java ─Especially in case of quaternions and splines

| 2004 JavaOne SM Conference | Session TS Example: Set up keyframes KeyframeSequence createKeyframeSequence() { // 10 keys, 3 components, spline interpolation KeyframeSequence ks; ks = new KeyframeSequence(10,3,KeyframeSequence.SPLINE); // x grows linearly, y wiggles up-down, z remains 0.0f float[] tb = { 0.0f, 0.0f, 0.0f }; tb[0] = -4.0f; tb[1] = 0.0f; ks.setKeyframe(0, 0, tb); tb[0] = -3.0f; tb[1] = 2.0f; ks.setKeyframe(1, 10, tb); … tb[0] = 5.0f; tb[1] = 0.0f; ks.setKeyframe(9, 90, tb); ks.setDuration(100); // sequence duration in local time ks.setValidRange(0, 9);// all 10 keyframes are valid ks.setRepeatMode(KeyframeSequence.LOOP); return ks; }

| 2004 JavaOne SM Conference | Session TS Example: Set up controller controller = new AnimationController(); // The animation is active between these world times controller.setActiveInterval(0, 1500); // Set speed, scale “around” given time (0) controller.setSpeed(2.0f, 0); // Create animation track with keyframes & controller ks = createKeyframeSequence(); at = new AnimationTrack(ks,AnimationTrack.TRANSLATION); at.setController(controller); // Attach it to our cube cube.addAnimationTrack(at);

| 2004 JavaOne SM Conference | Session TS Demo Keyframe animation 61

| 2004 JavaOne SM Conference | Session TS Summary M3G is the 3D API for J2ME ™ technology ─Will be supported in millions of devices Not just an API ─M3G also defines a binary file format M3G is flexible ─Using the immediate mode gives you full control ─High-level features make your code faster and smaller

| 2004 JavaOne SM Conference | Session TS Demo 3D snowboarding 63

| 2004 JavaOne SM Conference | Session TS For More Information M3G specification SDKs, additional documentation (click “Series 40”) developer.superscape.com/developer.superscape.com/ (registration needed) M3G content creation tools

| 2004 JavaOne SM Conference | Session TS Q&A Thanks: Kari Kangas, Sean Ellis, Nokia M3G team, JSR-184 Expert Group 65

java.sun.com/javaone/sf | 2004 JavaOne SM Conference | Session TS Advanced Game Development with the Mobile 3D Graphics API Tomi Aarnio, Kari Pulli Nokia Research Center