1. java.sun.com/javaone/sf | 2004 JavaOne SM Conference | Session TS-2121 1 Advanced Game Development with the Mobile 3D Graphics API Tomi Aarnio, Kari Pulli Nokia Research Center

2. | 2004 JavaOne SM Conference | Session TS-2121 2 Gain an insight to the Mobile 3D Graphics API and take your games to the next level

3. | 2004 JavaOne SM Conference | Session TS-2121 3 Prerequisites Fundamentals of 3D graphics OpenGL ® or some other modern 3D API Java™ technology, preferably MIDP

4. | 2004 JavaOne SM Conference | Session TS-2121 4 Objectives Get a quick overview of the whole API Gain a deeper understanding of selected topics Learn practical tricks not found in the spec

5. | 2004 JavaOne SM Conference | Session TS-2121 5 Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

6. | 2004 JavaOne SM Conference | Session TS-2121 6 Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

7. | 2004 JavaOne SM Conference | Session TS-2121 7 Mobile 3D Graphics API (M3G) Also known as JSR-184 Designed for mobile devices ─Primarily CLDC / MIDP ─But also CDC Mobile 3D Graphics API Java Virtual Machine (CLDC 1.1) MIDP Midlets Graphics Hardware

8. | 2004 JavaOne SM Conference | Session TS-2121 8 Overcome the performance barrier Benchmarked on an ARM9 processor Native (C/C++) vs. Java on mobiles

9. | 2004 JavaOne SM Conference | Session TS-2121 9 Leverage hardware acceleration Benefits for developers ─Mask hardware differences (3D HW, FPU, DSP,…) ─Exploit any new hardware automatically Benefits for hardware vendors ─Gives a concrete target which features to offer

10. | 2004 JavaOne SM Conference | Session TS-2121 10 Speed up development cycles Separate content from code ─Quick prototyping without programming ─Artists create looks and behavior ─Export all in a binary file format ─Load the file, populate scene graph Commonly needed functionality built in ─High-level functionality such as scene graph, keyframe animation

11. | 2004 JavaOne SM Conference | Session TS-2121 11 Why a new standard? OpenGL (ES) is too low-level ─Lots of Java code needed for simple things Java 3D™ API is too bloated ─A hundred times larger than M3G ─Does not fit together with MIDP ─Tried and failed, but… Now knew what we wanted! ─Basic Java 3D™ ideas: nodes, scene graph ─Add file format, keyframe animation ─Remain compatible with OpenGL ES

12. | 2004 JavaOne SM Conference | Session TS-2121 12 Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

13. | 2004 JavaOne SM Conference | Session TS-2121 13 Key classes World Graphics3D Loader 3D graphics context Performs all rendering Scene graph root node Can load individual objects and entire scene graphs (M3G and PNG files)

14. | 2004 JavaOne SM Conference | Session TS-2121 14 Graphics3D Contains global state ─Target surface, viewport, depth buffer ─Camera, light sources ─Rendering quality hints Each renderable object has its own local state ─Geometry and appearance (material, textures, etc.) ─Transformation relative to parent or world

15. | 2004 JavaOne SM Conference | Session TS-2121 15 Graphics3D: Rendering modes Retained mode ─Render a scene graph, rooted by the World ─Take the Camera and Lights from the World Immediate mode ─Render a branch or an individual node at a time ─Explicitly give the Camera and Lights to Graphics3D

16. | 2004 JavaOne SM Conference | Session TS-2121 16 Graphics3D: How-To-Use Bind a target to it, render, release the target Tip: Never mess with a bound target Tip: Graphics3D is a singleton (threads!) void paint(Graphics g) { myGraphics3D.bindTarget(g); myGraphics3D.render(world); myGraphics3D.releaseTarget(); }

17. | 2004 JavaOne SM Conference | Session TS-2121 17 Graphics3D: Rendering targets GraphicsCanvas Image CustomItem Graphics3DImage2D World Can render to textures also M3G MIDP

18. | 2004 JavaOne SM Conference | Session TS-2121 18 Everything is synchronous ─A method returns only when it’s done ─No separate thread for renderer or loader There are no callbacks ─No abstract methods, interfaces, or events ─Tip: Do not even try to override any methods Scene update is decoupled from rendering ─ render : draws the scene, no side-effects ─ animate : updates the scene to the given time ─ align : applies auto-alignments, e.g., billboards Things to keep in mind

19. | 2004 JavaOne SM Conference | Session TS-2121 19 “Hello, World” import javax.microedition.midlet.MIDlet; import javax.microedition.lcdui.Display; import javax.microedition.lcdui.game.GameCanvas; import javax.microedition.m3g.*; public class Player extends MIDlet { public void pauseApp() {} public void destroyApp(boolean b) {} public void startApp() { PlayerCanvas canvas = new PlayerCanvas(true); Display.getDisplay(this).setCurrent(canvas); try { canvas.run(); } catch (Exception e) {} notifyDestroyed(); } A simplified animation player

20. | 2004 JavaOne SM Conference | Session TS-2121 20 “Hello, World” class PlayerCanvas extends GameCanvas { PlayerCanvas(boolean suppress){super(suppress);} public void run() throws Exception { Graphics3D g3d = Graphics3D.getInstance(); World w = (World) Loader.load("/file.m3g")[0]; long start, elapsed, time = 0; while (getKeyStates() == 0) { start = System.currentTimeMillis(); g3d.bindTarget(getGraphics()); try { w.animate(time); g3d.render(w); } finally { g3d.releaseTarget(); } flushGraphics(); elapsed = System.currentTimeMillis()-start; time += (elapsed < 100) ? 100 : (int)elapsed; if (elapsed < 100) Thread.sleep(100-elapsed); }

21. | 2004 JavaOne SM Conference | Session TS-2121 21 Demo Animation playback 21

22. | 2004 JavaOne SM Conference | Session TS-2121 22 Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

23. | 2004 JavaOne SM Conference | Session TS-2121 23 Renderable Objects Mesh Sprite3D Made of triangle strips Base class for meshes (two subclasses) 2D image placed in 3D space Good for labels, etc.

24. | 2004 JavaOne SM Conference | Session TS-2121 24 Mesh VertexBuffercoordinates normals colors texcoords IndexBufferAppearance Composed of submeshes Common buffer of vertex data Submesh has triangle strip indices to vertices One Appearance for each submesh

25. | 2004 JavaOne SM Conference | Session TS-2121 25 Appearance components CompositingMode Material colors for lighting Can track per-vertex colors PolygonMode Fog Texture2D Material Blending, depth buffering Alpha testing, masking Winding, culling, shading Perspective correction hint Fades colors based on distance Linear and exponential mode Texture matrix, blending, filtering Multitexturing: One Texture2D for each unit

26. | 2004 JavaOne SM Conference | Session TS-2121 26 Sprite3D Scaled mode for billboards, trees, etc. Unscaled mode for text labels, icons, etc. Image2D 2D image with a position in 3D space Sprite3DAppearance Image2D CompositingMode Fog

27. | 2004 JavaOne SM Conference | Session TS-2121 27 Rendering tricks Use layers to impose rendering order ─Appearance contains a layer index (integer) ─Defines a global ordering for submeshes and sprites ─Tip: Disable z-buffering for sky boxes, use layers Tip: Maximize triangle strip length ─Even if it requires adding degenerate triangles ─Better benefits from vertex caching

28. | 2004 JavaOne SM Conference | Session TS-2121 28 Example: Rotating cube // Corners of a cube as (X,Y,Z) triplets static short[] cubeVertices = { -1, -1, 1, 1, -1, 1, -1, 1, 1, 1, 1, 1, -1, -1, -1, 1, -1, -1, -1, 1, -1, 1, 1, -1 }; // A color for each corner as an (R,G,B) triplet static byte[] cubeColors = { (byte) 255, (byte) 0, (byte) 0, // red … (byte) 0, (byte) 255, (byte) 0, // green }; // Define the cube as a single triangle strip static int[] indices = { 6,7,4,5,1,7,3,6,2,4,0,1,2,3 }; static int[] stripLen = { 14 }; Define raw data for a cube

29. | 2004 JavaOne SM Conference | Session TS-2121 29 Example: Rotating cube // Fill in VertexArrays from short[] and byte[] int numVertices = vertices.length/3; VertexArray pos = new VertexArray(numVertices, 3, 2); VertexArray col = new VertexArray(numVertices, 3, 1); pos.set(0, numVertices, cubeVertices); col.set(0, numVertices, cubeColors); // Attach the VertexArrays to a VertexBuffer // Note the scale (1.0) and bias (0,0,0) vertices = new VertexBuffer(); vertices.setPositions(pos, 1.0f, null); vertices.setColors(col); // Fill in the triangle strip triangles = new TriangleStripArray(cubeIndices, stripLen); // Create a Mesh with default Appearance cube = new Mesh(vertices, triangles, new Appearance()); Copy raw data into Objects

30. | 2004 JavaOne SM Conference | Session TS-2121 30 Example: Rotating cube Camera cam = new Camera(); // 60-degree field of view, screen aspect ratio // Near clipping plane at 1.0, far plane at 1000 float aspect = (float) getWidth() / (float) getHeight(); cam.setPerspective(60.0f, aspect, 1.0f, 1000.0f); // Place the camera at z=5.0 in world space // View vector is along the negative Z axis transform = new Transform(); transform.postTranslate(0.0f, 0.0f, 5.0f); g3d = Graphics3D.getInstance(); g3d.setCamera(cam, transform); Set up a Camera

31. | 2004 JavaOne SM Conference | Session TS-2121 31 Example: Rotating cube g3d = Graphics3D.getInstance(); g3d.bindTarget(getGraphics()); try { g3d.clear(null); g3d.render(cube, transform); } finally { g3d.releaseTarget(); } xAngle += 1; yAngle += 2; zAngle += 3; transform.setIdentity(); transform.postRotate(xAngle, 1.0f, 0.0f, 0.0f); transform.postRotate(yAngle, 0.0f, 1.0f, 0.0f); transform.postRotate(zAngle, 0.0f, 0.0f, 1.0f); Clear the buffers, render, animate

32. | 2004 JavaOne SM Conference | Session TS-2121 32 Demo Rotating cube 32

33. | 2004 JavaOne SM Conference | Session TS-2121 33 Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

34. | 2004 JavaOne SM Conference | Session TS-2121 34 Add bitmap graphics 3D graphics is mostly vector graphics ─Mathematical shape definitions ─Define camera, lights, simulate ─“Computer graphics look”, pretty boring Bitmap graphics adds visual richness ─Background images ─Texture maps on surfaces ─Sprites

35. | 2004 JavaOne SM Conference | Session TS-2121 35 Texturing tricks Tip: Use light maps for lighting effects ─Usually faster than per-vertex lighting ─Use luminance textures, not RGB ─Multitexturing is your friend Tip: Use the perspective correction hint ─Almost always faster than adding more vertices ─But first check if the implementation supports it!

36. | 2004 JavaOne SM Conference | Session TS-2121 36 More bitmap tricks Tip: Use background images ─Can be scaled, tiled and scrolled very flexibly ─Generally much faster than sky boxes or similar Tip: Use sprites as impostors, particles ─Generally (much) faster than textured quads ─Unscaled mode is (much) faster than scaled Tip: Load your images with the Loader ─ Loader.load(“/img.png”) outputs Image2D ─Going through MIDP Image is a waste of memory

37. | 2004 JavaOne SM Conference | Session TS-2121 37 Demo Backgrounds and texturing 37

38. | 2004 JavaOne SM Conference | Session TS-2121 38 Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

39. | 2004 JavaOne SM Conference | Session TS-2121 39 The scene graph SkinnedMesh Group Mesh Sprite Light World GroupCamera GroupMorphingMesh Actually, it’s just a tree Not allowed!

40. | 2004 JavaOne SM Conference | Session TS-2121 40 VertexArray Background Image2D VertexBuffer IndexBuffer Appearance Image2D Texture2D MeshGroup Mesh GroupLightCamera World How to find an object in the scene 0 1 2 34 4 56 1 0 6 78 9 10 11 UserID defaults to 0 UserIDs not necessarily unique find(4) Diagram courtesy of Sean Ellis, Superscape Mesh 4

41. | 2004 JavaOne SM Conference | Session TS-2121 41 From this node to the parent node ─Ignored for the root node Composed of four parts ─Translation T ─Orientation R ─Non-uniform scale S ─Generic 3x4 matrix M Composite: C = T R S M Node transformations Group Mesh Sprite C CC CC World C

42. | 2004 JavaOne SM Conference | Session TS-2121 42 Node transformations Tip: Keep the transformations simple ─Favor the T R S components over M ─Avoid non-uniform scales in S Tip: How to scale along an arbitrary axis ─There is no direct support (C = T R N -1 S N M) ─Method 1: R’ = R N -1 and M’ = N M ─Method 2: Use an extra Group node Tip: How to rotate about an arbitrary point ─Again, no direct support (C = T P -1 R P S M) ─Method 1: Use an extra Group node ─Method 2: If S is identity, combine into T and M

43. | 2004 JavaOne SM Conference | Session TS-2121 43 Example: Set up a hierarchy private Group group1, group2; private Mesh cube1, cube2, cube3; … cube1 = new Mesh(cubeVB, triangles, appearance); cube2 = new Mesh(cubeVB, triangles, appearance); cube3 = new Mesh(cubeVB, triangles, appearance); group1 = new Group(); group2 = new Group(); group1.addChild(cube1); group1.addChild(cube2); group1.addChild(group2); group2.addChild(cube3); group2 group1 cube1 cube3 cube2

44. | 2004 JavaOne SM Conference | Session TS-2121 44 Animate and render the hierarchy g3d.render(group1, null); … cube1.setOrientation( yAngle, 0.0f, 1.0f, 0.0f ); cube1.setTranslation( 0.0f, 2.0f, 0.0f ); cube2.setOrientation(-yAngle, 0.0f, 1.0f, 0.0f ); cube2.setTranslation( 0.0f,-2.0f, 0.0f ); group2.setOrientation(-yAngle, 0.0f, 1.0f, 0.0f ); group2.setTranslation( -2.0f, 0.0f, 0.0f ); cube3.setOrientation( zAngle, 0.0f, 0.0f, 1.0f ); cube3.setTranslation( -2.0f, 0.0f, 0.0f ); cube3.setScale( 0.5f, 0.5f, 0.5f );

45. | 2004 JavaOne SM Conference | Session TS-2121 45 Demo Hierarchic transformations 45

46. | 2004 JavaOne SM Conference | Session TS-2121 46 Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

47. | 2004 JavaOne SM Conference | Session TS-2121 47 MorphingMesh A base mesh and several morph targets Result = weighted sum of morph deltas Can morph any vertex attribute ─Vertex positions ─Colors ─Normals ─Texture coordinates For vertex morphing animation

48. | 2004 JavaOne SM Conference | Session TS-2121 48 MorphingMesh BaseTarget 1 eyes closed Target 2 mouth closed Animate eyes and mouth independently Example: Animating a rabbit’s face

49. | 2004 JavaOne SM Conference | Session TS-2121 49 SkinnedMesh Articulated characters without cracks at joints Stretch a mesh over a hierarchic “skeleton” ─The skeleton consists of scene graph nodes ─Each node (“bone”) defines a transformation ─Each vertex is linked to one or more bones

50. | 2004 JavaOne SM Conference | Session TS-2121 50 SkinnedMesh Neutral pose, bones at rest Weighted skinning

51. | 2004 JavaOne SM Conference | Session TS-2121 51 SkinnedMesh Bone B rotated 90 degrees Weighted skinning

52. | 2004 JavaOne SM Conference | Session TS-2121 52 SkinnedMesh No skinning Local skinning one bone per vertex Smooth skinning two bones per vertex Example: Animating an arm

53. | 2004 JavaOne SM Conference | Session TS-2121 53 Agenda Why should I care? Getting started Low level features Bitmaps and texturing Scene graph Dynamic meshes Animation

54. | 2004 JavaOne SM Conference | Session TS-2121 54 Animation: Relevant classes KeyframeSequence AnimationController AnimationTrack A link between sequence, controller and target Object3D Base class for all objects that can be animated Controls the playback of one or more animations Storage for keyframes Defines interpolation mode

55. | 2004 JavaOne SM Conference | Session TS-2121 55 KeyframeSequence Keyframe is a time and the value of a property at that time Can store any number of keyframes Several keyframe interpolation modes Can be open or closed (looping) sequence time t v KeyframeSequence Diagram courtesy of Sean Ellis, Superscape

56. | 2004 JavaOne SM Conference | Session TS-2121 56 AnimationController Can control several keyframed animations together Determines relationship between world time and sequence time 0dsequence time world time t 0 t s 0dsequence time 0d AnimationController Diagram courtesy of Sean Ellis, Superscape

57. | 2004 JavaOne SM Conference | Session TS-2121 57 Animation Relates animation controller, keyframe sequence, and object property together. 0dsequence time Identifies animated property on this object Call to animate(worldTime) s v Calculate sequence time from world time Look up value at this sequence time Apply value to animated property Diagram courtesy of Sean Ellis, Superscape Object3D AnimationTrack AnimationController KeyframeSequence

58. | 2004 JavaOne SM Conference | Session TS-2121 58 Animation Tip: You can read back the animated values ─Use the animation engine for anything you want ─Much faster than doing interpolation in Java ─Especially in case of quaternions and splines

59. | 2004 JavaOne SM Conference | Session TS-2121 59 Example: Set up keyframes KeyframeSequence createKeyframeSequence() { // 10 keys, 3 components, spline interpolation KeyframeSequence ks; ks = new KeyframeSequence(10,3,KeyframeSequence.SPLINE); // x grows linearly, y wiggles up-down, z remains 0.0f float[] tb = { 0.0f, 0.0f, 0.0f }; tb[0] = -4.0f; tb[1] = 0.0f; ks.setKeyframe(0, 0, tb); tb[0] = -3.0f; tb[1] = 2.0f; ks.setKeyframe(1, 10, tb); … tb[0] = 5.0f; tb[1] = 0.0f; ks.setKeyframe(9, 90, tb); ks.setDuration(100); // sequence duration in local time ks.setValidRange(0, 9);// all 10 keyframes are valid ks.setRepeatMode(KeyframeSequence.LOOP); return ks; }

60. | 2004 JavaOne SM Conference | Session TS-2121 60 Example: Set up controller controller = new AnimationController(); // The animation is active between these world times controller.setActiveInterval(0, 1500); // Set speed, scale “around” given time (0) controller.setSpeed(2.0f, 0); // Create animation track with keyframes & controller ks = createKeyframeSequence(); at = new AnimationTrack(ks,AnimationTrack.TRANSLATION); at.setController(controller); // Attach it to our cube cube.addAnimationTrack(at);

61. | 2004 JavaOne SM Conference | Session TS-2121 61 Demo Keyframe animation 61

62. | 2004 JavaOne SM Conference | Session TS-2121 62 Summary M3G is the 3D API for J2ME ™ technology ─Will be supported in millions of devices Not just an API ─M3G also defines a binary file format M3G is flexible ─Using the immediate mode gives you full control ─High-level features make your code faster and smaller

63. | 2004 JavaOne SM Conference | Session TS-2121 63 Demo 3D snowboarding 63

64. | 2004 JavaOne SM Conference | Session TS-2121 64 For More Information M3G specification www.forum.nokia.com/java/jsr184 SDKs, additional documentation www.forum.nokia.com/www.forum.nokia.com/ (click “Series 40”) developer.superscape.com/developer.superscape.com/ (registration needed) M3G content creation tools www.discreet.com/ www.superscape.com/

65. | 2004 JavaOne SM Conference | Session TS-2121 65 Q&A Thanks: Kari Kangas, Sean Ellis, Nokia M3G team, JSR-184 Expert Group 65

66. java.sun.com/javaone/sf | 2004 JavaOne SM Conference | Session TS-2121 66 Advanced Game Development with the Mobile 3D Graphics API Tomi Aarnio, Kari Pulli Nokia Research Center