Trends and Standards for 3D Graphics for Handsets Christophe Quarre’ Advanced System Technology, Shanghai Lab STMicroelectronics State Key Lab of CAD&CG, Zhejiang University 1-3 March 2005

2 Outline  Handhelds VS Desktops.... new industry  Challenges for ISVs  Standards for graphics mobile.... and 3D acceleration  The key players... ISVs & IHVs  Conclusion

3 Mobile 3D industry  Mobile phones are about to be transformed in personal compute devices Graphics capabilities will be a key ingredient – just as on the PC  Before we had 2D, but now we want 3D !  The Mobile 3D industry is embryonic - but moving fast! We are where PC graphics were in 1996 But evolving about three times faster!

4 Game market in Korea [USD million] Surging Video and mobile game market

5 Desktop vs. Handheld Systems DesktopHandheld CPUINTEL 1GHz up to >3GHz ARM as de-facto standard 100MHz to 500+MHz FPUYES Today ARM9 (Nomadik ST)  NO Tomorrow ARM11(Nomadik ST)  YES CACH, MEMORY, BANDWIDTH Very big, power hungry ~10s of KB of eSRAM ~1 MB of eDRAM is now feasible ~100s of MB DDR-SDRAM Ultra low power design library RESOLUTIONSVGA >1024 * 768 pixels QVGA (320*240) VGA (640*480) GPU (e.g.) Nvidia: GeForce 6800, ATI: Radeon X800 IMG, Falanx, ATI, nVidia, Bitboys and others 3D APIOpenGL1.5 / 2.0 DirectX Java3D OpenGL ES 1.0/1.1and future 2.0 D3DM JSR184 M3G

6 Challenges on Handheld Platforms  Small battery, small CPU  Management of Host CPU  unload max work on Host CPU  and hardware resources  can save 90% of battery life !  Limited system memory size and bandwidth  Memory architecture will make the difference !!  Cache memory; buffers compression  A wide range of Mobiles and gfx architectures, from Low-End  High-End  Means scalable and flexible solutions !!!

7 Challenges for Software Vendors 1/2  If we consider 3 likely classes of handheld device Class A : High-performance CPU, GPU with vertex processing + FPU Class B : High-performance CPU, GPU with vertex processing Class C : CPU, rasterizer Classes B and C, low cost will likely ship in higher volume,  therefore may offer more revenue opportunities for software vendors  but yet platforms do not have floating-point support (FPU) Class A device may win out for better capacity and performance  Software vendors must cover all the bases to guarantee success Observation

8 Challenges for Software Vendors2/2  How to develop or port rich and attractive applications with such a spread of computational capabilities?  System Software must be scalable with Unified driver architecture to rapidly be adapted to different system architectures  Middleware is a key component E.g. RenderWare (Criterion) optimized for Nokia N-gage Action

9 Standardization  Different class of mobiles and Handsets, different capabilities  Many Graphics accelerator Suppliers  Graphics low power for mobile is a new industry  Need for standardizations

10 An Important and leading Group  Member-funded industry consortium  Lead specification of open standard, multi-platform, royalty-free low level APIs for HW acceleration  Leverage graphics technology by working closely with ISVs IHVs 2002 Embedded 3D Graphics Acceleration 2004 Embedded 2D Vector Graphics Acceleration 2000 Streaming Media Control and Video Acceleration 2004 Embedded Media Primitive Acceleration

11 Khronos Members Promoting Members Contributing Members

12 OpenML Standard for Dynamic media authoring  royalty-free, cross-platform programming environment  Abstraction layer for capturing, transporting, processing, displaying, and synchronizing digital media :– audio/video streams 2D/3D graphics streams  OpenGL extensions for accelerated video processing  Lowest level of unified functionality, thinnest layer on top of HW Application Windows system OpenGLOpenGL Extension MLdcML Synchronisation

13 OpenVG 1/2 Standard for Vector Graphics acceleration  Why OpenVG ? 2D vector graphics is necessary for high quality GUIs or Font display on small display Need hardware acceleration for useful performance Existing vector graphics APIs are proprietary and developers must rewrite code for each platform Existing APIs usually designed for software rendering  Vector Graphics Exemple Original image SVG zoom Bitmap zoom

14 OpenVG2/2 Standard for Vector Graphics acceleration  OpenVG: Royalty-free, cross-platform, low-level APIs for hardware acceleration for high quality 2D vector graphics.  Interface for vector graphics libraries such as Flash and SVG  Same framework as OpenGL|ES  combine with 3D APIs (Blending, lighting, texturing…)  OpenVG1.0 specs is currently under review by selected ISVs. Official release expected in March 2005 !

15 OpenMAX Standard for Media acceleration primitive  OpenMAX: royalty-free, cross-platform, common APIs for multimedia application  standardizes access to media processing primitives used extensively in graphics, audio and image libraries and video codecs (eg. MPEG-4)  Benefit: Accelerate porting of multimedia SW  Currently in development, expected release in End 2005 Video MPEG-4 H.264 Etc. Voice G.7xx Etc Audio MP3 AAC etc. Image JPEG JPEG2k etc Multimedia API’s RISC DSP SMP H/W accel Multimedia Accelerators  The problem PORTING PROBLEM!  slow down multimedia sw development  POORLY optimized

16 OpenMAX Standard for Media acceleration primitive  Accelerating a broad range of media types 3D Small footprint 3D for embedded systems Video Dynamic Media Authoring Vector 2D Low-level hardware acceleration API Media Engines – CPUs, DSP, Hardware Accelerators etc. Accelerated media primitives for cross-platform acceleration of media libraries Image libraries Video Codecs Sound Libraries Applications Three media libraries called by applications. Can be ported directly to silicon.

17 OpenGL ES Standard for Embedded 3D graphics acceleration Khronos has created a small-footprint subset of OpenGL  Created with the blessing and cooperation of the OpenGL ARB Full functionality for 3D games  On a wide variety of platforms – including fixed point processors Eliminate Redundancy Eliminate Workstation Functionality ARB Feedback and Ratification Embedded Focus Workstation Focus Small footprint e.g. 50KB software engines

18 OpenGL ES Roadmap Update OpenGL ES every year by default  To expose rapidly developing handheld platforms capabilities BUT ONLY introduce features with proven demand from ISVs or IHVs  Guarding against unnecessary pieces of HW Track and adapt developments in desktop OpenGL OpenGL ES 2.0 OpenGL 2.0 OpenGL ES 1.1 OpenGL ES 1.0 OpenGL 1.3 OpenGL 1.5 Enabling software AND hardware implementations – including small-footprint, low-end fixed point platforms Shader programmability for embedded devices Widely available cross-platform 3D graphics API Full high-level shading language capability to harness the power of programmable hardware Mid-03Mid-04Mid-05 Increasing emphasis on enabling emerging hardware with video support and enhanced 3D pipeline

19 Java The multiplatform language  Game developers prefer Java, better for creating and managing tools and high level gfx libraries  But still need low level access to the 3D HW  Binding to OpenGL and OpenGL|ES JSR 231 – OpenGL (Expert Group created in Oct 2003) JSR 239 – OpenGL ES (Expert Group created in March 2004)  Wrappers around native OpenGL ES implementation  Higher level, Scene Graph API JSR 184 – Mobile 3D Graphics API for J2ME TM (Released in Nov 03)  can be built on a native OpenGL ES implementation  can be layered on JSR 239  Reduce application size for OTA downloading

20 SW Standards layers for 3D Mobile graphics C/C++ Applications Scenegraph APIs M3G (JSR 184) Middleware Libraries Games Engines Java Applications Hardware OpenGL ES Engines Software OpenGL ES Engines Operating Systems Low-level 3D Graphics API High-level Graphics Libraries Applications J2ME Hardware

21 Graphics SW (Engine, Middleware…) Providers  Games developers  Game Engine, Middleware, Dev. tools  OpenGL|ES implementation and driver, Authoring tools...  JSR184/239 Mascot Capsule X-FORGE

22 Graphics HW acceleration Providers AcceleonG30/G40™ Acceleon G30/G40™ PowerVR MBX / MBX Lite GoForce 4000/3000 IMAGEON 2300 And of course !! Mali50/100

23 Mobile Multimedia chips Providers ARM9 (200MHz), VFP, MBX ARM11 (330MHz), TMS310 (220MHz), VFP, MBX ARM11 (500MHz), MMDSP(ST), VFP, 3D Acc (ST) Hitachi semiconductor Mitsubishi semiconductor SH-CPU (600MHz), SH-DSP, VFP, MBX

24 Conclusion  3D graphics for mobile is a new industry,  for a new market...  To be competitive, SW systems need to be scalable, with  unified low level standards,  unified drivers...  Middleware is the key for success,  to provide efficient data conditioning...