1. © Fraunhofer IGD Introduction and Tutorial X3DOM – Declarative (X)3D in HTML5 Yvonne Jung, Johannes Behr Fraunhofer IGD / VCST Darmstadt, Germany

2. © Fraunhofer IGD 3D Information inside the Web Websites (have) become Web applications Increasing interest in 3D for Product presentation Visualization of abstract information Experiencing Cultural Heritage data etc. Supporting decision making, e.g. in Virtual Engineering Enhancing user experience with more sophisticated visualizations Yesterday: Flash-based site with videos Today: Immersive 3D inside Browsers

3. © Fraunhofer IGD Virtual Engineering and Cultural Heritage on the Web Titel, Ort, Datum - Vorname Name3 PPT_Master_IGD_v2009.200.ppt

4. © Fraunhofer IGD Technology Trends 3D Data and Documents 3D Printer 3D Scanner Geo Data 3D PLM Web-based Engineering Cloud-based Rendering Augmented/ Mixed Reality Convergence of Application Platforms HTML5 Titanium Peper NaCI Windows Store apps W3C WebApps Chrome OS PhoneGap Divergence of System Platforms Divergence of System Platforms Touch Table Android BlackBerry Linux Cloud Bases WindowsPhone Mac OS Amazon Kindle Windows iOS Web Service Architectures

5. © Fraunhofer IGD OpenGL + GLSL on the Web: WebGL JavaScript Binding for OpenGL ES 2.0 in Web Browser  Firefox, Chrome, Safari, Opera Only GLSL shader based, no fixed function pipeline No variables from GL state No Matrix stack, etc. HTML5 element provides 3D rendering context gl = canvas.getContext(’webgl’); API calls via GL object X3D via X3DOM framework http://www.x3dom.org

6. © Fraunhofer IGD X3DOM – Declarative (X)3D in HTML5 X3DOM := X3D + DOM DOM-based integration framework for declarative 3D graphics in HTML5 Seamless integration of 3D contents in Web Browser

7. © Fraunhofer IGD X3DOM – Declarative (X)3D in HTML5 Brings together both, declarative content design as known from web design “old-school” imperative approaches known from game engine development Allows utilizing well-known JavaScript and DOM infrastructure for 3D Embed a live scene-graph in the DOM Hello X3DOM World

8. © Fraunhofer IGD X3DOM – Declarative (X)3D in HTML5 Completes todays graphics technologies Declarative Scene-graph Part of HTML document DOM Integration CSS / Events Imperative Procedural API Drawing context Flexible 2D (Final HTML5 spec) 3D (No W3C spec yet)

9. © Fraunhofer IGD Benefits: Why Declarative 3D in HTML? Native Web Browser integration Plug-in/ App free No issues with user permissions, installation, and security OS independent, especially on mobile devices Cluttered: Windows Phone, Android, iOS, Symbian… Web Browsers for most devices available Browser already provides complete deployment structure Eases proliferation of technology and accessibility of content No special APIs (such as in game engines) No expert knowledge required (OpenGL, mathematics, …) Integrates with standard Web techniques (e.g. DHTML, Ajax) Rapid application development

10. © Fraunhofer IGD Benefits: Why Declarative 3D in HTML? Declarative, open, human-readable (wraps low-level graphics) Utilizing standard Web APIs for integrating content and user interactions Open architectures (also for authoring) and ease of access Integration into HTML document instead of closed systems Metadata: index and search “content” on WebGL apps? Allows “mash-ups” (i.e. recombination of existing contents) Open formats enable automated connection of existing data (e.g., geo-information, Flickr) with 3D content Unify 2D and 3D media development Declarative content description Flexible content (cultural heritage, industry,…) Interoperability: Write once, run anywhere (web/ desktop/ mobile)

11. © Fraunhofer IGD Excursus: Web-based APIs and DOM  Browser provides complete deployment structure  DOM (Document Object Model) is standardized interface that allows manipulating content, structure and style of (X)HTML/ XML documents  Document is structured as tree with nodes  document.getElementById(„myID“);  Nodes/ tags and attributes can be added, removed and modified (usually with JavaScript)  document.createElement(), appendChild(), removeChild()  setAttribute(), getAttribute()  UI events (e.g. ‘mouseover’) can be attached to most elements (e.g.,,, etc.)  Separation of style and content via CSS DOM structure (example)

12. © Fraunhofer IGD Short introduction of HTML My 3D page Hello X3DOM World A blue box will soon appear.

13. © Fraunhofer IGD First HTML needs to know about (X)3D My 3D page <link rel="stylesheet" type="text/css" href="http://www.x3dom.org/x3dom/release/x3dom.css"> <script type="text/javascript" src="http://www.x3dom.org/x3dom/release/x3dom.js"> …

14. © Fraunhofer IGD 3D only works inside the tag … Hello X3DOM World A blue box will soon appear.

15. © Fraunhofer IGD All 3D objects are children of the element … Hello X3DOM World

16. © Fraunhofer IGD Every object has a … Hello X3DOM World

17. © Fraunhofer IGD …and a geometry, like e.g. a … Hello X3DOM World

18. © Fraunhofer IGD …and an

19. © Fraunhofer IGD …with a (e.g. red)

20. © Fraunhofer IGD Materials with specular highlights <material diffuseColor="red" specularColor="#808080">

21. © Fraunhofer IGD Change Background Colors in (R,G,B) with red/green/blue  [0,1] <material diffuseColor="red" specularColor="#808080">

22. © Fraunhofer IGD Change Background (now using CSS) … Change size of element to full size …

23. © Fraunhofer IGD Geometric base objects See screenshot – from left to right:

24. © Fraunhofer IGD Defining own geometries Example: simple rectangle with an

25. © Fraunhofer IGD Defining own geometries Example: simple rectangle with an Important building blocks The vertices of a Polygon (here “face”), given as The index to a vertex, given as list: “coordIndex” x y 1 1 5 5 (2, 2, 0)(7, 2, 0) (2, 5, 0)(7, 5, 0) z 1

26. © Fraunhofer IGD Defining own geometries Example: simple rectangle with an The end of one polygon and the begin of a new one is marked as “-1” in the index array This way arbitrarily complex 3D objects can be created x y 1 1 5 5 (2, 2, 0)(7, 2, 0) (2, 5, 0)(7, 5, 0) z 1

27. © Fraunhofer IGD Defining own geometries Example: simple rectangle with an The indices (except “-1”) refer to the array position of a 3D coordinate in The coordinates of a certain polygon are listed counterclockwise x y 1 1 5 5 (2, 2, 0)(7, 2, 0) (2, 5, 0)(7, 5, 0) z 1 01 32

28. © Fraunhofer IGD DOM holds structure and data More than 95% are usually unstructured data

29. © Fraunhofer IGD  Real 3D applications lead to huge HTML files  DOM holds structure and data  More than 95% are usually unstructured data  Unpleasant non-interactive user experience  Browsers not built to hold Gigabytes of DOM attribute data  Possible solutions  Reference external sub trees (e.g. X3D Inline)  Binary XML decompression (e.g. X3DB format)  Separate structure from raw data  Khronos: 3D Transmission Format  http://www.perey.com/ARStandards/Notes_during_ Transmission_Format_Requirements_Session_Nov_9_2012.pdf 29 Declarative (X)3D in HTML Problem: Large Datasets

30. © Fraunhofer IGD 30 Declarative (X)3D in HTML Problem: Large Datasets  Separation of structure and data  Structure in DOM, attributes in binary containers  J. Behr, Y. Jung, T. Franke, T. Sturm: Using images and explicit binary container for efficient and incremental delivery of declarative 3D scenes on the Web. Web3D 2012  Batching draw calls by merging geometries  Subdivision into meshes with max. 2 16 vertices  Compact vertex data encoding  Improve transmission (over Web and CPU  GPU)  No decoding, just transfer as-is to GPU memory  (Client-/server-based approaches)

31. © Fraunhofer IGD New Geometry node types Data transcoding (example with input file „model.ply“) Without mesh restructuring aopt -i model.ply -G binGeo/:sac -N model.html With mesh/graph optimization (cleanup, patching, and binary creation) aopt -i model.ply -F Scene:"cacheopt(true)” -G binGeo/:sac -N model.html <binaryGeometry vertexCount='1153083' primType='"TRIANGLES"' position='19.811892 -57.892578 -1.699294' size='92.804482 159.783081 26.479685' coord='binGeo/BG0_interleaveBinary.bin#0+24' coordType='Int16' normal='binGeo/BG0_interleaveBinary.bin#8+24' normalType='Int16‘ color='binGeo/BG0_interleaveBinary.bin#16+24‘ colorType='Int16' >

32. © Fraunhofer IGD Vertex Data Encoding  Compact encoding of vertex attribute data  Using 16 bit (pos, texCoord) or 8 (normal, color)  Required number of bits for pixel-level accuracy for viewport of size w × h, with N := max(w, h), is:  No expensive CPU-based decoding required  Optionally normals as spherical coords in pos.w  Simplified conversion to Cartesian coords for mobile shaders  aopt -i model.x3d -F Scene:”maxtris(20000)” -G binGeo/:sacp -N model.html  Ensure 32 bit alignment for rendering speed Titel, Ort, Datum - Vorname Name32 PPT_Master_IGD_v2009.200.ppt

33. © Fraunhofer IGD Vertex Data Encoding

34. © Fraunhofer IGD Light sources in X3DOM …are part of the Directional lightPoint lightSpot light

35. © Fraunhofer IGD Other rendering effects Note: like only as child node of possible! fogtexturesshadows

36. © Fraunhofer IGD Parameterizing shadows  Implemented as post-process, thus transparent to user-defined shaders  Additional fields in all light nodes:  shadowIntensity (default 0), shadowMapSize (default 1024), shadowFilterSize (default 0), shadowOffset (default 0), and – very rarely needed: zNear (default -1), zFar (default -1)  Fields in Directional- and SpotLight:  shadowCascades (default 1), shadowSplitOffset (default 0.1), shadowSplitFactor (default 1) – 0 equidistant splitting, 1 logarithmic Titel, Ort, Datum - Vorname Name36 PPT_Master_IGD_v2009.200.ppt

37. © Fraunhofer IGD Appearance example: a textured box Interesting alternative – using a video as texture:

38. © Fraunhofer IGD Excursus: the lighting model (diffuse and specular reflection) Final color I := ambient material + diffuse material * (N·L) + specular material * (N·H) s For more light sources:

39. © Fraunhofer IGD Two objects in one scene (?!) … OK ???

40. © Fraunhofer IGD Two objects in one scene Problem: both appear at same position

41. © Fraunhofer IGD Two objects in one scene Problem: both appear at same position Reason: 3D objects are usually created in coordinate origin and need to be repositioned afterwards x y 1 1 3 z 1

42. © Fraunhofer IGD Excursus: (2D) coordinate systems Object coordinates in image plane (given by x & y) x y 1 1 5 5 (2, 2)(7, 2) (2, 5)(7, 5)

43. © Fraunhofer IGD Excursus: (3D) coordinate systems Object coordinates in 3D space (z orthogonal on x & y) x y 1 1 5 5 (2, 2, 0)(7, 2, 0) (2, 5, 0)(7, 5, 0) z 1

44. © Fraunhofer IGD Two objects in one scene Now with translation

45. © Fraunhofer IGD Two objects in one scene Now with translation x y 1 1 3 z 1

46. © Fraunhofer IGD The scene graph: Grouping and transformations 3D elements are usually organized hierarchically Starting from the root node (i.e. from element) all 3D elements (e.g., etc.) are inserted into the “tree” (scene graph) as child or sibling elements Note: tree ≠ graph and elements help to group and reposition objects: …

47. © Fraunhofer IGD DOM Manipulation: Node appending / removal HTML/X3D code: … JS script to add nodes: root = document.getElementById(‘root’); trans = document.createElement('Transform'); trans.setAttribute(‘translation’, ‘1 2 3’ ); root.appendChild(trans); JS script to remove nodes: root.removeChild(trans); JS script with setAttribute() (also useful for libs like jQuery): document.getElementById(‘mat’).setAttribute(‘diffuseColor’,’red’);

48. © Fraunhofer IGD How do I know, which elements can be combined? Titel, Ort, Datum - Vorname Name48 PPT_Master_IGD_v2009.200.ppt

49. © Fraunhofer IGD How do I know, which elements can be combined?  Back to the roots: VRML Transform { translation 4 2 1 children [ Shape { appearance Appearance { material Material { diffuseColor 1 1 0 } geometry IndexedFaceSet { coord Coordinate { point [ 0 0 0, 1 0 0, 1 1 0, 0 1 0 ] } coordIndex [ 0 1 2 3 -1, 3 2 1 0 -1 ] } ] } Titel, Ort, Datum - Vorname Name49 PPT_Master_IGD_v2009.200.ppt http://doc.instantreality.org/tools/x3d_encoding_converter/

50. © Fraunhofer IGD HTML Events: user interaction through DOM Events …

51. © Fraunhofer IGD HTML Events: user interaction through DOM Events - V2 document.onload = function() { document.getElementById('box').addEventListener('click', function() { document.getElementById('mat').setAttribute('diffuseColor', 'olive'); }, false) };

52. © Fraunhofer IGD Efficient Picking and Interaction  JS-based SG traversal slow  Using „picking buffer“ instead  WebGL limitations  8 bit RGBA texture  No MRT in FBO  Single-pass render-buffer  Get position, normal and object ID within one step/pass

53. © Fraunhofer IGD Efficient Picking and Interaction  Single-pass render-buffer approach  Distance d to camera in RG channel (16 bit)  Object- or per-vertex id in BA channel (16 bit)  Allows identifying up to 65 535 different objects  Normal vectors derived using 2 × 2 window  Important for navigation and interaction  Read back 4 values instead of only one  Calculate cross product

54. © Fraunhofer IGD HTML Events: 3DPickEvent extends DOM MouseEvent interface 3DPickEvent : MouseEvent { readonly attribute float worldX; // 3d world coordinates at pick position readonly attribute float worldY; readonly attribute float worldZ; readonly attribute float normalX; // picked surface normal readonly attribute float normalY; readonly attribute float normalZ; … } … function updateTrafo(event) { var t = document.getElementById(‘trafo‘); var norm = new x3dom.fields.SFVec3f(event.normalX, event.normalY, event.normalZ); var qDir = x3dom.fields.Quaternion.rotateFromTo(new x3dom.fields.SFVec3f(0, 1, 0), norm); var rot = qDir.toAxisAngle(); t.setAttribute('rotation', rot[0].x+' '+rot[0].y+' '+rot[0].z+' '+rot[1]); t.setAttribute('translation', event.worldX+' '+event.worldY+' '+event.worldZ); }

55. © Fraunhofer IGD X3DOM Math Lib SFVec3f copy(v) parse( str) setValues(that) at(i) add(that) addS caled(that,s) subtract(t hat) negate() dot(th at) cross(that) reflec t(n) length() normali ze(that) multComponen ts(that) multiply(n) d ivide(n) equals(that,eps ) toGL() toString() setValueByStr(str) Titel, Ort, Datum - Vorname Name55 PPT_Master_IGD_v2009.200.ppt Quaternion parseAxisAngle(str) axisAngle(axis,a) rotateFromTo(fromVec,toVec) multiply(that) toMatrix() toAxisAngle() angle() setValue(matrix) dot(that) add(that) subtract(that) setValues(that) equals(that,eps) multScalar(s) normalize(that) negate() inverse() slerp(that,t) toGL() toString() setValueByStr(str) SFColor parse(str) colorParse(color) setValues(color) equals(that,eps) add(that) subtract(that) multiply(n) toGL() toString() setValueByStr(str) SFColorRGBA parse(str) setValues(color) equals(that,eps) toGL() toString() setValueByStr(str)

56. © Fraunhofer IGD X3DOM Math Lib SFMatrix4f copy(that) identity() zeroMatrix() translation(vec) rotationX(a) rotationY(a) rotationZ(a) scale(vec) lookAt(from,at,up) parseRotation(str) parse(str) e0() e1() e2() e3() setTranslate(vec) setScale(vec) Titel, Ort, Datum - Vorname Name56 PPT_Master_IGD_v2009.200.ppt mult(that) multMatrixPnt(vec) multMatrixVec(vec) multFullMatrixPnt(vec) transpose() negate() multiply(s) add(that) addScaled(that,s) setValues(that) setValue(v1,v2,v3,v4) toGL() at(i,j) sqrt() normInfinity() norm1_3x3() normInf_3x3() adjointT_3x3() equals(that) getTransform(translation, rotation,scaleFactor, scaleOrientation,center) decompose(t,r,s,so) polarDecompose(Q,S) spectralDecompose(SO,k) log() exp() det3(a1,a2,a3,b1,b2,b3, c1,c2,c3) det() inverse() toString() setValueByStr(str) Example Basic decomposition of x3dom.fields.SFMatrix4f (given as mat) var quat = new x3dom.fields.Quaternion(0, 0, 1, 0); quat.setValue(mat); var rotation = rot.toAxisAngle(); var translation = mat.e3(); Example Basic decomposition of x3dom.fields.SFMatrix4f (given as mat) var quat = new x3dom.fields.Quaternion(0, 0, 1, 0); quat.setValue(mat); var rotation = rot.toAxisAngle(); var translation = mat.e3();

57. © Fraunhofer IGD element Part of DOM/ HTML document like every other HTML element (e.g., etc.) element Part of DOM/ HTML document like every other HTML element (e.g., etc.) document.getElementById('body_color'). setAttribute("diffuseColor", '#000066'); Click on element… Interaction via standard Web technologies (e.g. JavaScript Events etc.) …causes attribute change of url (i.e., other wheel rims appear) Example 1: Interactive Car Configurator Using HTML + JavaScript, to change color and rims

58. © Fraunhofer IGD Example 2: Painting Textures of 3D Objects element Part of DOM/ HTML document like every other HTML element (JavaScript implementation based on new WebGL API of HTML5 element) element Part of DOM/ HTML document like every other HTML element (JavaScript implementation based on new WebGL API of HTML5 element) jQuery UI (User Interface) jQuery JavaScript library: http://jqueryui.com/ jQuery UI (User Interface) jQuery JavaScript library: http://jqueryui.com/ HTML5 element Painted image used as texture on 3D object HTML5 element Painted image used as texture on 3D object

59. © Fraunhofer IGD Application (Large Data and Picking): 3D-Internet Design Review 59

60. © Fraunhofer IGD Navigation: moving the virtual camera interactively  Built-in navigation modes  Examine, walk, fly, lookat, game, helicopter and none   Abstract behavior dynamically maps to various user inputs: mouse, keys, multi-touch  Application-specific navigation  Use nav. type ‘none’  Move camera by updating position and orientation of

61. © Fraunhofer IGD Animations CSS 3D Transforms & CSS Animation  Utilized to transform and update nodes (WebKit version) #trans { -webkit-animation: spin 8s infinite linear; } @-webkit-keyframes spin { from { -webkit-transform: rotateY(0); } to { -webkit-transform: rotateY(-360deg); } } …

62. © Fraunhofer IGD Animations X3D TimeSensor and Interpolator nodes <orientationInterpolator id="oi" key="0.0 0.5 1.0" keyValue="0 1 0 0, 0 1 0 3.14, 0 1 0 6.28"> <ROUTE fromNode='ts' fromField='fraction_changed' toNode='oi' toField='set_fraction'> <ROUTE fromNode='oi' fromField='value_changed' toNode='trafo' toField='set_rotation'> The „ts“ triggers via the first ROUTE the „oi“, which provides the values for the rotation around the y-axis (0,1,0) The resulting value is then ROUTE‘d to the field ‘rotation’ of the node “trafo”, which results in an animation

63. © Fraunhofer IGD Entry points for getting started Some books “X3D: Extensible 3D Graphics for Web Authors” “The Annotated VRML 97 Reference” (explains concepts) X3DOM online documentation and code examples http://x3dom.org/docs/dev/http://x3dom.org/docs/dev/ (tutorials and docs) http://www.x3dom.org/school/http://www.x3dom.org/school/ (12 simple examples) http://www.x3dom.org/iX/http://www.x3dom.org/iX/ (7 examples with animation) http://www.x3dom.org/x3dom/test/functional/http://www.x3dom.org/x3dom/test/functional/ (lots of feature tests) More docs and tools http://www.instantreality.org/downloads/http://www.instantreality.org/downloads/ (InstantPlayer and aopt converter) http://doc.instantreality.org/documentation/getting-started/http://doc.instantreality.org/documentation/getting-started/ (links to X3D)

64. © Fraunhofer IGD Conclusions  Development costs: Web developer vs. graphics expert  Adaptability: Declarative material abstraction allows shading adoption per client hardware (e.g. GLSL, ray-tracing…)  Efficiency: UI events, culling, rendering can be implemented in native code, thus utilizes battery resources efficiently  Accessibility: High level navigation and interaction styles allow very late adaptations for specific use cases  Metadata: Allow indexing and searching content  Mash-ups: Asset reuse in new context  Security: No plugins or even direct GPU calls necessary  Powerful Abstraction for Web Applications Titel, Ort, Datum - Vorname Name64 PPT_Master_IGD_v2009.200.ppt

65. © Fraunhofer IGD X3DOM – Declarative (X)3D in HTML5 Introduction and Tutorial Thank You! Questions? http://www.x3dom.org/ http://examples.x3dom.org/