1. TEI of CreteMultimedia Lab Direct Mapping of X3D Scenes to MPEG- 7 Descriptions Markos Zampoglou, Patti Spala, Konstantinos Kontakis, Athanasios G. Malamos, J Andrew Ware

2. TEI of CreteMultimedia Lab What is our motivation i-Promotion: we work on a platform for VR web advertising. The goal of this project is to join cloud stored X3D objects to more complicated scenes and to publish them in the way that we used to publish GIF Banners! The first questions we had to answer was how can we search and retrieve the resources according to technical as well as functional and contextual characteristics. The resources might be the 3D geometry as well as Routes, Textures (images or videos), audios and Sounds embedded on the 3D. What is missing is an efficient annotation of X3D in MPEG7. MPEG-7 defines several descriptors to facilitate the description of 3D entities involving geometries, textures, animations and content metadata. However, the majority of 3D descriptors are centered on vertices representation of 3D objects, lacking in the ability to provide optimized semantic descriptions based on XML encoded 3D objects. However the existence of the 3D descriptors helped us to extend them to fit in our case allowing us to still use the older and the newer descriptors at the same time. I-Promotion is a project granted by the European Union and the Hellenic General Secretary of Research and Technology under the ”COOPERATION 2009 /09SYN-72-956” Framework.

3. TEI of CreteMultimedia Lab The i-Promotion platform

4. TEI of CreteMultimedia Lab The i-Promotion platform architecture A cloud-based repository for 3D scenes A remote rendering server – supplementing a potential local rendering approach Service-oriented content distribution HTML5-based clients – PCs, mobile devices, wall-to-wall projections… Supplementary modules for content adaptation – Device Interfaces, Media Framework Additional tools – 3D Worlds authoring tool, 3D banners promotion manager

5. TEI of CreteMultimedia Lab The i-Promotion platform: Standards conformance MPEG-7: Content description for 3D scenes – Content-based search MPEG-21: Multimedia Item declaration & adaptation – Scene organization (thumbnails, descriptors, rights) – Content adaptation for extreme device flexibility ???? : Scene structure composition X3D: 3D scene format – Offers a choice between server-based rendering or local, HTML5 compatible rendering (via X3DOM)

6. TEI of CreteMultimedia Lab MPEG-7 for 3D scenes MPEG-7 – Part 3: Visual includes descriptors for 3D model shape. Efforts towards full 3D scene description (shape, color, structure, light etc.) – 3DSEAM: X3D scene structure and object location (Bilasco et al. 2006) – Multimedia Interaction Model (Chmielewski 2012) – MPEG-7 Shape and semantics for MPEG-4 3D scenes (Bonhomme et al. 2007) – Various RDF and OWL Semantic annotations

7. TEI of CreteMultimedia Lab What MPEG 7 does support in 3D Geometry Descriptors MPEG-7 Part 3, Visual, defines two shape descriptors for 3D objects: the Shape3D Spectrum descriptor and the MultipleView Descriptor Container. Motion Descriptors MPEG-7 Part 3, Visual, defines three descriptors, The Camera Motion descriptor, The Motion Trajectory descriptor and the Motion Activity descriptor. Texture and Color Descriptors In order to describe textures, MPEG-7 Part 3, Visual, provides three texture descriptors: the TextureBrowsing Descriptor, the HomogeneousTexture Descriptor and EgdeHistogram Descriptor. The color description tools presented in, MPEG-7 Part 3, Visual, include four distinct color descriptors to annotate different color features, the DominantColor Descriptor, the ScalableColor Descriptor, and the ColorLayout and ColorStructure Descriptors. Meanwhile, two additional supporting descriptors are included, the ColorSpace and ColorQuantization Descriptors, which are used in conjunction with the DominantColor Descriptor, providing supplementary details to representative colors.

8. TEI of CreteMultimedia Lab MPEG-7 Extension Descriptors MPEG-7 PartNameUsage MPEG-7 Part 3 – Visual BoundingBox3D Position and size of a 3D object in the scene Geometry3D Types of primitive or complex geometries and their curvature. Metadata3D Additional metadata information (such as MatadataFloat, MetadataInteger, MetadataSet). Interactivity3D Interactions with other objects in the scene or with the end user, (based on X3D ROUTE elements). MotionTrajectory Animation characteristics of a moving object: type of animation, through the Interpolator or Follower nodes bound to that object, along with the motion path keypoint values. Viewpoint3D Viewpoint nodes’ position, orientation, animation, and coordinates. Lighting3D The type of X3D Light nodes used within an X3D Scene, their illumination specifications and their scope (global or not). MPEG-7 Part 5 – MDS Profile3D The X3D profile. Script3D If the file contains Script nodes, specifies the script class location (internal or external) and the scripting language used.

9. TEI of CreteMultimedia Lab Automatic extraction of MPEG-7 descriptions X3D and MPEG-7 are both XML-based standards We use XSLT for the direct extraction of (extended-) MPEG-7 compliant descriptions from X3D scenes – Both scenes are kept: the original X3D scene is retained for rendering and display, while the new MPEG-7 description is used for search and retrieval

10. TEI of CreteMultimedia Lab XSLT Mapping

11. TEI of CreteMultimedia Lab Extensions in MPEG-7 Part 3 Visual

12. TEI of CreteMultimedia Lab BoundingBox3D Elements NameDefinition BoundingBox3DType Descriptor Volumetric coordinates as obtained from the X3D BoundingBox element BoundingBox3DSize Width, Height and Length coordinates of the BoundingBox, passed into the BoxWidth, BoxHeight and BoxDepth attributes. Default values are (-1, -1, -1) as in the X3D bounding box specification. BoundingBox3DCenter Center coordinates of the BoundingBox. Each of the three values contained in bboxCenter is passed into the BoxCenterW, BoxCenterH and BoxCenterD attributes. Default values are (0, 0, 0) as in the X3D bounding box specification.

13. TEI of CreteMultimedia Lab BoundingBox3D

14. TEI of CreteMultimedia Lab BoundingBox3DType Fields MPEG-7 Description FieldsValue TypeInput from X3D Fields BoundingBox3DType Descriptor - Created if an X3D Grouping Node contains valid bboxCenter and bboxSize attributes BoundingBox3DSize- Created if an X3D Grouping Node contains a valid bboxSize attribute BoxWidthFloat Position[1] of the value of the X3D bboxSize attribute String BoxHeightFloat Position[2] of the value of the X3D bboxSize attribute String BoxDepthFloat Position[3] of the value of the X3D bboxSize attribute String BoundingBox3DCenter- Created if an X3D Grouping Node contains a valid bboxCenter attribute BoxCenterWFloat Position[1] of the value of the X3D bboxCenter attribute String BoxCenterHFloat Position[2] of the value of the X3D bboxCenter attribute String BoxCenterDFloat Position[3] of the value of the X3D bboxCenter attribute String

15. TEI of CreteMultimedia Lab BoundingBox Mapping

16. TEI of CreteMultimedia Lab Geometry3D Elements NameDefinition Geometry3DType Describes the types of primitive or complex shapes contained in an X3D file. Geometry3D Describes one shape from the X3D file. Metadata3D Specifies the additional metadata values, such as MetadataDouble, MetadataFloat, and MetadataSet.

17. TEI of CreteMultimedia Lab Geometry3D

18. TEI of CreteMultimedia Lab Geometry3DType Fields MPEG-7 Description Fields Value TypeInput from X3D Fields Geometry3D Descriptor - Created if an X3D Shape element contains an X3D Geometry node creaseAngleFloat Value of the X3D creaseAngle attribute convexBooleanValue of the X3D convex attribute ObjectTypeString Value of the X3D Geometry node type (node name) DEFStringValue of the X3D DEF attribute Metadata3D- Created if an X3DMetadataObject exists within the X3D Geometry node.

19. TEI of CreteMultimedia Lab Geometry3D Mapping

20. TEI of CreteMultimedia Lab Metadata3DType Elements NameDefinition Metadata3DType Additional metadata values contained within an X3D node, such as MatadataFloat, MetadataInteger, MetadataSet. name The name of the metadata type being described, from the Name attribute of the X3DMetadataObject. type Specifies the type of metadata being described, such as MatadataFloat, MetadataInteger, MetadataSet. reference The value of the reference field from the X3DMetadataObject. If the field is not provided or is empty, the meaning of the name field is considered implicit to the characters in the string. value All metadata values as derived from the corresponding X3DMetadataObject. In the case of a MetadataSet, the value element contains all internal elements and values. Ref The XPath location from the X3D XML graph of the metadata element being described.

21. TEI of CreteMultimedia Lab Metadata

22. TEI of CreteMultimedia Lab MetadataSet

23. TEI of CreteMultimedia Lab Metadata3DType Fields MPEG-7 Description Fields Value TypeInput from X3D Fields Metadata3D Descriptor - If an X3D Metadata element such as MatadataFloat, MetadataInteger, MetadataSet, exists. nameString Value of the X3DMetadataObject name attribute typeString Value of the X3DMetadataObject type attribute referenceString Value of the X3DMetadataObject reference attribute value Integer/ Float/ String/ Double/ nested Metadata3D element Value of the X3DMetadataObject attributes’ value. Depends on the X3DMetadataObject type (Float, Integer, String). In the case of X3D MetadataSet, the value attribute holds the X3D MetadataSet children RefXPath XPath route to the specific X3DMetadataObject.

24. TEI of CreteMultimedia Lab MetadataSet Mapping

25. TEI of CreteMultimedia Lab MotionTrajectory Elements NameDefinition MotionTrajectory Type Specifies the motion trajectory of a moving object, as defined in the MPEG- 7 ISO Standard. A MotionTrajectory Descriptor is formed for each X3D Interpolator node. CoordRef Allows the definition of a CoordRef or CoorfDef element, depending on the coordinate system. The CoordRef element is used to reference a two- dimensional coordinate system. It is therefore not used in the case of X3D descriptions. CoordDef If the attribute value is set to meter, then the trajectory is considered three- dimensional and the coordinate system is local to the specified trajectory Params the spatio-temporal key-points and interpolations used to express the trajectory data, as per official MPEG-7 ISO. The X3D Interpolator node key and keyValue attributes are provided in the Params element to describe the trajectory of the objects’ motion. In the case of dynamic components such as the X3DFollowerNode, annotation of the trajectory can only be described by their initial estimation through their initialValue and initialDestination attributes, as a smooth motion trajectory between dynamically produced vertices created within the scene during the rendering process.

26. TEI of CreteMultimedia Lab MotionTrajectory

27. TEI of CreteMultimedia Lab MotionTrajectory Fields MPEG-7 Description FieldsValue TypeInput from X3D Fields MotionTrajectory Descriptor- Created if when an X3D Interpolator exists. motionTypeString The X3D Interpolator motion activity type. motionComponentString The X3D Interpolator node name. CoordDef- Child element of MPEG-7 MotionTrajectory Descriptor UnitsString Value of “meter”, to specify 3D content held in Params element Params- Child element of MPEG-7 MotionTrajectory Descriptor KeyTimePoint- Child element of the Params element in the MPEG-7 MotionTrajectory Descriptor MediaRelIncrTimePointInteger Value of key attribute of the X3D Interpolator node. One element for each X3D Intepolator node key attribute. InterpolationFunctions- Child element of the Params element in the MPEG-7 MotionTrajectory Descriptor KeyValueFloat Value of the keyValue attribute of the X3D Interpolator node. One element for each X3D Intepolator keyValue attribute.

28. TEI of CreteMultimedia Lab The MotionType attribute X3D Interpolator NodeMPEG-7 Motion Type ColorInterpolatorLinear CoordinateInterpolatorLinear CoordinateInterpolator2DLinear NormalInterpolatorLinear OrientationInterpolatorLinear PositionInterpolatorLinear PositionInterpolator2DLinear ScalarInterpolatorLinear SplinePositionInterpolatorNon-Linear Spline SplinePositionInterpolator2DNon-Linear Spline SplineScalarInterpolatorNon-Linear Spline SquadOrientationInterpolatorNon-Linear Orientation Ease-In-Ease-Out X3DFollowerNodeSmooth

29. TEI of CreteMultimedia Lab MotionTrajectory Mapping

30. TEI of CreteMultimedia Lab Interactivity3D Elements NameDefinition Interactivity3DType Describes the interactions that occur within an X3D file, specifically from the X3D ROUTE elements. TriggerSource The source that triggers a specific animation. If the node is triggered internally depending on other nodes or script functions, the TriggerSource is defined as an InternalTrigger. Route The corresponding X3D ROUTE node. Specifies the nodes it connects together and their equivalent node types.

31. TEI of CreteMultimedia Lab Interactivity3D

32. TEI of CreteMultimedia Lab Interactivity3D Fields MPEG-7 Description FieldsValue TypeInput from X3D Nodes Interactivity3DType- Specifies interactions contained within an X3D file, as defined in the X3D ROUTE node. TriggerSourceString “UserTrigger” or “InternalTrigger”, depending on the X3D fromNode attribute RouteXPath XPath route to the specific X3D ROUTE node. fromNodeStringValue of the X3D fromNode attribute fromNodeTypeString The X3D Node Type of the X3D fromNode attribute toNodeStringValue of the X3D toNode attribute toNodeTypeString The X3D Node Type of the X3D toNode attribute

33. TEI of CreteMultimedia Lab Interactivity3D

34. TEI of CreteMultimedia Lab MetadataHeader Fields MPEG-7 Description Fields Value TypeInput from X3D Fields MultimediaContent Descriptor - Created when the MPEG-7 DescriptionMetadata Descriptor is formed. id String String value of the X3D document file name. Multimedia - Created when the MPEG-7 MultimediaContent element is formed. MediaLocator MediaUri String String value of the X3D document file path.

35. TEI of CreteMultimedia Lab MetadataHeader MultimediaContent Descriptor Mapping

36. TEI of CreteMultimedia Lab Viewpoint3D Elements NameDefinition Viewpoint3DType Position, orientation, animation properties and coordinate system of each X3D Viewpoint node. Viewpoint3D Contains the description, fieldOfView, isAnimated and animatedBy attributes. Viewpoint3DPosition Consists of three sub-elements, Orientation, Position and CenterOfRotation, providing coordinates of the X3D Viewpoint. Orientation Orientation coordinates of the Viewpoint node. Position Positional coordinates of the Viewpoint node. CenterOfRotation If the X3D Viewpoint node contains a centerOfRotation attribute, the value is passed into the CenterOfRotation.

37. TEI of CreteMultimedia Lab Viewpoint3D

38. TEI of CreteMultimedia Lab Viewpoint3D fields MPEG-7 Description Nodes Value TypeInput from X3D Nodes Viewpoint3DType- Viewpoint3D- descriptionString String value of the description provided in the corresponding X3D Viewpoint node fieldOfViewFloat Float value of the X3D Viewpoint fieldOfView attribute isAnimatedBoolean Determines whether the X3D Viewpoint is bound to an X3D animation node animatedByString The animation node type bound to the Viewpoint node Viewpoint3DPosition- The specific positional characteristics of the X3D Viewpoint node within the X3D file. OrientationFloat VectorThe X3D Viewpoint orientation attribute PositionFloat VectorThe X3D Viewpoint position attribute CenterOfRotationFloat VectorThe X3D Viewpoint CenterOfRotation attribute

39. TEI of CreteMultimedia Lab Viewpoint3D Mapping

40. TEI of CreteMultimedia Lab Lighting3D Elements NameDefinition Lighting3DTypeDescribes each X3D Lighting node contained in a scene. Lighting3DOne Lighting node from the corresponding X3D file. Lighting3DCharacteristicsDefines specific characteristics on the X3D Lighting node. Lighting3DSpecsThe exact specifications on the X3D Lighting node definition within the scene. It consists of four sub- elements, namely Attenuation, Colour, Direction, and Location. AttenuationThe attenuation properties of the X3D Lighting node. DominantColorTcolour properties of the X3D Lighting node. DirectionThe directional lighting properties of the lighting node. LocationThe location properties of the illuminated geometry.

41. TEI of CreteMultimedia Lab Lighting3D

42. TEI of CreteMultimedia Lab Lighting3D Fields MPEG-7 Description NodesValue TypeInput from X3D Nodes Lighting3DType - Lighting3D -One Light node from the X3D document name StringThe DEF name provided in a Light node lightingType StringThe Light type: Directional, Point or SpotLight. isOn BooleanThe X3D Light node “on” attribute usedGlobally BooleanThe X3D Light node usedGlobally attribute Lighting3DCharacteristics -Specific characteristics of the Light node ambientIntensity FloatThe X3D Light node ambientIntensity attribute intensity FloatThe X3D Light node intensity attribute beamWidth FloatThe X3D Light node beamWidth attribute cutOffAngle FloatThe X3D Light node cutOffAngle attribute radius FloatThe X3D Light node radius attribute Lighting3DSpecs -Specifications of the X3D Light node Attenuation Float VectorThe X3D Light node attenuation attribute Direction Integer VectorThe X3D Light node direction attribute Location Integer VectorThe X3D Light node location attribute DominantColor MPEG-7: DominantColor Type MPEG-7 DominantColor descriptor formed by the X3D Light colour attribute

43. TEI of CreteMultimedia Lab Lighting3D Mapping

44. TEI of CreteMultimedia Lab Extensions in MPEG-7 Part 5 Multimedia Description Schemes

45. TEI of CreteMultimedia Lab Profile3D & Script3D

46. TEI of CreteMultimedia Lab Profile3D Elements NameDefinition Profile3DTypeIndicates the profile type as defined in the corresponding X3D file. (Core, Interactive, MPEG_interactive, Immersive)

47. TEI of CreteMultimedia Lab Profile3D Fields MPEG-7 Description Fields Value TypeInput from X3D Fields Profile3D Descriptor StringThe X3D root node profile attribute Version Descriptor FloatThe X3D root node version attribute

48. TEI of CreteMultimedia Lab Profile3D Version Mapping

49. TEI of CreteMultimedia Lab Script3D Elements NameDefinition Script3DTypeThe scripting type of the X3D Script node. externalScript The external scripting class that performs the corresponding script functionalities on the X3D file described. internalScriptSpecifies that the Script node contains internal script functions and does not contain an outer scripting class reference. The internal script type is Javascript by default, as specified by the X3D standard.

50. TEI of CreteMultimedia Lab Script3D Fields MPEG-7 Description Fields Value TypeInput from X3D Fields Script3D Descriptor-One for each Script node internalScriptString “JavaScript”, if no url attribute is present in the X3D Script node externalScriptString “Java” or “Jscript”, if a url attribute is present and has a url with a.class or.js extension respectively

51. TEI of CreteMultimedia Lab Script3D Mapping

52. TEI of CreteMultimedia Lab Use of the ContentCollection Element An MPEG-7 DescriptionCollection element is created for each top-level X3D/Scene/X3D Grouping Node element that contains at least one Shape element. Subsequently, an MPEG-7 Content element is formed as a child element of the ContentCollection element previously defined, for each internal (child) X3D Grouping or Transform Node containing a Shape Node. The process for generating the Content elements is repeated recursively for all internal Grouping nodes at any level and a Content element is also created once for the top-level ContentCollection element.

53. TEI of CreteMultimedia Lab Use of the ContentCollection Element The XSLT texture retrieval template utilizes the same algorithm as the Grouping retrieval. The main difference is that each MPEG-7 ContentCollection Descriptor is created for every X3D Grouping node that contains an X3D Shape node with a child X3D Appearance object. If the Appearance node contains any type of X3D Texture Objects (e.g. ImageTexture or MovieTexture), then an MPEG-7 Content element is created.

54. TEI of CreteMultimedia Lab ContentCollection Fields MPEG-7 Description FieldsValue TypeInput from X3D Fields Collection Descriptor- Created if a top-level X3D Grouping node exists ContentCollection Descriptor- nameString The X3D DEF attribute in the X3D Grouping node. If no DEF attribute exists, a unique name is generated by the XSLT process. idString A unique id, generated by the XSLT process depending on the position of the X3D Grouping node Content- One for each X3D Grouping Node at any sub-level. A content element is also created for the top-level Grouping node. Multimedia- Created when the MPEG-7 Content element is formed. MediaLocator- MediaUriXPath XPath route to the specific X3D Grouping node.

55. TEI of CreteMultimedia Lab ContentCollection mapping

56. TEI of CreteMultimedia Lab Content Fields for X3D Appearance nodes MPEG-7 Description Fields Value Type Input from X3D Fields Content - Created for each X3D Appearance node that contains an X3D Texture object. Multimedia - Created when the MPEG-7 Content element is created. MediaLocator - MediaUri URL String String value from the URL attribute of the X3D Texture object.

57. TEI of CreteMultimedia Lab Content Mapping for X3D Appearance nodes

58. TEI of CreteMultimedia Lab http://www.medialab.teicrete.gr:8080/X3DtoMPEG7Tool/mainPage.html