1. Hyerim Park, Woontack Woo KAIST UVR Lab, Republic of Korea

2. Keywords: 5W1H metadata schema, ubiquitous computing, context-aware service, film-induced tourism We design what is termed the Location-based Film Experience in an augmented place to suggest a new means of delivering multimedia content on the AR platform. This becomes possible when the system collects and analyzes various data from distributed computing recourses and infers the context of the users and related objects. In this respect, it is essential to standardize and formulate metadata schema, and we propose the 5W1H metadata schema for a context-aware multimedia service on the AR platform in a ubiquitous computing environment. It’s filmed here. Enjoy the real world as the background of the scene~! It’s filmed here. Enjoy the real world as the background of the scene~!

3. The context-aware service is available when the system collects information from various sensors and distributed computing resources, integrates them, and infers the situation of the user and the environment [7] A smart multimedia service should consider integrated context-awareness of places where multimedia content is represented, the people who enjoy the content, and content that is provided through ubiquitous media [5, 6]. the 5W1H (Who, When, Where, What, How, Why) metadata schema is efficient when used to categorize a variety of information and intuitive for interpreting the contexts of objects. user place content Integration of context

4. The important feature of personal AR is that the displays are only perceived by a single user, raising the possibility of creating rich, personal experiences for all occupants of a shared space. The dramatic and narrative possibilities of AR experiences in traditionally static installations, such as theme parks, museums and historical sites, goes far beyond what is currently possible with non- AR technology (such as projection screens or the sequential or location-aware audio augmentations common in museums today). [1] Blair MacIntyre, Augmented Reality as a New Media Experience, In Augmented Reality, 2001. Proceedings. IEEE and ACM International Symposium on, pages 197-206, 2001 Figure 2. A Mad Tea Party. The user looks across the table at (a) the Mad Hatter to their left, (b) the Dormouse across from them, and (c) the March Hare to their right. The Mad Hatter has just been splashed with tea by the user, causing the March Hare to laugh at the Hatter. The Dormouse is asleep, but will soon wake up from the noise.

5. Context-awareness assists ubiquitous media applications in discovering the changeable contextual information and adapting their behaviors accordingly. A wide spectrum of context-aware schemes have been proposed over the last decade. However, most of them provide partial functionalities of context-awareness in ubiquitous media applications. They are specified to a certain task and lack of systematic research on context-awareness. 4.6 Investigation of existing context-aware multimedia systems Table 4 reports the comparison of selected context-aware media infrastructure. This implies that location is the most favorable context, and key-value context model is simple but yet powerful. This also shows that context reasoning and inconsistency functionalities are not available in most multimedia systems. [5] D. Zhang, H. Huang, et al., Survey on context-awareness in ubiquitous media, In Multimedia Tools and Applications, volume 67, issue 1, pages 179-211, 2013

6. They proposed unified context that represents user-centric context in terms of 5W1H and evaluated it with seven factors. Experiments showed that the proposed context can simply represent a user’s context in environments by assorting complicated information into six categories. Also, the unified context can enable sensor, user, and service to differently generate or exploit a defined 5W1H-semantic structure. Finally, unified context is structured with elements of 5W1H and attributes of each element so that any service can easily exploit the context for improving service. As a result, the proposed unified context enables context-aware services to quickly provide personalized services by exploiting unified user-centric context. [14] S. Jang and W. Woo, “Unified context describing user-centric situation: Who, where, what, when, how and why,” In The first Korea/Japan Joint Workshop on Ubiquitous Computing & Networking Systems, ubiPCMM, 2005

7. The 5W1H-based schema guarantees expansibility because the forthcoming metadata can also be classified into the 5W1H components without any prior knowledge. [9] E. Kim, J. Kim, W. Woo, Metadata Schema for Context-Aware Augmented Reality Application in Cultural Heritage Domain, In Digital Heritage, 2015 Fig. 1: 5W1H-based metadata schema for context- aware AR applications based on PoI-AR Anchor-AR Contents relation Fig. 2: Whole metadata structure for context-aware augmented reality service. Cultural Heritage Entity Information, Virtual Object, and Media Asset are subclasses of the AR Contents Class.

8.  3.1 5W1H Metadata schema Figure 2. The structure of Video Metadata for Location-based Film Experience in Augmented Places

9.  3.1 5W1H Metadata schema B. Class:: Video Clip This class defines the metadata of the video clips, and it considers how to show each video clip on the users’ personal mobile devices. Location Type (Enum={Indoor, Outdoor, Set, Location): The type of filming location Watching Point (GPS): The PoI to look at the video clip in the director’s position Acting Point (GPS): The PoI to take a photograph or record the scene in the actor’s position Coordinate Data ((Float, Float)): Coordinate of a video clip on a personal device display Scale ((Float, Float)): (horizontal, vertical) Size of a video clip on a personal device display Transparency (Float): The level of transparency of a video clip on a personal device screen for registration Order (String []): The order of the video clip instance to follow the story

10.  3.1 5W1H Metadata schema E. Class:: Additional Contents This class describes the additional AR content of each video clip. AR Content (Binary File): 2D or 3D AR content files associated with a scene Type (Enum={2D Model, 3D Model}): Type of AR content represented on a personal device display Coordinate Data ((Float x, Float y, Float z)): Coordinate of 2D or 3D model on a personal device display Scale ((Float, Float)): (horizontal, vertical) Size of the AR content on a personal device display

11.  3.1 5W1H Metadata schema D. Class:: Story This class describes the story of each video clip. Character (String []): Personal figures who lead a narrative about a scene Time (String []): The period of the story (e.g., The middle ages, modern times, season) Sub_Time (String): The specific time when the story occurs (e.g., day or night) Place (String): The location where the story occurs (e.g., house) Sub_Place (String): The sub-place where the story occurs (e.g., kitchen) Object (String): A description of an object related to the action in a scene Action: (String): A description of the action of the characters in a scene Reason (String): The reason for an action

12.  3.1 5W1H Metadata schema C. Class:: Navigation This class defines how to show video clips and additional AR content Remaining Time (Time): The remaining time for a tour if a tourist sets the ‘tour schedule' Expected Time (Time): The expected time when a tourist stays in one PoI Total Expected Time (Time): The total expected time when a tourist stays in the remaining PoIs Total Tour Time (Time): The total expected tour time on the site Point Guide (Binary File): The augmented arrows, circles, and markers that guide tourists to a particular PoI Pose Guide (Binary File): The augmented contour that guides tourists to follow the poses of actors in the video clip Watching Navigation ((Float, Float)): Coordinate of a point guide to the watching point on a personal device display Acting Navigation ((Float, Float)): Coordinate of a point guide to the acting point on a personal device display Pose Navigation ((Float, Float)): Coordinate of a pose guide to the acting point on a personal device display

13.  3.1 5W1H Metadata schema F. Class:: User This class defines the user context and information. It makes the context-aware service available. Tour Schedule (Time): Time limit for a tour that the user can set; it affects the number of PoIs to visit Location (GPS): The current location data from GPS in mobile device Direction ((Float x, Float y, Float z)): The current direction data from a gyro sensor in a mobile device Speed (Float): The current speed data of the user from a accelerometer sensor in a mobile device External Sensor (GPS): The specific location of the user from external sensors in the environment Service Record (String []): List of video ID instances Tag (String []): The new terms users add to video clips and additional content for later retrieval Picture (Binary File): A picture of users using the service Video (Binary File): Video of users using the service Text (Binary File): Text of users using the service User Rating (String []): The users’ rating of video clips and additional content; this modifies the path data and helps other users to use the service. User Review (String []): A user review of the AR service; this helps other users to use the service.

14.  3.1 5W1H Metadata schema A. Class:: Video The video class includes all other classes and denotes general information for a TV series or film. Tourist Attraction (Binary File): The information related to PoIs for tourists Classification (Enum={Movie, TV series}): Classification of the movie or TV series Viewership (String): Television viewership or the number of viewers of the movie Audience (String): The notification of the motion picture rating system Navigation. ID (String): ID of the navigation instance called in video class

15.  3.2 System Design

16.  4.1 Story Navigation Scenario: The story navigation function guides a tourist to follow the path of the characters in the story. As a tourist follows the story navigation, he watches video clips in the order of the story at the filming locations and thus understands the story more effectively. Metadata: Tour schedule, watching points, order, total tour time, official site, expected time, user ratings, user review, location and speed

17.  4.2 Watching Point Navigation and Playing a Video Clip Scenario: tourist can watch a video clip as if he views the camera lens as the director. Metadata: watching point with the watching navigation metadata, director, character, point guide, coordinate data, scale and transparency

18.  4.3 Acting Point Navigation and Pose Guide Scenario: many tourists capture photographs approximating the actor, even with the same pose in the same position Metadata: acting navigation, acting point, character, pose navigation, pose guide, picture, video.

19.  4.4 Augmented Content for Reproducing a Scene Scenario: There are many cases in which the real situation of the location differs from the scene in the production, due to the season, time, and the possible use of computer graphics. The metadata is necessary for reproducing the scene. Metadata: AR content, shooting time, location type, time, sub_time, place, and types(2D and 3D content), coordinate data and scale, tags, text, picture, video, watching point, acting point, modified time, accessed time, user rating, user review

20. A standardized and formulated metadata schema is critical in a ubiquitous computing environment and the 5W1H metadata schema will be a promising alternative for interpretation and management. In future work, we will implement the AR mobile application and qualitative and qualitative experiments will evaluate the system, and the proposed metadata will be complemented and modified to improve the location-based multimedia experiences on the AR platform., We also consider how to provide a new experience with video content in an augmented location