1. QoE Assessment in Olfactory and Haptic Media Transmission: Influence of Inter-Stream Synchronization Error Sosuke Hoshino, Yutaka Ishibashi, Norishige Fukushima, and Shinji Sugawara Department of Scientific and Engineering Simulation Nagoya Institute of Technology Nagoya 466-8555, Japan

2. Outline Background Purpose Olfactory and Haptic Media Display System Method of Assessments QoE Assessment Method QoE Assessment Results Conclusions and Future Work

3. Background Multi-sensory communications have been emerging. Olfactory, haptic and visual media Transmission over the Internet Degradation of quality of experience (QoE) It is necessary to clarify the influence of inter-stream synchronization error on QoE. For efficient media synchronization control We need to carry out media synchronization control. This study focuses on these media. Network delay Delay jitter Packet loss Vision, auditory, olfaction, tactile, gustation

4. No one has investigated influence of inter-stream synchronization error between olfactory and haptic media on QoE. Purpose Influence of inter-stream synchronization error between olfactory media and video is investigated on QoE. (inter-stream synchronization errors between about 30 ms and around +20 ms are hardly perceived). Previous Work We deal with harvesting fruit in a 3-D virtual space for an olfactory and haptic media display system. By QoE assessment, we investigate the influence of inter-stream synchronization error between olfactory and haptic media. This Work The haptic media are assumed to be synchronized with the visual media for simplicity.

5. Olfactory and Haptic Media Display System A user can touch and catch a grapefruit by manipulating PHANToM. The smell of grapefruit is diffused by SyP@D2 when a grapefruit is picked since the picked grapefruit approaches the user’s nose.

6. Olfactory Display (SyP@D2) Distance between a user and SyP@D2: About 0.3 m Intensity of diffusion: Maximum It takes about 2 seconds for a user to perceive the smell. SyP@D2 displays a smell to a user by blowing air into a smell cartridge attached to the main body.

7. Functions of Olfactory and Haptic Media Display System Every millisecond Every 10 milliseconds

8. We assume that olfactory and haptic media are synchronized when a user starts to perceive a smell on picking a grapefruit. Smell of grapefruit Method of Assessments (1 of 4) A user catches and starts to pull a grapefruit. The user releases the grapefruit. Haptic media Olfactory media Reaction force when pulling a grapefruit Reaction force by weight of grapefruit : Starting to output the smell : Outputting the reaction force when picking a grapefruit from a tree The user’s force exceeds 2.45 N. Buffering When we advance output timing of olfactory media Smell threshold value When we delay output timing of olfactory media

9. Method of Assessments (2 of 4) We start to output the olfactory media when the force given by a user exceeds smell threshold value. We generate a delay by buffering of the olfactory media. We generate temporal difference by outputting olfactory media earlier or later than haptic media. If a user picks grapefruit quickly, it is impossible for smell to reach a user before picking grapefruit even if we use the smell threshold value. However, We employ picking time in order to enable the smell to reach the user before picking grapefruit.

10. Olfactory and haptic media are synchronized when a user starts to perceive a smell on picking a grapefruit. Smell of grapefruit Method of Assessments (3 of 4) A user catch and start to pull a grapefruit. The user release the grapefruit. Haptic media Olfactory media Reaction force when pulling grapefruit Reaction force by weight of grapefruit : Starting to output the smell : Outputting the reaction force when picking a grapefruit from a tree The user’s force exceed 2.45 N. If the user’s force continues to exceed 2.45N for a specified time, the grapefruit is picked. picking time Smell threshold value Picking time

11. Method of Assessments (4 of 4) picking time long short It is impossible for smell to reach a user before picking a grapefruit. As we increase the picking time, it takes a longer time for a user to pick a grapefruit. We make two assessments (Assessments 1 and 2). The feel of picking a grapefruit in our system is different from that of reality. Picking time Assessment 10 ms Assessment 21500 ms

12.  Task: Each subject operates PHANToM and picks grapefruits in the virtual space.  Assessment item:  Total assessment time in each of Assessments1 and 2: About half an hour  We make Assessment 1 first. Then, we make Assessment 2 on a different day. QoE Assessment Method (1 of 2)  Inter-stream synchronization quality We obtain mean opinion score (MOS). ScoreDescription 5Excellent 4Good 3Fair 2Poor 1Bad Assessments 1 and 2

13. Subjects:  Number : 20  Age: Between 21 and 30 Subjects pick grapefruits while listening to classical music (The Four Seasons by Vivaldi).  Reason: To prevent the subject from being influenced by sound of blowing from SyP@D2. Subjects take a 30-second rest every judgment.  Reason: To prevent him/her from being influenced by lingering smell in the air and human olfactory adaptation. QoE Assessment Method (2 of 2)

14. QoE Assessment Results (1 of 5) Inter-stream synchronization quality in Assessment 1 (picking time: 0 ms) Outputting olfactory media earlier It takes approximately 2 seconds to perceive smell. About 1380 ms About 250 ms Average of inter-stream synchronization error Outputting olfactory media later Constant delay of olfactory media The subjects hardly perceive the inter-stream synchronization error when it takes from about 620 ms to around 1750 ms to perceive the smell after picking a grapefruit

15. QoE Assessment Results (2 of 5) Inter-stream synchronization quality in Assessment 1 (picking time: 0 ms) The inter-stream synchronization error is allowable when it takes from about 620 ms to around 2400 ms to perceive the smell after picking a grapefruit. About 1380 ms About +400 ms We cannot assess the inter-stream synchronization quality when the inter-stream synchronization error is smaller than about 1380 ms. It takes approximately 2 seconds to perceive smell. Outputting olfactory media earlier Outputting olfactory media later

16. QoE Assessment Results (3 of 5) Inter-stream synchronization quality in Assessment 2 (picking time: 1500 ms) The subjects hardly perceive the inter-stream synchronization error when it takes from about 500 ms to around 1000 ms to perceive the smell after picking a grapefruit. About 1500 ms Around 1000 ms Outputting olfactory media earlier It takes approximately 2 seconds to perceive smell. Outputting olfactory media later Constant delay of olfactory media Average of inter-stream synchronization error

17. Inter-stream synchronization quality in Assessment 2 (picking time: 1500 ms) QoE Assessment Results (4 of 5) The inter-stream synchronization error is allowable when it takes from about 110 ms to around 1400 ms to perceive the smell after picking a grapefruit. About 1890ms Outputting olfactory media earlier It takes approximately 2 seconds to perceive smell. Outputting olfactory media later Around 600ms

18. QoE Assessment Results (5 of 5) Comparison between Assessments 1 (picking time: 0 ms) and 2 (picking time: 1500 ms) We adopt results of Assessment 2 for safety. To clarify the reason of this result is for further study. Outputting olfactory media earlier The MOS value of the inter- stream synchronization quality in Assessment 2 is smaller than that in Assessment 1. Outputting olfactory media later

19. Conclusions  When it takes from about 500 ms to around 1000 ms to perceive the smell after picking a grapefruit, the inter-stream synchronization error is hardly perceived.  When it takes from about 110 ms to around 1400 ms to perceive the smell after picking a grapefruit, the Inter-stream synchronization error is allowable.  The difference in the feel of picking between two picking times is not large. We investigated the influence of inter-stream synchronization error between olfactory and haptic media on QoE. We also examine which of picking times (0 ms and 1500 ms) is felt more real.

20. Future Work We plan to clarify the reason why the MOS value of the inter-stream synchronization quality in Assessment 2 (picking time: 1500 ms) is smaller than that in Assessment 1 (picking time: 0 ms). We will investigate the influence of inter-stream synchronization error between olfactory and haptic media when the diffusion of the smell stops after harvesting a grapefruit. We will examine the influences of packet loss and network delay jitter on QoE. We will study media synchronization control which takes advantage of the results obtained in this study.

22. Olfactory display (SyP@D2) We can attach six cartridges to SyP@D2. SyP@D2 can diffuse each smell at any time. There are some parameters of SyP@D2 which we can set up. Name of smell, Duration of diffusion, Intensity of diffusion Maximum wind speed of SyP@D2: approximately 0.15 m/s We set the distance between a user and SyP@D2 to about 0.3 m. We measured the average time that it takes for the user to perceive a smell.

23.  Task: Each subject operates PHANToM and picks grapefruits in the virtual space.  Assessment item:  Total assessment time: About half an hour Subjects: Number : 20 Age: Between 21 and 30 QoE Assessment Method  Inter-stream synchronization quality We obtain MOS. ScoreDescription 5Excellent 4Good 3Fair 2Poor 1Bad We assume that olfactory and haptic media are synchronized when a user ceases to perceive a smell on releasing a grapefruit (if the user release the grapefruit, the grapefruit disappears).

24. Inter-stream synchronization quality QoE Assessment Results (when the diffusion of the smell stops after harvesting a grapefruit) Outputting olfactory media earlier Outputting olfactory media later Inter-stream synchronization error [ms] About 1310 ms About 960 msAbout 520 ms About +400 ms I 95% confidence interval

25. Comparison between Assessments 1 (picking time: 0 ms) and 2 (picking time: 1500 ms) QoE Assessment Results The allowable range in this study is narrower than that in previous study. This is because we need higher interactivity in the case where the olfactory and haptic media are perceived together than in the case where the olfactory media and video are perceived together. When the inter-stream synchronization error is between about 30 seconds and around +20 seconds, the inter-stream synchronization error between olfactory media and video is allowable.

26. QoE Assessment Results (haptic and visual media) Inter-stream synchronization error [ms] I 95% confidence interval Inter-stream synchronization quality when we change the haptic media

27. QoE Assessment Results (haptic and visual media) Inter-stream synchronization error [ms] I 95% confidence interval Inter-stream synchronization quality when we change visual media

28. Method of Assessments (5 of 5) Picking timeInter-stream synchronization error Assessment 1 0 ms About 1380 ~ around 90 ms 0 ~ +1000 ms (intervals of 100ms) +1000 ~ +3000 ms (intervals of 500ms) Assessment 2 1500 ms About 2680 ~ around 1740 ms 1500 ~ 500 ms (intervals of 100ms) 500 ~ +1500 ms (intervals of 500ms) We change the smell threshold value from 0 N to 2.10 N at intervals of 0.35 N in order to alter the average of the inter-stream synchronization error.

29. We also investigate difference in feel of picking a grapefruit.  Task: Subjects pick some grapefruits for each value of picking times (0 ms and 1500 ms) and are asked to choose one from among three options.  Total assessment time of this assessment: Around three minutes  We make this assessment immediately after carrying out Assessment 2. 1: Feel in the case where picking time is set to 0 ms is better. 2: Feel in the case where picking time is set to 1500 ms is better. 3: There is almost no difference between the two cases. QoE Assessment Method (2 of 3)

30. Difference in Feel of Picking between Two Picking Times The difference in the feel of picking between two picking times is not large.