1. Making Graphical Information Visible in Real Shadows on Interactive Tabletops Mariko Isogawa, Daisuke Iwai, and Kosuke Sato (Osaka Univ., Japan) IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 20, NO. 9, SEPTEMBER 2014 1

2. Outline Introduction Shadow-based interface for interactive tabletops Optical configuration Multitouch interface Image correction technique Experiments Applications Conclusion 2

3. Introduction 3 Aimed at realizing intuitive interaction with interactive tabletop systems, previous studies employed gesture or multi-touch user interfaces rather than the traditional mouse and keyboard.

4. Introduction 4 However, the sensing techniques for such interfaces suffer from false recognition and delay. Consequently, providing secure collaborative workplaces where users can safely input and browse private information such as PIN codes using interactive tabletops remains difficult.

5. Introduction 5 Another problem occurs when installing interactive tabletops in our daily residential spaces such as a kitchen, dining room, or study room. In particular, graphical information frequently appears inappropriately on the tabletop because inevitably the user’s hands or fingers unintentionally touch the tabletop surface or make a predefined gesture during routine activities such as cooking on a kitchen table. Such unintentional visual disturbance bothers the user.

6. Introduction 6 However, installing interactive tabletops in residential spaces opens up potential applications. Reading digital recipes on a kitchen table while cooking Checking weather forecasts on a dining table before or after a meal Web browsing on a study desk.

7. Introduction 7 To solve these problems, we divide the interaction scheme into two independent layers: 1.one for processing the user’s input, such as touch or gesture 2.the other for displaying graphical information. As a result, the interactive tabletop system will not display the resultant graphical information as long as the user does not activate the second layer. This means the second layer must be independent from the touch and gesture inputs.

8. Introduction 8 For the second layer interface, we focus on utilizing the shadow-based interface proposed by Minomo et al. [6], which completely relies on optical phenomena and does not require any computational processing, making it independent from touch/gesture input. Their system allows users to view graphical information by casting shadows with their bodies on a floor. [6] Y. Minomo, Y. Kakehi, M. Iida, and T. Naemura, “Transforming your shadow into colorful visual media: Multiprojection of complementary colors,” ACM Comput. Entertainment, vol. 4, no. 3, article 10, 2006.

9. Introduction 9 This is achieved with two overlapping projectors: 1.one projects the original image 2.the other projects its complementary image. The projected images that are overlaid on the floor become uniform gray owing to the additive nature of light. original imagecomplementary image

10. Introduction 10 It projects the complementary image onto the surface of an occluder polarization

11. Introduction 11

12. Outline Introduction Shadow-based interface for interactive tabletops Optical configuration Multitouch interface Image correction technique Experiments Applications Conclusion 12

13. Optical configuration one in a shadow area (“A”) the other in a non-shadow area (“B”) 13

14. Optical configuration rear projector front projectors 14

15. original image compensation image Optical configuration 15 based on the additive nature of light

16. Optical configuration 16 The compensation image appears on the occluder Therefore, we use a second front projector to project the complementary image of the compensation image onto the occluder so that a uniform gray image appears on the occluder.

17. Optical configuration 17 the resultant image will be a uniform gray and will not disturb the users’ activities.

18. Optical configuration 18 apply polarization filters so that does not reach the screen.

19. Outline Introduction Shadow-based interface for interactive tabletops Optical configuration Multitouch interface Image correction technique Experiments Applications Conclusion 19

20. Multitouch interface FTIR-based multi-touch interface [9]. 20 [9] J. Y. Han, “Low-cost multi-touch sensing through frustrated total internal reflection,” in Proc. Annu. ACM Symp. User Interface Softw. Technol., 2005, pp. 115–118. total internal reflection

21. Outline Introduction Shadow-based interface for interactive tabletops Optical configuration Multitouch interface Image correction technique Experiments Applications Conclusion 21

22. Image correction technique Because of the different color spaces among the projectors and the different optical devices (e.g., beam splitter and polarization filters) in the optical path of each projector, we need to match the color spaces of the projectors so that the target images are accurately reproduced on the screen and occluders. 22

23. Image correction technique The geometric registration of the projected images must be precisely implemented, because the distance between a user’s viewpoint and the screen is usually very short in interactive tabletops, and thus the user is sensitive to errors in the geometric registration of overlaid images. 23

24. Image correction technique Minomo et al. [6] use a special piece of equipment, i.e., a spectroradiometer, to calibrate the system. (manual operation.) We propose an image correction technique in which we use a standard RGB camera placed above and directed toward the screen to realize fully automatic, quick, and per-pixel correction. 24

25. Image correction technique We apply a gray code pattern projection technique for geometric registration. In particular, we obtain the pixel correspondence between the camera and each projector in the system by projecting gray code patterns onto the tabletop surface. 25

26. Image correction technique 26

27. Image correction technique Color correction technique proposed in [19], which compensates the color differences among the projectors on a perpixel basis using the camera. 27 [19] T. Yoshida, C. Horii, and K. Sato, “A virtual color reconstruction system for real heritage with light projection,” in Proc. Int. Conf. Virtual Syst. Multimedia, 2003, pp. 161–168.

28. Outline Introduction Shadow-based interface for interactive tabletops Optical configuration Multitouch interface Image correction technique Experiments Applications Conclusion 28

29. Experiments Prototype System 29

30. Experiments 30 when was turned off

31. Experiments 31

32. Experiments 32

33. Experiments 33 We used a high dynamic range (HDR) visual difference predictor 2 (HDR-VDP-2) [20] for the evaluation. It predicts the perceptual visual difference as a mean opinion score (MOS) [21] between two images. [20] R. Mantiuk, K. J. Kim, A. G. Rempel, and W. Heidrich, “HDRVDP- 2: A calibrated visual metric for visibility and quality predictions in all luminance conditions,” ACM Trans. Graph., vol. 30, no. 4, article 40, 2011.

34. Experiments 34 [21] Methodology for the Subjective Assessment of the Quality of Television Pictures, ITU-R-BT.500-11, 2002.

35. Experiments 35

36. Outline Introduction Shadow-based interface for interactive tabletops Optical configuration Multitouch interface Image correction technique Experiments Applications Conclusion 36

37. Applications 37 Secure Interactive Tabletop for Cooperative Work

38. Applications 38 Map Viewer

39. Outline Introduction Shadow-based interface for interactive tabletops Optical configuration Multitouch interface Image correction technique Experiments Applications Conclusion 39

40. Conclusion 40 Realizing an interactive tabletop surface without using any recognition techniques, including computer vision. The proposed optical design utilizes polarization in addition to the additive nature of light in order to ensure that the desired graphical information is displayed only in the shadow area on a tabletop surface. We combined the proposed shadow-based interface with a multi-touch detection technique

41. Conclusion 41 This approach enables a quick (almost no delay) response and does not cause any errors and misrecognitions. Case where the number of users is usually large. Our approach does not require such expensive sensing equipment. User would hold his or her arm above the surface to cast a shadow on the tabletop surface. Maintaining this posture for a long time inherently leads to a certain amount of fatigue.

42. Thanks for listening! 42