1. Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab & Brown University Armando de la Re Vega October 14th 2011

2. Introduction Transform a LCD into a display that supports 2D multitouch Unencumbered 3D gestures Inspired by LCD embedded optical sensors. Exploit the spatial light modulation capability of LCD to allow imaging without interfering with display functionality. 2

3. Introduction Using light sensors to detect multiple points in contact with the surface of LCD. Sharp Co. and Planar Systems inc. LCD with arrays of optical sensors interlaced within the pixel grid. Touch is determined from the spatial position of occluded sensors that receive less light. Touch, but not gestures. 3

4. Introduction This paper describes how to modify LCDs to allow capture and display. Captures the angle and intensity of light entering a co-located sensor array. Enables the detection of gestures. 4

5. BiDirectional Screen Sensor array located slightly behind the spatial light modulating layer of a conventional LCD. Two modes: Display mode: backlight and liquid crystal spatial light modulator function normal. Capture mode: backlight disabled and light modulator displays an array of pinholes or tiled broad banded code. Two applications: touch+gestures interaction and a light gun mode for interaction. 5

6. Contributions Thin depth sensing LCDs Support on 2D multitouch and 3D gestures. Alternates displayed image and optical mask. Maximize display and capture frame rate using optimally light-efficient mask patterns. Lensless Light Field Capture. Lensless light field camera, composed of optical sensor array and a spatial light modulator. Evaluation of pinhole arrays and tile broad band masks. Unencumbered 3D interaction. Novel interaction scenarios to recognize on- and off- screen gestures. 6

7. Benefits and Limitation Ability to capture multiple orthographic images. Potentially thin device. Not blocking the backlight or portions of the display. Separates LCD layers. Uses a pair of cameras increasing the device dimensions. Will reduce the native frame rate. Reduce of contrast by external illumination. 7

8. Design Goals Capture 3D enable depth and lighting aware interaction. Prevent image capture from interfering with image display. Support walk-up interaction. Achieve these goals with a portable, thin form facto device. 8

9. Comparison of Design Alternatives Capacitive, Resistive or Acoustic Modalities. Are effective for multitouch but not 3D gestures. Some capacitives detect approaching, but not their distance. This technologies not support lightning aware interaction. Optical sensing does. 9

10. Comparison of Design Alternatives Cameras behind, To the Side, or In Front the Display. Behind interferes with backlighting, casting shadows and brightness variations. In front or to the side, risks being occluded by users. In the bezel, increase the display thickness and suffer from user self occlusion. 10

11. Comparison of Design Alternatives Photo detector Arrays. Array located behind the LCD. Not suffer from user self occlusion. Detector layer can be extremely thin and optically transparent. Requires a small gap between the spatial light modulating and lighting planes. This gap allows to measure the angle of incident light and intensity. 11

12. Comparison of Design Alternatives Camera Arrays. A dense camera array is similar to Photo detector array. But they must be synchronized and assembled, increasing the engineering complexity. The sensors and lenses required by each cam, introduce backlighting non uniformity. 12

13. Designing a Thin-Depth Sensing LCD LCD Components Backlight: cold cathode fluorescent lamp or array of LEDs, a light guide, a rear reflecting surface, a diffuser and several brightness enhancing films. Spatial light modulator: a pair of crossed linear polarizers and a layer of liquid crystal molecules. 13

14. Designing a Thin-Depth Sensing LCD Hardware Design Remove light, light guide, reflector, brightness enhancing films and final diffuser. Use spatial light modulator to display masks. A coded image equivalent to the mask is formed on the diffuser that cameras can photograph. Additional array of LEDs behind the diffuser. 14

15. Designing a Thin-Depth Sensing LCD Optical design with Pinhole Arrays 18 LCD pixels between each. 0.2% of incident light reaches the diffuser. Extremely bright external light. 15

16. Designing a Thin-Depth Sensing LCD Optical design with Tiled Broadband Masks 19x19 LCD pixels. 50% of incident light reaches the diffuser. Allows external light to be dimmed by a 180 factor. 16

17. Multi-view Processing Heterodyne decoding method of Veeraraghavan et al. (2007). http://www.merl.com/papers/docs/TR2007-115.pdf Focus method of Nayar and Nakagawa (1994). http://www1.cs.columbia.edu/CAVE/publications/pdfs/Nayar_PAMI94.pdf Methods for synthetic aperture photography by Vaish et al.(2006). http://graphics.stanford.edu/papers/sap-recons/final.pdf Synthetically focus at a distance, computationally efficient approach of Ng (2005). http://graphics.stanford.edu/papers/fourierphoto/fourierphoto-600dpi.pdf 17

18. Interaction Modes Multitouch and 3D Interaction Supports on screen multitouch and off screen gestures. Real time depth map: Allows 3D tracking of objects in front of display. 18

19. Interaction Modes Lighting Sensitive Interaction Altering the light striking the screen. Model lightning application allows interactive relighting of virtual scenes. 19

20. Performance Implementation LCD Sceptre X20WG NagaII 20.1 LCD Spatial light separated from backlight and front diffuser polarizer. Spatial light mounted backwards. Backlight diffuser placed 2.5cm behind and with a linear polyvinyl alcohol-iodine (PVA) filter. 16 Luxeon Endor Rebel cool white LEDs. 20

21. Performance Implementation LCD Point Grey Flea2 cameras 1m behind the diffuser (1280x960 8 bit grayscale image 7fps). Intel Xeon 8 Core 2.66GHz, 4GB RAM NVIDIA Quadro FX 570. External halogen lamps with tiled-MURA. Pinhole mask requires additional halogen lamp above the region in front of display. 21

22. Performance Limitations Lower limit on the pixel sizes in LCD and sensor. Limit in the maximum angular and spatial resolution. Optimized for real time interaction, rather than high resolution photography. Frame rate limited to 7.5fps, video cameras and transfer rate. External lightning is required in image capture. Reduce of display contrast. Objects close can be occluded from ambient light. 22

23. Discussion and Future Directions Capable of dynamically updating the mask. Should be scaled to provide photographic quality images. Higher frame rates should allow flicker-free viewing and more accurate tracking. Could allow to track multiple users. 23

24. Conclusions Inspire the inclusion of features to light sensing displays. Including an array of low resolution cameras, will increase the angular resolution directly facilities unencumbered 3D interaction with thin displays. 24

25. Thanks 25