1. LightSync: Unsynchronized Visual Communication over Screen-Camera Links Wenjun Hu Hao Gu Qifan Pu

2. 2D barcodes everywhere

3. http://www.windowsphone.com/zh-cn (“Linking” to online information)

4. A video of barcodes instead “Transmitting” information (vs linking) – E.g., information on Gibraltarian attractions – E.g., movie show times for nearby theaters Receiver Transmitter

5. Visual communications on phones No need for additional connectivity Free, directional, lightweight links “Optical” software-defined radio – Run as apps on commodity devices

6. Many challenges Frame synchronization Poor image quality Limited computation power on phones Aesthetics…

7. In the rest of the talk… Frame synchronization challenges LightSync design Implementation and evaluation

8. FRAME SYNCHRONIZATION CHALLENGES

9. Frame rate mismatch Screen side (TX) 30 fps video playback Fairly stable Camera side (RX) 15 – 30 fps in practice Variable – Lighting conditions – Device capability – API/system factors High rate, steady Low rate, unsteady

10. Original frame Received frame patterns

11. Original frame Single frame 2-frame mix Mixing pattern varies by line Received frame patterns

12. 2-frame mix Mixing pattern (& #components) varies by line Due to Rolling Shutter CMOS camera sensor scans one line (of pixels) at a time

13. Received frame patterns Original frame Single frame 2-frame mix 3-frame mix Consecutive frames indicating a missing frame Mixing pattern varies by line

14. Display frame rate C – Frame capture rate C/2 C 2C Occasional missed frames 4-frame mixes 3-frame mix Occasional missed frames

15. Synchronized communication Display frame rate C – Frame capture rate C/2 C 2C The only decodable pattern Frame synchronized Under-utilized and inefficient! Carries information But undecodable and dropped Cannot be used (of worst camera)

16. Unsynchronized communication Display frame rate (fps) 0 30 Works with any display and capture rates Higher camera capture rate, better performance Supported frame capture rate (fps)3015

17. Requirements Need to be able to decode mixed frames – In-frame color tracking per line Need to guard against lost frames/lines – Inter-frame erasure coding Essentially a temporal code across frames – Works with any (monochrome) per-frame code

18. Our solution: LightSync Need to be able to decode mixed frames – In-frame grayscale tracking per line Need to guard against lost frames/lines – Inter-frame erasure coding Essentially a temporal code across frames – Works with any (monochrome) per-frame code

19. LIGHTSYNC DEMO

20. LightSync Need to be able to decode mixed frames – In-frame grayscale tracking per line Need to guard against lost frames/lines – Inter-frame erasure coding

21. Per-line tracking: Mixing patterns 3 frames mixing → 3 blocks mixing Bl or w per block → 8 mixing possibilities ??? Frame 1 Frame 2 Frame 3 → RXTX

22. Per-line tracking: Mixing patterns RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3

23. Per-line tracking: Mixing patterns Trivial cases RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3

24. Per-line tracking: Mixing patterns Easy cases RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3

25. Per-line tracking: Mixing patterns Tricky cases, to be tracked per line RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3

26. Code area Per-line tracking: Reference blocks Corner Scanning direction Tracking bar orientation Tracking bars

27. Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar 2 Tracking bar 3 Tracking bar 4 4 reference blocks per line

28. Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar 2 Tracking bar 3 Tracking bar 4

29. Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar 2 Tracking bar 3 Tracking bar 4 Decoding by comparison Reference colors of mixes

30. LightSync Need to be able to decode mixed frames – In-frame grayscale tracking per line Need to guard against lost frames/lines – Inter-frame erasure coding

31. Inter-frame coding Simple pairwise XOR Group of 3 original frames Coded frames

32. Inter-frame coding Looping display on screen Coded frames … … All original frames Camera stops frame capture after message decoded

33. PERFORMANCE

34. Implementation Current frame capture rate Inter-frame intervals

35. Performance Goal recap: Works with any display/capture rate combinations Metric: Effective frame capture rate – Actual throughput scales with per-frame capacity

36. Schematic performance For a 20fps phone camera

37. Schematic performance For a 20fps phone camera Ideal Peaks at the camera rate Level at higher display rates Ideal

38. Schematic performance Peaks at ~ half the camera rate For a 20fps phone camera Conventional Drops to 0 at higher display rates Ideal

39. Schematic performance Conventional LightSync For a 20fps phone camera Peaks at the camera rate Slight drops at higher display rates Ideal

40. LightSync performance Nokia Lumia 920 (~30 fps)

41. LightSync performance Nokia Lumia 920 (~30 fps) HTC One X (~20 fps)

42. LightSync performance Nokia Lumia 920 (~30 fps) HTC One X (~20 fps) Huawei Ascend C8812E (~15 fps) Works with any display and capture rates Higher camera capture rate, better performance

43. Conclusion and outlook LightSync: Unsynchronized visual communication – Primitive for similar scenarios Lots of potential in the space of visual communication

44. Thank you! wenjun@microsoft.com

45. Related work Temporal barcode design (inter-frame) – Unsynchronized 4D Barcodes Basically slows down display, only for TX up to 10fps Spatial barcode design (per-frame layout) – COBRA: Optimized for phone-to-phone – PixNet: High-end camera, computationally intensive, offline processing Other work on visible light communications