LightSync: Unsynchronized Visual Communication over Screen-Camera Links Wenjun Hu Hao Gu Qifan Pu

2D barcodes everywhere

(“Linking” to online information)

A video of barcodes instead “Transmitting” information (vs linking) – E.g., information on Gibraltarian attractions – E.g., movie show times for nearby theaters Receiver Transmitter

Visual communications on phones No need for additional connectivity Free, directional, lightweight links “Optical” software-defined radio – Run as apps on commodity devices

Many challenges Frame synchronization Poor image quality Limited computation power on phones Aesthetics…

In the rest of the talk… Frame synchronization challenges LightSync design Implementation and evaluation

FRAME SYNCHRONIZATION CHALLENGES

Frame rate mismatch Screen side (TX) 30 fps video playback Fairly stable Camera side (RX) 15 – 30 fps in practice Variable – Lighting conditions – Device capability – API/system factors High rate, steady Low rate, unsteady

Original frame Received frame patterns

Original frame Single frame 2-frame mix Mixing pattern varies by line Received frame patterns

2-frame mix Mixing pattern (& #components) varies by line Due to Rolling Shutter CMOS camera sensor scans one line (of pixels) at a time

Received frame patterns Original frame Single frame 2-frame mix 3-frame mix Consecutive frames indicating a missing frame Mixing pattern varies by line

Display frame rate C – Frame capture rate C/2 C 2C Occasional missed frames 4-frame mixes 3-frame mix Occasional missed frames

Synchronized communication Display frame rate C – Frame capture rate C/2 C 2C The only decodable pattern Frame synchronized Under-utilized and inefficient! Carries information But undecodable and dropped Cannot be used (of worst camera)

Unsynchronized communication Display frame rate (fps) 0 30 Works with any display and capture rates Higher camera capture rate, better performance Supported frame capture rate (fps)3015

Requirements Need to be able to decode mixed frames – In-frame color tracking per line Need to guard against lost frames/lines – Inter-frame erasure coding Essentially a temporal code across frames – Works with any (monochrome) per-frame code

Our solution: LightSync Need to be able to decode mixed frames – In-frame grayscale tracking per line Need to guard against lost frames/lines – Inter-frame erasure coding Essentially a temporal code across frames – Works with any (monochrome) per-frame code

LIGHTSYNC DEMO

LightSync Need to be able to decode mixed frames – In-frame grayscale tracking per line Need to guard against lost frames/lines – Inter-frame erasure coding

Per-line tracking: Mixing patterns 3 frames mixing → 3 blocks mixing Bl or w per block → 8 mixing possibilities ??? Frame 1 Frame 2 Frame 3 → RXTX

Per-line tracking: Mixing patterns RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3

Per-line tracking: Mixing patterns Trivial cases RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3

Per-line tracking: Mixing patterns Easy cases RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3

Per-line tracking: Mixing patterns Tricky cases, to be tracked per line RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3

Code area Per-line tracking: Reference blocks Corner Scanning direction Tracking bar orientation Tracking bars

Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar 2 Tracking bar 3 Tracking bar 4 4 reference blocks per line

Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar 2 Tracking bar 3 Tracking bar 4

Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar 2 Tracking bar 3 Tracking bar 4 Decoding by comparison Reference colors of mixes

LightSync Need to be able to decode mixed frames – In-frame grayscale tracking per line Need to guard against lost frames/lines – Inter-frame erasure coding

Inter-frame coding Simple pairwise XOR Group of 3 original frames Coded frames

Inter-frame coding Looping display on screen Coded frames … … All original frames Camera stops frame capture after message decoded

PERFORMANCE

Implementation Current frame capture rate Inter-frame intervals

Performance Goal recap: Works with any display/capture rate combinations Metric: Effective frame capture rate – Actual throughput scales with per-frame capacity

Schematic performance For a 20fps phone camera

Schematic performance For a 20fps phone camera Ideal Peaks at the camera rate Level at higher display rates Ideal

Schematic performance Peaks at ~ half the camera rate For a 20fps phone camera Conventional Drops to 0 at higher display rates Ideal

Schematic performance Conventional LightSync For a 20fps phone camera Peaks at the camera rate Slight drops at higher display rates Ideal

LightSync performance Nokia Lumia 920 (~30 fps)

LightSync performance Nokia Lumia 920 (~30 fps) HTC One X (~20 fps)

LightSync performance Nokia Lumia 920 (~30 fps) HTC One X (~20 fps) Huawei Ascend C8812E (~15 fps) Works with any display and capture rates Higher camera capture rate, better performance

Conclusion and outlook LightSync: Unsynchronized visual communication – Primitive for similar scenarios Lots of potential in the space of visual communication

Thank you!

Related work Temporal barcode design (inter-frame) – Unsynchronized 4D Barcodes Basically slows down display, only for TX up to 10fps Spatial barcode design (per-frame layout) – COBRA: Optimized for phone-to-phone – PixNet: High-end camera, computationally intensive, offline processing Other work on visible light communications