1. SocialSync: Synchronization for a Smartphone Camera Network Richard Latimer Rice University September 7, 2014

2. 2 2 Intro

3. 3 Why Smartphones? 1 billion smartphones sold in 2014 Growth in computer vision applications Advanced development platform Best camera is the one with you!

4. 4 4 What if Smartphones Could Collaborate? RefocusingFree Viewpoint Television 3D Depth Lytro Pelican Philips Kilner et al.

5. 5 Prior Work: Camera Arrays Traditional camera arrays are assumed to be embedded and tightly synchronized. High-Speed Videography Using a Dense Camera Array [Wilburn et al.] High Performance Imaging Using Large Camera Arrays [Wilburn et al.]

6. 6 6 A Smartphone Camera Array Involves the OS Stack Application Application Framework Library Runtime Linux Kernel/Drivers Hardware

7. 7 7 Social Capture: Problem Statement If you use multiple smartphone cameras to capture video simultaneously, there is an inherit synchronization challenge that manifests (due to OS stack) when the scene is moving.

8. 8 8 Scene Setup: How Bad is Misalignment? Scene Setup 100100 100100 100100 100100 100100 100100 100100 100100 Resolution Target Smartphone Camera Array

9. 9 9 Outcome: How Bad is Misalignment? Even when photos are captured simultaneously, misalignment of frames can cause problems when target is moving. Moving Resolution Target Captured by 4 Phones Simultaneously

10. 10 Source of Problem Frame 0Frame 1Frame 2 Frame 0Frame 1 Frame 2Frame 3 time Camera 1: Camera 2: Camera 3: Frame 2 Frame 0

11. 11 Goal of SocialSync Objective: Align frame sequences from smartphone cameras within tolerance ranges less than t milliseconds, where t < frame duration. Frame 0Frame 1Frame 2 Frame 0Frame 1 Frame 2Frame 3 time Camera 1: Camera 2: Camera 3: Frame 2 Frame 0

12. 12 1.Characterization 2.SocialSync Protocol 3.Evaluation

13. 13 1.Characterization Measurement Test Apparatus Results 2.SocialSync Protocol 3.Evaluation

14. 14 What We Use: Android 4.4 I/O Error Network Error Clock Error I/O Error Network Error Clock Error I/O Error Network Error Clock Error

15. 15 Network Setup I/O Error Network Error Clock Error I/O Error Network Error Clock Error I/O Error Network Error Clock Error 100100 100100 Hub and Spoke Network Server provides global reference clock Server broadcasts image capture request 100100 100100 100100 100100 100100 100100 100100 100100

16. 16 System for Measuring Capture Timestamps 16

17. 17 1.Characterization Measurement Test Apparatus Results 2.SocialSync Protocol 3.Evaluation

18. 18 Random Variables Associated with Capturing a Frame Setup Time: Delay to setup camera before capture of first frame Delivery Time: After capture, there is delay to when frame reaches application and a timestamp can be recorded.

19. 19 Camera Setup SetupFrame 0Frame 1Frame 2 time

20. 20 Camera 1: Is Setup Time Consistent? Setup time

21. 21 Camera 1: Is Setup Time Consistent? Setup time Frame 0

22. 22 Variability in Camera Setup Makes Aligning Frames Challenging Each time camera started, the start of the image sequence will be delayed by a variable amount of time. Setup can vary at least 35 ms StDev = 9.4 ms Not deterministic

23. 23 Frame Delay SetupFrame 0 time Frame 1Frame N

24. 24 Measurement of Frame Delay

25. 25 Delivery Time of Frame to Application If we keep measuring capture times of frames, they are spaced evenly Delivery time of frame to app is a proxy for estimating capture time with delay: μ = 67 ms σ = 4.5 ms

26. 26 1.Characterization 2.SocialSync Protocol Estimation Alignment 3.Evaluation

27. 27 SetupFrame 0Frame 1Frame 2 SetupFrame 0Frame 1 SetupFrame 0Frame 1Frame 2 time Camera 1: Camera 2: Camera 3: Frame 2 Goal of Social Sync

28. 28 Social Sync Protocol Estimation Step: Estimating Capture Timestamps to Measure Misalignment of Frames Alignment Step: Using Repeated Attempts at Launching Image Sequence to Reduce Sub-Frame Misalignment 28

29. 29 SetupFrame 0Frame 1Frame 2 SetupFrame 0Frame 1 SetupFrame 0Frame 1Frame 2 time Camera 1: Camera 2: Camera 3: Frame 2 If Misalignment is Due to Camera Setup

30. Frame 2 30 SetupFrame 0Frame 1Frame 2 SetupFrame 0Frame 1 SetupFrame 0Frame 1Frame 2 time Camera 1: Camera 2: Camera 3: Could We Estimate Capture Time By Calibrating for Setup Time?

31. 31 Recall Wide Variability in Setup If setup time had little variance, could align capture timestamps by offsetting Setup can vary at least 35 ms, when worst case frame misalignment was 33 ms (inverse of FPS) Can’t calibrate for setup time to reduce misalignment!

32. 32 Does Frame Delay Provide a Better Estimate of the Capture Timestamp? SetupFrame 0 time Frame 1Frame N

33. 33 Measurement of Frame Delay

34. 34 How does estimate of mean improve with number of samples? Stats 101: Standard error of a sample mean is Based on Central Limit Theorem, 95% confidence sample mean is within range:

35. 35 Mean Frame Delay Recap SetupFrame 0 time

36. 36 Mean Frame Delay Recap SetupFrame 0 time Mean Delay Est.:

37. 37 Mean Frame Delay Recap SetupFrame 0 time Frame 1 Mean Delay Est.:

38. 38 Mean Frame Delay Recap SetupFrame 0 time Frame 1Frame N Mean Delay Est.:

39. 39 Samples Required for Estimate Since mean delay is, we can determine the capture image sequence position for milliseconds using: 22 samples for 2 ms accuracy 85 samples for 1 ms accuracy 39

40. 40 Social Sync Protocol Estimation Step: Estimating Capture Timestamps to Measure Misalignment of Frames Alignment Step: Using Repeated Attempts at Launching Image Sequence to Reduce Sub-Frame Misalignment 40

41. 41 Synchronization Tolerance time Goal: Want synchronization to a reference image sequence to be aligned within some tolerance

42. 42 Synchronization Attempt 1 SetupFrame 0 time

43. 43 Synchronization Attempt 1 time Frame N SetupFrame 0

44. 44 Synchronization Attempt k SetupFrame 0 time

45. 45 Synchronization Attempt k time Frame N SetupFrame 0

46. 46 Obtaining Synchronization Let T be the worst case misalignment between frames Let t be the range specified by the sync tolerance, where t ≤T Let p = t/T, be the probability of synchronization error being reduced for a single phone 46

47. 47 Single Camera Sync Probability After k Attempts P k = 1-(1-p) k Expected Number of Sync Cameras n*P k 47

48. 48 Expected Attempts to Sync N Cameras 48

49. 49 Characterization SocialSync Protocol Evaluation

50. 50 Evaluating Synchronization NaïveSyncSocialSync 4 Cameras 8 Cameras 23 ms 35 ms 5 ms 6 ms 50 Smartphones in array to reduce artifacts not related to time synchronization error

51. 51 8 Cameras Naïve Sync SocialSync Reference Views Depth Map 51

52. 52 4 Cameras ReferenceNaïveSyncSocialSync 52

53. 53 4 Cameras NaïveSyncSocialSyncReference 53

54. 54 Conclusion 1.Characterize Android Camera 2.Developed SocialSync Protocol 3.Demonstrated SocialSync Future Work: - Greater control of Android camera system - In wild implementation 54

55. 55 Acknowledgements 1.Ashok, Lin, Ashu and their students 2.Co-Author Jason Holloway 3.LF4CV Reviewers, Paper Accepted 4.National Science Foundation 55