1Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

 Introduction  Context  Problem Modeling  Devices Synchronization  Computer vision  Conclusion Rémi Devinant DII5 / Devices synchronization for modeling 3D plane2

 More and more augmented reality applications  Development of more performant technologies  Best performances  Reliability tools for users Rémi Devinant DII5 / Devices synchronization for modeling 3D plane3

 Smartphones  More than 100 applications  2 known techniques ▪ Geolocalization and camera ▪ Camera, gyro, accelerometer ▪ Embedded systems constraints  « EyeTape »  prototype  Head up display  Display information superimposed on vision Rémi Devinant DII5 / Devices synchronization for modeling 3D plane4

 The project  Based on a CCD camera ▪ Taking frames in video ▪ 2 kinds of information to ▪ Static (buildings, urban environment…) ▪ Dynamic (people, cars…) Rémi Devinant DII5 / Devices synchronization for modeling 3D plane5

 Ubiquitous Computing  Mobile systems around users  Interacts with the environment  Devices used in parallel  Mobile system ▪ Video treatment ▪ 2 informations flow  Embedded systems problematic  Light mobile system  Light CPU power  Memory constraints Rémi Devinant DII5 / Devices synchronization for modeling 3D plane6

 Smartphone (iPhone 4) composed of  CCD  5mPixels (video 720p à 30fps)  Gyro  L3G4200D Digital 3-axis  Accelerometer  ST Micro LIS331DLH 3-axis  CPU  Apple A4 APL0398 (ARM Cortex A8)  DMA Memory  RAM  512Mo DRAM Samsung Rémi Devinant DII5 / Devices synchronization for modeling 3D plane7

 Devices synchronisation  Apple iOS  Apple iPhone OS  Based on a BSD Kernel  Mac OS X with cellphone services Rémi Devinant DII5 / Devices synchronization for modeling 3D plane8

 Video frames overlap  Video’s framerate preset  Purpose  Reduce or increase framerate  Original  30fps  Smartphone motion in space  Slow motion : video frames can overlap significantly  need for suppressing useless ones  Need for fusioning devices datas  Allow selection in video frames Rémi Devinant DII5 / Devices synchronization for modeling 3D plane9

 2 known methods  Hardware ▪ Can exist if it’s built-in ▪ Smartphones dont' have this characteristic  Software ▪ Can be implemented in the OS ▪ Require an personalized algorithm Rémi Devinant DII5 / Devices synchronization for modeling 3D plane10

 Introduction  Context  Problem Modeling  Devices Synchronization  Computer vision  Conclusion Rémi Devinant DII5 / Devices synchronization for modeling 3D plane11

 Each devices have different time delays  In retrieving datas  In transmitting datas  Require modeling each delays representing the time between the datas are required and the time where they are in memory Rémi Devinant DII5 / Devices synchronization for modeling 3D plane12

 Basic system architecture Rémi Devinant DII5 / Devices synchronization for modeling 3D plane13 Gyroscope Accéléromètre Capteur CCD Device Controller gyro Device Controller accéléromètre Device Controller CCD Buffer mémoire DMA

 Accelerometer  3 datas send into 1 block  3 axis acceleration sensor Rémi Devinant DII5 / Devices synchronization for modeling 3D plane14

Rémi Devinant DII5 / Devices synchronization for modeling 3D plane15 Accéléromètre Device Controller MémoireCPU Data Bus Adresses Bus Control Bus

 Accelerometer  Retrieving datas delay  Devices bus output: fbus  Datas volume: Vdata  Datas acquisition delay: ∆tacqacc  Bus acquisition delay (if busy): delay  Writing in memory delay depending on datas volume : ∆twrite  ∆tacc = ∆tacq + (fbus / Vdata) + ∆twrite + delay Rémi Devinant DII5 / Devices synchronization for modeling 3D plane16

 Gyro  3 datas send into 1 block  3 axis rotations Rémi Devinant DII5 / Devices synchronization for modeling 3D plane17

Rémi Devinant DII5 / Devices synchronization for modeling 3D plane18 Gyroscope Device Controller MémoireCPU Data Bus Adresses Bus Control Bus

 Gyro  Retrieving datas delay  Devices bus output: fbus  Datas volume: Vdata  Datas acquisition delay: ∆tacqacc  Bus acquisition delay (if busy): delay  Writing in memory delay depending on datas volume : ∆twrite  ∆tacc = ∆tacq + (fbus / Vdata) + ∆twrite + delay Rémi Devinant DII5 / Devices synchronization for modeling 3D plane19

 CCD Camera  Treatment of frames before they are placed in memory  sRGB colors  720p  Red/ Green/ Blue  Heavy data volume  DMA Rémi Devinant DII5 / Devices synchronization for modeling 3D plane20

Rémi Devinant DII5 / Devices synchronization for modeling 3D plane21 CCD Device Controller MémoireCPUDMA Data Bus Adresses Bus Control Bus

 CCD Camera  Retrieving datas delay  Devices bus output: fbus  Datas volume: Vdata  Datas acquisition delay: ∆tacqacc  Bus acquisition delay (if busy): delay  Writing in memory delay depending on datas volume : ∆twrite  ∆tacc = ∆tacq + (fbus / Vdata) + ∆twrite + delay Rémi Devinant DII5 / Devices synchronization for modeling 3D plane22

 Introduction  Context  Problem Modeling  Devices Synchronization  Computer vision  Conclusion Rémi Devinant DII5 / Devices synchronization for modeling 3D plane23

Rémi Devinant DII5 / Devices synchronization for modeling 3D plane24 T0 T1 ∆t Recording at 30fps Windows of N scan in memory for finding frame Start memory scan window

Rémi Devinant DII5 / Devices synchronization for modeling 3D plane25 Frame Spotted at ti - ti : date of scan - ∆t : retrieveing image delay - ti - ∆t = real date of the frame capture ∆t

 In this window  If a frame is spotted at pi  S∆ = pi – (N - 1) / 2  Next window start at  T1 + S∆  Window placed every time near the last image was spotted  Real time and embedded constraints context Rémi Devinant DII5 / Devices synchronization for modeling 3D plane26

 When the window is started, creation of coordinate table  Accelerometer ▪ ta[] = tspotted - ∆tacc  Gyro ▪ tg[] = tspotted - ∆tgyr Rémi Devinant DII5 / Devices synchronization for modeling 3D plane27

 Coordinate are chosed  tCoordonnees = tImgSpotted - ∆tImage  We try to keep the nearest coordinate Rémi Devinant DII5 / Devices synchronization for modeling 3D plane28

 Introduction  Context  Problem Modeling  Devices Synchronization  Computer vision  Conclusion Rémi Devinant DII5 / Devices synchronization for modeling 3D plane29

 A frame = spatial coordinates  Smartphone’s spatial localization  Spatial localization= frame plane  Creates frame 3D plane Rémi Devinant DII5 / Devices synchronization for modeling 3D plane30

Rémi Devinant DII5 / Devices synchronization for modeling 3D plane31

 Modeling 3D plane Rémi Devinant DII5 / Devices synchronization for modeling 3D plane32