1. VR/AR project Progress Report 2016/07/14

2. Live Reality Fusion Concatenate the live videos from two or more rooms together. ◦ Observer room + remote room(s) Remote room Observer Room

3. Short-term Goal “See Through Wall” ◦ Concatenate R106 and R107 in IIS building. ◦ Static => Dynamic => Real-time

4. Plan Construct the model of the remote room. ◦ View + depth of items. Determine the distance and angle between observer and target wall in the observer room. Replace target wall in the live video with the view of the remote room.

5. Stereo Camera ZED stereo camera ◦ 3D Camera for Depth Sensing and Motion Tracking. ◦ https://www.stereolabs.com/ ◦ https://github.com/stereolabs/

6. Plan Construct the model of the remote room. ◦ View + depth of items. ◦ Offline construct the 3D model of the remote room using ZED stereo camera. Determine the distance and angle between observer and target wall in the observer room. Replace target wall in the live video with the view of the remote room.

7. Plan Construct the model of the remote room. ◦ View + depth of items. Determine the distance and angle between observer and target wall in the observer room. ◦ Stereo camera + HMD. Replace target wall in the live video with the view of the remote room.

8. Stereo Camera + HMD HMDs such as Oculus Rift DK2 do not have camera. Stack ZED stereo camera on top of HMD. ◦ Works as camera, distance meter, and tracker.

9. Plan Construct the model of the remote room. ◦ View + depth of items. Determine the distance and angle between observer and target wall in the observer room. Replace target wall in the live video with the view of the remote room. ◦ Determine which part of the current image is the wall to be replaced according to the depth map.

10. Issues Remain How to achieve/implement the previous mentioned functionalities with the stereo camera? Tracking dynamic objects in the remote room. Changes in the remote room. ◦ Light source, new static objects, …etc.

11. Current Status Waiting for the ZED stereo camera to arrive. APIs and codes studying. Implementing ◦ Wall/Background identification of a given image from the stereo camera.

12. Discussion