1. Development of a system to reproduce the drainage from Tsujun Bridge for environment education Hikari Takehara Kumamoto National College of Technology

2. Background Tsujun Bridge – an aqueduct bridge made of stones in Kumamoto, Japan – a representative structure in water environment surrounding Shiraito plateau – This drains water to remove sediment in its water pipe The drainage from Tsujun Bridge – The drainage from Tsujun Bridge is utilized for education of water environment in Kumamoto

3. Problem and Situation Problem – Tsujun Bridge has been given damages whenever it drains water – The reason Leaking water from Tsujun Bridge Situation – A number of the drainage from Tsujun Bridge has decreased for protecting Tsujun Bridge – We cannot always watch the drainage from Tsujun Bridge

4. Purpose Creating a contents for learning the water environment surrounding Shiraito plateau – at any time – without giving any damages to Tsujun Bridge We developed a system which reproduces the drainage scene from Tsujun Bridge using Mixed Reality (MR) technology

5. Mixed Reality (MR) MR technology – Synthesize image of real environment (actually visible landscape) and image of virtual environment (Computer Graphics: CG) – Making the image as if there is a teapot on table image of real environment image of virtual environment synthesized image

6. Overview of system Capturing the camera image Geometric registration Synthesizing camera image and 3DCG model

7. Program flow Camera Calibration Capturing camera image Geometric Registration Synthesizing the CG and camera image Once Each frame

8. Program flow Camera Calibration Capturing camera image Geometric Registration Synthesizing the CG and camera image Once Each frame

9. Camera Calibration Intrinsic parameters which express how a camera projects a real space (object) on the projection plane are obtained object projection plane Cameras focus

10. Camera Calibration Camera coordinate Image coordinate Intrinsic parameters matrix A

11. Program flow Camera Calibration Capturing camera image Geometric Registration Synthesizing the CG and camera image At once Each frame intrinsic parameters

12. Capturing the camera image Capturing the camera image was used with OpenCV OpenCV(Open source Computer Vision library) – Library for Computer Vision

13. Program flow Camera Calibration Capturing camera image Geometric Registration Synthesizing the CG and camera image At once Each frame intrinsic parameters camera image

14. Geometric registration – to relate the position and orientation of camera with the coordinate system (world coordinate system) which is set at will in real environment – This relation is expressed by extrinsic parameters World coordinate C Camera coordinate M XWXW YWYW ZWZW Projection plane XCXC YCYC ZCZC

15. Extrinsic parameters – expressed as 3x4 matrix using rotation matrix R and translation vector T – This study assumed that the position of camera is fixed and that the orientation of camera can be changed freely – So, T is not need to be obtained, and R is only need to be obtained Extrinsic parameters matrix M

16. Extrinsic parameters – R matrix is obtained using Zhangs method with homography matrix Homography matrix H – transforms an image of a plane taken from one perspective into an image taken from the other perspective – expressed as 3x3 matrix XCXC YCYC ZCZC X C Y C Z C v H u v A plane u

17. Homography matrix – Obtaining this matrix needs more than 4 corresponding points between 2 images from each other perspectives – As the method of obtaining corresponding points, the method based on natural features is adopted – As the method of extracting natural features, SURF is used

18. Program flow Camera Calibration Capturing camera image Geometric Registration Synthesizing the CG and camera image At once Each frame intrinsic parameters camera image extrinsic parameters

19. Synthesizing the CG and camera image Camera image and 3DCG model of drainage were synthesized on the basis of the intrinsic and extrinsic parameters using OpenGL OpenGL (Open source Graphics Library) – Library for 3D graphics Intrinsic parameters Extrinsic parameters 3DCG model of drainage Camera image Synthesized image

20. Result of reproducing drainage scene

21. Result The drainage scene was reproduced to the correct position at the most part. An average of execution time – 0.247 [s] An average of Estimation error – 6.758 [pixel]

22. Conclusion In the future – The execution speed of this system will be improve using other method of obtaining corresponding feature points such as the optical flow – This system will be available even if the position of camera will be freely – This system will be operated in actual Tsujun Bridge, and will be utilized for learning water environment

23. End

24. SURF SURF (Speeded Up Robust Features) – The method extracts robust feature points for scaling and rotation of image and describes feature quantity – In following image, the corresponding points is obtained using SURF