1. 1 Halden Project VR Workshop, 2-3 March 2005 Development of a Tracking Method for Augmented Reality Applied to Nuclear Plant Maintenance Work Presentation by Hirotake Ishii Research Associate, Kyoto University, Japan Guest Scientist, Institute for Energy Technology, Norway

2. 2Halden Project VR Workshop, 2-3 March 2005 Background Serious situation of NPP Both of improvement of safety and reduction of cost are required. Introduction of free electricity market. Difficulties of maintenance for aged NPPs. Decrease of expert maintenance workers. Need further development of hardware and software for NPP operation Augmented Reality (AR) is one of the promising technologies that can improve efficiency and safety. Support for maintenance work in plant field There are some rooms to improve its efficiency and safety by introducing state-of-the-art information technologies.

3. 3Halden Project VR Workshop, 2-3 March 2005 What is the AR? Augmented Reality(AR) expands the surrounding real world by superimposing computer-generated information on the user ’ s view. By using Augmented Reality, it becomes possible for the workers to understand various information of the maintenance work intuitively. Destination Superimposed information

4. 4Halden Project VR Workshop, 2-3 March 2005 Key Technology of the AR Tracking Measure the position and rotation of user ’ s view in real time to superimpose virtual object/information at correct position. Many kinds of tracking methods are developed GPS (Differential GPS, Real Time Kinematics GPS) Ultrasonic/Magnetic/Infrared Sensor Inertial Sensor Artificial Marker / Marker-less Hybrid of above (Combination) can not be used in NPP

5. 5Halden Project VR Workshop, 2-3 March 2005 Requirement of Tracking Method applied to NPP Limitations of NPP field from the viewpoint of tracking Indoor Various size of equipment in wide area Lots of metal objects / obstacles / magnetic source Requirement Wide area Accuracy and stability Easy / no installation Less expensive The artificial marker technique has a possibility to be used in a plant.

6. 6Halden Project VR Workshop, 2-3 March 2005 Artificial Marker Technique Calculates the position and rotation of the camera from the position of markers pasted in the environment by image processing technique. Is applied to many AR applications. Strengths Accurate and stable Less expensive High scalability Weaknesses Available only when the distance between the marker and the camera is short (this means many markers need to be pasted in the environment) or the size of the markers is large.

7. 7Halden Project VR Workshop, 2-3 March 2005 Available Distance of Artificial Marker Maximum distance ARToolKit (VGA, f=4mm) 1m : 8 cm 3m : 25 cm 5m : 40 cm Problem There are many small objects like pipes in a plant. The surface of the objects is not flat. It is difficult to paste large markers. It is necessary to make the markers smaller or make it easier to be pasted in a complicated environment

8. 8Halden Project VR Workshop, 2-3 March 2005 Development of a New Tracking System Tracking system using barcode marker Long barcode marker Easy to paste on pipes Tracking system using circular marker Circle shaped marker with code inside Stable recognition in long distance

9. 9 Halden Project VR Workshop, 2-3 March 2005 Tracking System using Barcode Marker by Hiroshi Shimoda, Hirotake Ishii, Masayuki Maeshima, Toshinori Nakai, Zhiqiang Bian and Hidekazu Yoshikawa, (Kyoto University)

10. 10Halden Project VR Workshop, 2-3 March 2005 Design of Barcode Marker ID of marker (7 bits) Humming code (4 bits) 40mm 80mm 40mm 20mm Code”0” Code”1” Gap 11 bits data 7 bits for ID (128 kinds of markers) 4 bits for error correction (Humming code) Length From 640 mm to 1080 mm

11. 11Halden Project VR Workshop, 2-3 March 2005 Image of Tracking Method with Barcode Marker Field Worker Barcode Marker Check the crack of the upper pipe. Worker’s view Instruction information by AR Small Camera HMD Barcode markers are pasted on pipes 3 position (both edge and middle point) of each marker are registered in advance Position and rotation of workers are calculated by using 2 barcode markers (Solving P6P problem)

12. 12Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (1) Captured Image:

13. 13Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (2) Binarization: Binarize the captured image with the camera at preset threshold level.

14. 14Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (3) Labelling: Collect the connected pixels and mark a unique label on the connected part.

15. 15Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (4) Narrowing search area: Exclude the parts which have no possibility as the part of the marker by its area and shape.

16. 16Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (5) Extraction parts of marker: Pick up the 11 parts which are arranged in a line as a candidate of barcode marker.

17. 17Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (6) Decision of code: Decide the code of barcode marker from the area of each part. 0 1 0 0 0 0 1 1 0 0 1

18. 18Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (7) Comparison with pre-registered barcode marker: Correct the code of the marker with Humming code part and compare it with pre-registered barcode marker.

19. 19Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (8) Calculate position and rotation of camera Extract 3 points from 2 barcode markers and solve P6P problem

20. 20Halden Project VR Workshop, 2-3 March 2005 Basic Evaluation of Marker Recognition Purpose Evaluate basic ability of proposed tracking method Experimental Method Rotation Distance 120 lux Rotation Pipe: 60φ×1100 mm Pipe arrangement: Horizontal / Vertical Rotation: 0 / 20 / 40 / 60 / 80 degree Distance: 1 / 2 / 3 / 4 / 5 / 6 meters Camera

21. 21Halden Project VR Workshop, 2-3 March 2005 Examples of Captured Images Vertical, 0 degree, 6 meters Vertical, 80 degree, 4 meters Camera Resolution : H320xV240

22. 22Halden Project VR Workshop, 2-3 March 2005 Recognition Result Camera resolution: 320 x 240 Viewing angle: 53 x 40 degree Recognition rate: 10 – 30 fps (Pentium Mobile 1.4GHz)

23. 23Halden Project VR Workshop, 2-3 March 2005 Trial Use in Fugen NPP Water Purification Facility 10 barcode markers pasted in the area All marker ID and positions were registered in advance.

24. 24Halden Project VR Workshop, 2-3 March 2005 Example Recognized Markers

25. 25Halden Project VR Workshop, 2-3 March 2005 Result Walked around the area with prototype system. 1000 frame images were picked up, in which at least one marker was in the view. Recognition rate: 52.8% (47.2% failed) Erroneous recognition rate: 1.8% Cases of erroneous recognition a marker image was captured against the light, a marker was too far from the camera, and the angle of a marker against the camera direction was too large.

26. 26Halden Project VR Workshop, 2-3 March 2005 Example of Erroneous Recognition (Backlight)

27. 27Halden Project VR Workshop, 2-3 March 2005 Conclusion (Barcode Marker) Proposed marker-based tracking method for AR support of NPP maintenance work. Long barcode marker and simple image recognition. Evaluated basic ability of proposed tracking method in a laboratory. Long distance, enough fast and feasible. Trial use in Fugen NPP Recognition rate: 52.8%, erroneous recognition rate: 1.8%

28. 28 Halden Project VR Workshop, 2-3 March 2005 Tracking System using Circular Marker by Hirotake Ishii, Hidenori Fujino (Kyoto University) Asgeir Droivoldsmo (Institute for Energy Technology)

29. 29Halden Project VR Workshop, 2-3 March 2005 Weakness of Square Markers How to recognize square markers HIRO 1. Detect edges 2. Detect 4 lines 3. Calculate intersections 4. Calculate position and rotation Before binarize After binarize ●● ●●●●● ●● ●●●●● ● ● ● ● ●● ● ● ● ● ●●●●● ●●●●● Real line Estimated line Easily affected by jaggy shaped edges Distance between features Max half meters Distance between features is short (Max to size of marker) Accuracy depends on the distance between the features.

30. 30Halden Project VR Workshop, 2-3 March 2005 The center can be recognized accurately in any situation In Case of Circular Marker Well focusedBadly focusedLong distance Distance between features can be long. Distance between features Max several meters The center can be recognized accurately in any situation Triangulation by plural markers can be used.

31. 31Halden Project VR Workshop, 2-3 March 2005 Design of Circular Marker Outer black circle Center white circle Make as simple as possible Middle code area circle which consists of black or white fans Outer black circle and center white circle are used for calculating the threshold to analyze the code area. Diameter of the marker and the number of division of middle circle can be changed according to the situation. Number of division Number of unique markers 956 1099 11188

32. 32Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (1) Captured Image

33. 33Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (2) Calculation of logarithm

34. 34Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (3) 3x3 Sobel Filter

35. 35Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (4) Labeling

36. 36Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (5) Eliminate small area

37. 37Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (6) Recognize ellipse and eliminate non-ellipse area

38. 38Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (7) Recognize ID of marker and eliminate non-marker area Details are in the paper 21 7 3

39. 39Halden Project VR Workshop, 2-3 March 2005 Marker Recognition Algorithm (8) Result 21 7 3

40. 40Halden Project VR Workshop, 2-3 March 2005 Calculate position and rotation of camera Calculate by using both of the solutions from PnP solver and the rough information of each circular marker’s rotation (Details are in the paper) Marker position on the imageMarker shape on the image Accurate but plural solutions Single but unaccurate solution Compare Single and accurate solution PnP solver

41. 41Halden Project VR Workshop, 2-3 March 2005 Developed Software Camera Control library (C++) TCP/IP DLL for Client Marker Design Printed Markers Printer Marker Maker Marker Location (XML) Marker Visualize Core library (C++) TCP/IP library for Server Link Camera Driver CMU Camera Driver PGR IEEE1394 (IIDC) DragonFly FireFly USB, DV Direct Show Under development Tracking Result (Via TCP/IP) Tracking Server OpenGL DirectX JAVA3D JNI Display Software

42. 42Halden Project VR Workshop, 2-3 March 2005 Evaluation of the accuracy and stability (Single marker) Experimental Method One circular marker (diameter is 40mm) was pasted on a small box. The number of the division of middle circle was 8,9,10. Distance was changed from 300cm to 580cm with 20cm step. Angle was changed from 0 degree to 75 degrees with 15 degrees step. For each condition, 100 images were captured and the position and rotation was calculated. The average and the variance were calculated.

43. 43Halden Project VR Workshop, 2-3 March 2005 Example of the image (Distance:560cm, Angle:0 degree) Camera resolution: H1024 x V768 Focal Length: 8mm

44. 44Halden Project VR Workshop, 2-3 March 2005 Results (Maximum distance) Number of Division Angle (deg.) 01530456075 8520/560 500/560440/520380/440X/300 9520/560 500/560440/520380/440X/X 10520/560 500/560400/500380/420X/X Table : Maximum distance (diameter is 4cm) Succeeded in all frames(cm) / Failed in some frames(cm) Maximum distance of the circular marker is about 2 times of the square marker such as ARToolKit Camera resolution: H1024 x V768 Focal Length: 8mm

45. 45Halden Project VR Workshop, 2-3 March 2005 Results (single marker, depth) The accuracy of the position is not good.

46. 46Halden Project VR Workshop, 2-3 March 2005 Results (single marker, rotation)

47. 47Halden Project VR Workshop, 2-3 March 2005 Evaluation of the accuracy and stability (plural markers on a helmet) Experimental Method 22 circular markers were pasted on a helmet. The number of the division of middle circle was 8. Distance was changed from 300cm to 560cm with 20cm step. Angle was changed from 0 degree to 180 degrees with 15 degrees step. For each condition, 100 images were captured and the position and rotation was calculated. The average and the variance were calculated.

48. 48Halden Project VR Workshop, 2-3 March 2005 Example of the image (Distance:560cm, Angle:120 degree) Camera resolution: H1024 x V768 Focal Length: 8mm

49. 49Halden Project VR Workshop, 2-3 March 2005 Results (Position) The accuracy is greatly improved Black : plural Markers Red : Single Marker

50. 50Halden Project VR Workshop, 2-3 March 2005 Results (Rotation) Tracked in wide angle. In the case of single marker, maximum angle is about 60 degrees.

51. 51Halden Project VR Workshop, 2-3 March 2005 Application Example : Tracking in the Office

52. 52Halden Project VR Workshop, 2-3 March 2005 Conclusion (Circular marker) New circular marker has been designed Tracking system that recognizes plural circular markers at one time and calculates the position and rotation of the camera by triangular method has been developed. Two experiments have been conducted in order to evaluate the accuracy and stability of the proposed method. It was confirmed that the accuracy can be greatly improved by using plural markers at one time and the distance between the marker and the camera can be long compared to the conventional method.

53. 53Halden Project VR Workshop, 2-3 March 2005 Future Works Barcode Marker Improvement of proposed method (Multi-camera, camera resolution, recognition algorithm, etc.) Circular Marker Apply to real applications. (Visualization of radiation map in NPP, old church) Development of hybrid tracking Developed tracking methods can be used with other tracking methods such as ARToolKit at the same time. Combine marker-less method (short distance), square marker (middle distance), barcode marker and circular marker (long distance)