1/20 Obtaining Shape from Scanning Electron Microscope Using Hopfield Neural Network Yuji Iwahori 1, Haruki Kawanaka 1, Shinji Fukui 2 and Kenji Funahashi 1 1 Nagoya Institute of Technology, Japan 2 Aichi University of Education, Japan

2/20 Introduction Shape from Scanning Electron Microscope (SEM) images is the recent topic in computer vision. The position of a light source and a viewing point are the same under the orthographic projection. The object stand is rotated to some extent through the observation. Only these conditions can be used to recover the object shape. 2D Image of SEM Recovering 3D Shape

3/20 Previous Approaches (1) Photometric Stereo Estimation using the temporal color space use multiple images under the different light source directions. Linear Shape from Shading Photometric Motion the position of viewing point (camera) and light source should be widely located

4/20 Previous Approaches (2) Shape from Occluding Boundaries is limited to a simply convex closed curved surface Shape from Silhouette uses multiple images through 360 degree rotation, is also unavailable to object with local concave shape Surface Reflectance and Shape from Images Using 90 degree rotation to get the feature points However the rotation angle is limited to SEM

5/20 New Proposed Approach Uses optimization with two images observed through the rotation of the object stand 1. The appropriate initial vector is determined using the Radial Basis Function neural network (RBF-NN) from two images during rotation. 2. The optimization is introduced using the Hopfield like neural network (HF-NN).

6/20 Characteristics of SEM Image (1) Orthographic projection Rotation angle Reflectance property i : incident angle, < 70° s ≈ 0.5 R(i) is normalized to the range of 0 and 1.

7/20 Characteristics of SEM Image (2) z = F(x, y) F : height distribution p, q : gradient parameters l = (0, 0, 1) : light source direction : surface normal cos i = n ・ l = n z … (3) From Eq.(1)(2)(3), Cross Section of Reflectance Map(q=0)

8/20 Rotation Axis on Object Stand Under the orthographic projection the gradient of the rotation axis is the same for both images observed during rotation. ex.

9/20 Estimation of Rotation Axis 1. Assume A and B move A’ and B’ during the rotation 2. Set A and A’ be the same pixel 3. Then rotation axis is determined so that it becomes perpendicular to the line BB’ and passes through the point A.

10/20 Shape Recovery from Two Images Using Hopfield Neural Network Hopfield Neural Network (HF-NN) the mutual connection network the connection between the neurons are the symmetric HF-NN can be applied to solve the optimization problem of the energy function m1m1 m2m2 m3m3

11/20 Energy Function to be Minimized C 1, C 2, C 3 : the regularization parameters D : the target region of the object E 1 : the smoothness constraint E 2 : the error of the observed image brightness I(x,y) and the reflectance map R(p, q) E 3 : the error of the geometric relation for z and (p, q) (p,q,z): unknown variables

12/20 Initial Vector for Optimization (1) Radial Basis Function Neural Network (RBF-NN) is introduced to obtain the approximation of gradient p, q Assume the same pixel (x, y) during the rotation. The integration ofalong x direction results in the height distribution. RBF NN I 1 (x, y) I 2 (x, y) nxnx nznz

13/20 Initial Vector for Optimization (2) How to make dataset of RBF-NN A sphere is used to make I 1 and I 2 using R(p, q), where, R is since a sphere has the whole combination of the surface gradient. How to use learned RBF-NN The corresponding point of the target object is assumed to be the same during the rotation.

14/20 Updating Equation using HF-NN The equation is iteratively used to optimize the energy function, that is, each partial difference becomes 0.

15/20 Iteration for Optimization The optimization is applied to each of two images repeatedly. The height z’with the rotation angle is given by Gradient are also calculated from the height repeatedly during rotation. C 1 is gradually reduces E 1 : the smoothness constraint Optimization is terminated the value of energy function converges in comparison with that of one step before.

16/20 Experiments (synthesis image) Rotation angle is 10° Image size is 64×64 pixels Rotation axis is along the center of the image. RBF-NN Learning Data : 2000 Learning Epoch : 15 MSE Maximum Height Theoretical Height Initial HeightRecovered Height Input Images

17/20 Experiments (SEM image) Rotation angle taken is 10 ° Rotation axis is set from the known feature points A and B MSE Theoretical Depth Theoretical HeightInitial HeightRecovered Height Relaxation Method Input Images

18/20 Experiments (SEM image) Rotation angle taken is 10 ° Rotation axis is set from the known feature points A and B Input Images Initial HeightRecovered Height

19/20 Conclusion A new method is proposed to recover the shape from SEM images. HF-NN is introduced to solve the optimization problem. The energy function is formulated from two image during rotation. The initial vector is obtained using RBF-NN.

20/20 Further works Getting more accurate result using more images Treatment of the inter-reflection Thank you