Use 3D Convolutional Neural Network to Inspect Solder Ball Defects Presenter：Bing-Jhang Lin 林炳彰 Phone：0979258928 Email：lknight8631@gmail.com

Outline Introduction Methods Experiments

Introduction Head-in-Pillow (HIP) is a solder ball defect. HIP pattern https://alphaassembly.com/Utility/Blog/2012/October/Head-in-Pillow-An-Easily-Missed-and-Costly-Soldering-Defect

Introduction The HIP defect can be caused by the following reasons: Surface of solder ball oxidation Poor wetting of the solder Distortion of the Printed Circuit Board (PCB) by the heat of the soldering process 表面氧化 錫料濕潤不良 因加熱而變形

Introduction The current diagnosis of the HIP defects usually uses 2D X- Ray inspection machine, but it is hard to find out the position of the defect. http://stevezeva.homestead.com/Head-in-pillow_X-Ray.JPG http://ih.constantcontact.com/fs163/1108958702593/img/143.jpg?a=1113547242057

Introduction Some reliable methods are destructive tests, so they are not recommended for use.

Introduction Although it is hard to find out the HIP location from the 2D X-Ray image, we can reconstruct the 3D PCB model by different angles of the 2D X-Ray images.

Introduction The 3D PCB model not only provides us with more detailed information but also represents the location of HIP defects more clearly. In recent years, Deep Learning has been widely used in many computer vision tasks, especially the Convolutional Neural Network (CNN) has an outstanding performance in object recognition.

Introduction In this project, we propose a 3D CNN to inspect the HIP defects. Our network combines the advantages of many state-of-art CNNs like: VGG, GoogleNet Inception, ResNet, and DenseNet. We also design two types of the convolutional blocks and use the dense connectivity method to connect these blocks.

Introduction Because the HIP problems do not happen often, so we use the data augmentation method to expand our 3D solder ball data. In our experiments, our network can achieve excellent results on testing data; the size of our network is more refined; and the training and execution time is faster than other CNNs.

Methods 3D Solder Balls Volumetric Data 3D Convolutional Neural Network

3D Solder Balls Volumetric Data The 3D PCB volumetric data are reconstructed by different angles of 2D X-Rays images. In order to obtain clearer 3D PCB volumetric data during the reconstruction, we control the camera to take a 2D X- Rays image every 2.81 degrees.

3D Solder Balls Volumetric Data

3D Solder Balls Volumetric Data The 3D PCB volumetric data size is 1472 × 1176 × 46 pixels. Since the volumetric pattern of solder ball is an elliptical sphere, the middle slice of the volumetric contains the biggest solder ball cross section. Elliptical: 橢圓的 Sphere: 球

3D Solder Balls Volumetric Data The maximum bounding box size of the solder ball from the middle slice of the PCB volumetric data is 46 × 46 pixels, so the size of each 3D solder ball is 46 × 46 × 46 pixels.

3D Solder Balls Volumetric Data We totally received 277 3D solder balls.

3D Solder Balls Volumetric Data For an image object recognition task, the amount of data is not sufficient, so we also use the Data Augmentation method to obtain more data.

3D Solder Balls Volumetric Data We rotate the 3D solder ball data six times to obtain more data.

Methods Middle slice pattern in different plane. z y x xz Plane yz Plane xy Plane Diagonal (Right Rear to Left Front) Diagonal (Left Rear to Right Front) 3D Solder Ball

3D Convolutional Neural Network Our network: Input Convolution 3×3×3 Stride: 2, ReLU: True Dense Connection Block Num of Connection: 2 Convolution 1×1×1 Stride: 2, ReLU: True Global Average Pooling (GAP) × 4 Softmax Dropout

3D Convolutional Neural Network Our network contains four dense connection blocks. The block can fully utilize the features of each layer and avoid some important features lost in deep layers. Because the data may in different size, we add the Global Average Pooling (GAP) layer before the last layer to let our network can adapt different sizes of input.

3D Convolutional Neural Network Dense Connection Blocks: Input Convolution Block (Inception) Concatenate ReLU Convolution Block (Inception) Concatenate ReLU

3D Convolutional Neural Network We use the dense connectivity method to connect our convolutional blocks. The output of previous layer will be connected with the current output of the convolutional block and do the ReLU activation function. The features of the input will be fully used in the deep network and avoid some important information loss in the deep network.

3D Convolutional Neural Network Inception Block: Convolution 3×3×3 Stride: 1 Input Tensor Max Pooling 3×3×3 Stride: 1 Convolution 1×1×1 Stride: 1 Concatenate Average Pooling 3×3×3 Stride: 1 Convolution 1×1×1 Stride: 1

3D Convolutional Neural Network Inception with Residual Block: Convolution 3×3×3 Stride: 1 Max Pooling 3×3×3 Stride: 1 Convolution 1×1×1 Stride: 1 Concatenate Input Tensor Avg Pooling 3×3×3 Stride: 1 Convolution 1×1×1 Stride: 1 Convolution 1×1×1 Stride: 1 ReLU Convolution 3×3×3 Stride: 1 ReLU Convolution 1×1×1 Stride: 1

3D Convolutional Neural Network The convolutional blocks are composed of many different convolutional paths. The different convolutional paths allow the following layer to learn various features, so the network can choose better features to adapt the data. The residual method is a good way to prevent gradient vanishing problem, so we can more easily increase the depth of the network.

Experiments Solder Ball Data Experiments

Solder Ball Data We currently have 277 balls of 3D solder, of which 38 are labeled data, and the rest are unlabeled data. The volumetric size of each solder ball is 46 × 46 × 46 × 1 pixels. (The last channel represents the grayscale channel.) Half of labeled data (19) are defective solder balls, and the other are normal solder balls.

Solder Ball Data We split the labeled data into two parts: training and testing. The training part has 32 solder balls and the testing part has 6 solder balls. We rotate to expand the labeled data 6 times, so the training part has 192 (=32 × 6) solder balls, and the testing part has 36 (=6 × 6) solder balls.

Solder Ball Data Because the defective solder balls do not often appear on the PCB, so that the 239 unlabeled data are temporarily considered as normal solder balls in evaluating stage.

Experiments The information of each models.

Experiments The Result on testing data.

Experiments The Result on unlabeled data.

Experiments When the patterns of these solder balls are skewed near PCB border, we find that the models may misclassify the normal solder balls into defect. The skew situation often happens on the PCB border, and the pattern of skew solder ball is blurry which may cause misclassification.

Experiments