TGS Salt Identification Challenge —— Zheming Wang Course Work of CS539 in Fall 2018

Introduction Salt deposits below the surface is a big obstacle for oil and gas drillers. To detect them, expert human interpretation is still a must nowadays. The goal of this project is to tell the salt region with given seismic image as follow:

Data and method The data comes from the TGS challenge on Kaggle. The picture size is 101*101, while the train and test sets have sizes of 4,000 and 18,000. The tool kit I use is Keras with TensorFlow as backend in python. The code I used mainly referred to some kernels on Kaggle. I keep the metric and data transforming part unchanged, revised the model structure, and wrote some code to make the training resumable and automatic on Euler.

Basic Summary It is a typical image segmentation problem and there is a specific metric called Intersection over Union (IoU). In this project, I start from a baseline U-net model with IoU of 0.675(0.700) , ranking at 81%. After trying different methods and do lots of fine tuning, I get a 2-stage Res-Unet model with data augmentation, cross validation and voting, which obtains IoU of 0.839(0.858) , ranking at 10%.

Model Structure: U-net (baseline)

Model Structure: Runet-revised (No resize)

Result 1: model choosing description test score in leaderboard private public position U-net baseline 0.700 0.675 80.7% U-net - improved use greyscale 0.753 0.722 72.3% Res-U-net (Runet) add ResNet; add threshold 0.818 0.797 47.9% Runet-depth add depth as a channel 0.809 0.782 55.5% Runet-csum add cum-sum as a channel 0.813 0.793 50.6% Runet-revised keep input size 101*101 unchanged 0.829 0.800 46.3% General setting Batch size: 32 Epochs: 200 Early Stop : 10 Lr schedule : 1e-2/ 8/ 0.5/ 1e-5 Monitor : val_iou * May differ a little per model.

test score in leaderboard Result 2: fine-tuning Model description test score in leaderboard private public position Runet-lovasz 50 + 100 (baseline) 0.8323 0.8005 45.8% Runet-lovasz - tuned 100 + 200 0.8457 0.8171 32.7% cross validation 3 500 + 300 best-cv 0.8423 0.8180 31.9% combination - voting 0.8505 0.8264 21.1% cross validation 5-1 Augmentation; 300 + 300 best-cv 0.8456 0.8236 26.6% cross valiadation 5-2 Augmentation; 150 + 200 best-cv 0.8539 0.8334 14.0% 0.8581 0.8352 12.0% combination - best3 best 3 above 0.8582 0.8393 10.2%

Lessons Learned in Project 1. Keras has some problems when resume a saving model. After reading source code, I wrote a patch for it. https://github.com/wzmJimmy/MyKerasPatch 2. Parallelism did not accelerate single model training. I found this fact when trying Keras mult-gpu class. Then I changed to train multiple cross-validation like models at the same time. 3. In Data Science, the model choosing is a core problem, while the data preprocessing and hyper-parameter tuning can also make a great difference but need lots of experience and trails.

test score in leaderboard Further improvement Other techniques: Transfer learning: using other network as an encoder. Optimizer choosing: Adam, momentum, normal SGD. Feature engineering: tell zero-or-some salt first. …… Model description test score in leaderboard private public position model with X-ception as encoder, different optimizer, tricks like pseudo-labeling and Stochastic weight averaging. 0.8771 0.8547 4.2%

Thank you for listening.