Do Better ImageNet Models Transfer Better? CVPR 2019 Oral Session 1-2C: Scenes & Representation Simon Kornblith, Jonathon Shlens, and Quoc V. Le Google Brain The last author is PhD Student of Andrew Ng.

Do better ImageNet models transfer better? Motivating Question Do better ImageNet models transfer better? How is the transferability of both ImageNet features and ImageNet classification architectures? A large-scale empirical study that systematically explores the problem.

Evaluation Datasets Networks Metrics Settings

12 Datasets

16 Models

Metrics accuracy: the meaning of a 1% additive increase in accuracy is different if it is relative to a base accuracy of 50% vs. 99%. logit-transformed accuracy: correlation: PLCC between the logit-transformed ImageNet accuracy and the logit-transformed transfer accuracy averaged across the 12 datasets

3 Settings training a logistic regression classifier on the fixed feature representation from the ImageNet-pretrained network fine-tuning the ImageNet-pretrained network training the same CNN architecture from scratch on the new image task.

The First Setting training a logistic regression classifier on the fixed feature representation from the ImageNet-pretrained network fine-tuning the ImageNet-pretrained network training the same CNN architecture from scratch on the new image task.

Results … when using an ImageNet-pretrained network as fixed feature extractor Green ones, not statistically different p-value 0.05 Permutation test for comparison on the same dataset t-test for comparison cross datasets better ImageNet architectures are capable of learning better, transferable representations.

ImageNet training settings affect transfer of fixed features

ImageNet training settings affect transfer of fixed features Some widely-used regularizers that improve ImageNet performance do not produce better representations. Low dimensional embeddings of Oxford 102 Flowers using t-SNE on features from Inception v4, for 10 classes from the test set.

The Second Setting training a logistic regression classifier on the fixed feature representation from the ImageNet-pretrained network fine-tuning the ImageNet-pretrained network training the same CNN architecture from scratch on the new image task.

Results … when fine-tuning an ImageNet-pretrained network

ImageNet training settings have only a minor impact on fine-tuning performance

The Third Setting training a logistic regression classifier on the fixed feature representation from the ImageNet-pretrained network fine-tuning the ImageNet-pretrained network training the same CNN architecture from scratch on the new image task.

Results … when training from scratch whether ImageNet accuracy for transfer learning is due to the weights derived from the ImageNet training or the architecture itself. 7 datasets (samples<10000) r=0.29 Other datasets r=0.86

Other Analysis Benefits of better models are comparable to specialized methods for transfer learning ImageNet pretraining does not necessarily improve accuracy on fine- grained tasks ImageNet pretraining accelerates convergence Accuracy benefits of ImageNet pretraining fade quickly with dataset size

Benefits of better models are comparable to specialized methods for transfer learning

ImageNet pretraining does not necessarily improve accuracy on fine-grained tasks

ImageNet pretraining accelerates convergence

Accuracy benefits of ImageNet pretraining fade quickly with dataset size

Conclusion and Comment A large-scale empirical study concludes that better ImageNet networks provide better features for transfer learning with linear classification, and better performance when the entire network is fine- tuned some regularizers that improve ImageNet performance are highly detrimental to the performance of transfer learning based on fixed features architectures transfer well across tasks even when weights do not This kind of research, i.e., systematic and deep analysis of the existing research, sometimes is even more beneficial to the research community than ``simply proposing a novel method``. Further readings Recht, Benjamin, et al. "Do ImageNet Classifiers Generalize to ImageNet?." International Conference on Machine Learning. 2019. On two small fine-grained classification datasets, fine-tuning does not provide a substantial benefit over training from random initialization, but better ImageNet architectures nonetheless obtain higher accuracy.