1. Do Better ImageNet Models Transfer Better?
CVPR 2019 Oral Session 1-2C: Scenes & Representation
Simon Kornblith, Jonathon Shlens, and Quoc V. Le
Google Brain
The last author is PhD Student of Andrew Ng.

2. Do better ImageNet models transfer better?
Motivating Question
Do better ImageNet models transfer better?
How is the transferability of both ImageNet features and ImageNet classification architectures?
A large-scale empirical study that systematically explores the problem.

3. Evaluation
Datasets
Networks
Metrics
Settings

4. 12 Datasets

5. 16 Models

6. Metrics
accuracy: the meaning of a 1% additive increase in accuracy is different if it is relative to a base accuracy of 50% vs. 99%.
logit-transformed accuracy:
correlation: PLCC between the logit-transformed ImageNet accuracy and the logit-transformed transfer accuracy averaged across the 12 datasets

7. 3 Settings
training a logistic regression classifier on the fixed feature representation from the ImageNet-pretrained network
fine-tuning the ImageNet-pretrained network
training the same CNN architecture from scratch on the new image task.

8. The First Setting
training a logistic regression classifier on the fixed feature representation from the ImageNet-pretrained network
fine-tuning the ImageNet-pretrained network
training the same CNN architecture from scratch on the new image task.

9. Results
… when using an ImageNet-pretrained network as fixed feature extractor
Green ones, not statistically different
p-value 0.05
Permutation test for comparison on the same dataset
t-test for comparison cross datasets
better ImageNet architectures are capable of learning better, transferable representations.

10. ImageNet training settings affect transfer of fixed features

11. ImageNet training settings affect transfer of fixed features
Some widely-used regularizers that improve ImageNet performance do not produce better representations.
Low dimensional embeddings of Oxford 102 Flowers using t-SNE
on features from Inception v4, for 10 classes from the test set.

12. The Second Setting
training a logistic regression classifier on the fixed feature representation from the ImageNet-pretrained network
fine-tuning the ImageNet-pretrained network
training the same CNN architecture from scratch on the new image task.

13. Results
… when fine-tuning an ImageNet-pretrained network

14. ImageNet training settings have only a minor impact on fine-tuning performance

15. The Third Setting
training a logistic regression classifier on the fixed feature representation from the ImageNet-pretrained network
fine-tuning the ImageNet-pretrained network
training the same CNN architecture from scratch on the new image task.

16. Results … when training from scratch
whether ImageNet accuracy for transfer learning is
due to the weights derived from the ImageNet training or the architecture itself.
7 datasets (samples<10000) r=0.29
Other datasets r=0.86

17. Other Analysis
Benefits of better models are comparable to specialized methods for transfer learning
ImageNet pretraining does not necessarily improve accuracy on fine- grained tasks
ImageNet pretraining accelerates convergence
Accuracy benefits of ImageNet pretraining fade quickly with dataset size

18. Benefits of better models are comparable to specialized methods for transfer learning

19. ImageNet pretraining does not necessarily improve accuracy on fine-grained tasks

20. ImageNet pretraining accelerates convergence

21. Accuracy benefits of ImageNet pretraining fade quickly with dataset size

22. Conclusion and Comment
A large-scale empirical study concludes that
better ImageNet networks provide better features for transfer learning with linear classification, and better performance when the entire network is fine- tuned
some regularizers that improve ImageNet performance are highly detrimental to the performance of transfer learning based on fixed features
architectures transfer well across tasks even when weights do not
This kind of research, i.e., systematic and deep analysis of the existing research, sometimes is even more beneficial to the research community than ``simply proposing a novel method``.
Further readings
Recht, Benjamin, et al. "Do ImageNet Classifiers Generalize to ImageNet?." International Conference on Machine Learning
On two small fine-grained classification datasets, fine-tuning does not provide a substantial benefit over training from random initialization, but better ImageNet architectures nonetheless obtain higher accuracy.