1. Kuan-Chuan Peng Tsuhan Chen
A Framework of Extracting Multi-scale Features Using Multiple Convolutional Neural Networks
Kuan-Chuan Peng
Tsuhan Chen

2. Introduction Breakthrough progress in object classification. cat dog
lion
tiger
O. Russakovsky et al. ImageNet large scale visual recognition challenge. arXiv: , 2014.
N. Murray et al. AVA: A Large-Scale Database for Aesthetic Visual Analysis. CVPR12.

3. Introduction Humans are interested in more than objects.
For example, aesthetic quality.
N. Murray et al. AVA: A Large-Scale Database for Aesthetic Visual Analysis. CVPR12.

4. How do machines describe images?
Examples by state-of-art algorithm:
“man in black shirt is playing guitar.”
“woman is holding bunch of bananas.”
A. Karpathy and F.-F. Li. Deep visual-semantic alignments for generating image descriptions. CVPR15.

5. How do machines describe images?
Examples by state-of-art algorithm:
“man in black shirt is playing guitar.”
“woman is holding bunch of bananas.”
A. Karpathy and F.-F. Li. Deep visual-semantic alignments for generating image descriptions. CVPR15.

6. How do machines describe images?
Examples by state-of-art algorithm:
“man in black shirt is playing guitar.”
“woman is holding bunch of bananas.”
A. Karpathy and F.-F. Li. Deep visual-semantic alignments for generating image descriptions. CVPR15.

7. How do machines describe images?
Examples by state-of-art algorithm:
“man in black shirt is playing guitar.”
“woman is holding bunch of bananas.”
A. Karpathy and F.-F. Li. Deep visual-semantic alignments for generating image descriptions. CVPR15.

8. How do experts describe images?
Examples by the Pulitzer Prize winners:
“At bath times, Danielle appears serene. But no one know what lies beyond those eyes.” (by Lane DeGregory)
“The surgery has dragged on for hours with little progress, and Mulliken, taking a breather next to an array of Sam's CAT scans, is feeling the frustration and exhaustion.” (by Tom Hallman Jr.)

9. How do experts describe images?
Images convey more than objects.
“At bath times, Danielle appears serene. But no one know what lies beyond those eyes.” (by Lane DeGregory)
“The surgery has dragged on for hours with little progress, and Mulliken, taking a breather next to an array of Sam's CAT scans, is feeling the frustration and exhaustion.” (by Tom Hallman Jr.)

10. Beyond Objects Abstract attributes matter.
Attributes relating to or involving general ideas or qualities rather than specific people, objects, or actions. [Merriam-Webster dictionary]
Bridge the gap between machines and humans:
Teach machines to solve abstract tasks (tasks involving abstract attributes).

11. Goal
A general framework to achieve better performance in abstract tasks.
Multi-scale features by using convolutional neural networks (CNN).

12. Why CNN? speech recognition object classification video classification
O. Russakovsky et al. ImageNet large scale visual recognition challenge. arXiv: , 2014.
L. Deng et al. A deep convolutional neural network using heterogeneous pooling for trading acoustic invariance with phonetic confusion. ICASSP13.
A. Karpathy et al. Large-scale video classification with convolutional neural networks. CVPR14.

13. Existing Abstract Tasks
More and more abstract tasks are proposed.

14. Artistic Style & Artist Style Classification
[F. S. Khan et al. MVA14.]
Architectural Style Classification
[Z. Xu et al. ECCV14.]

15. Emotion Classification [J. Machajdik et al. ACMMM10.]
amusement
anger
awe
contentment
disgust
excitement
fear
sad
Emotion Classification
[J. Machajdik et al. ACMMM10.]
Aesthetic Classification
[N. Murray et al. CVPR12.]
high aesthetic quality
low aesthetic quality

16. Fashion Style Classification [M. H. Kiapour et al. ECCV14.]
Bohemian
Hipster
Fashion Style Classification
[M. H. Kiapour et al. ECCV14.]
Memorability Prediction
[P. Isola et al. CVPR11.]
Interestingness Prediction
[M. Gygli et al. ICCV13.]

17. Inspiration It is tricky to describe abstract attributes as objects.
Not easy to “locate” abstract attributes.
What if abstract attributes prevail everywhere?
Label-inheritable (LI) property.
contentment
[J. Machajdik et al. ACMMM10.] ?

18. Label-Inheritable (LI) Property
Dataset
Painting-91 [1]
arcDataset [2]
Caltech-101 [3]
Task
Artist style classification
Architectural style classification
Object classification
Label
Picasso
Baroque Architecture
Faces
Label-inheritable
Yes
Partial
Mostly No
[1] F. S. Khan et al. Painting-91: a large scale database for computational painting categorization. Machine Vision & Applications 14.
[2] Z. Xu et al. Architectural style classification using multinomial latent logistic regression. ECCV14.
[3] F.-F. Li et al. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. CVPRW04.

19. Label-Inheritable (LI) Property
Dataset
Painting-91 [1]
arcDataset [2]
Caltech-101 [3]
Task
Artist style classification
Architectural style classification
Object classification
Label
Picasso
Baroque Architecture
Faces
Label-inheritable
Yes
Partial
Mostly No
[1] F. S. Khan et al. Painting-91: a large scale database for computational painting categorization. Machine Vision & Applications 14.
[2] Z. Xu et al. Architectural style classification using multinomial latent logistic regression. ECCV14.
[3] F.-F. Li et al. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. CVPRW04.

20. Label-Inheritable (LI) Property
Dataset
Painting-91 [1]
arcDataset [2]
Caltech-101 [3]
Task
Artist style classification
Architectural style classification
Object classification
Label
Picasso
Baroque Architecture
Faces
Label-inheritable
Yes
Partial
Mostly No
[1] F. S. Khan et al. Painting-91: a large scale database for computational painting categorization. Machine Vision & Applications 14.
[2] Z. Xu et al. Architectural style classification using multinomial latent logistic regression. ECCV14.
[3] F.-F. Li et al. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. CVPRW04.

21. Multi-Scale CNN
Assume LI property holds for each image and the associated label.
A. Krizhevsky et al. ImageNet classification with deep convolutional neural networks. NIPS12.

22. AlexNet
The number of nodes in output layer is changed to be the number of classes in each task.
A. Krizhevsky et al. ImageNet classification with deep convolutional neural networks. NIPS12.

23. Experimental Results classification accuracy (%) Method \ Task
Artist style classification
Artistic style classification
Caltech-101 object classification
(15 / 30 training examples per class)
Architectural style classification
(10 / 25 classes)
Previous work
(baseline)
53.10 [1]
62.20 [1]
83.80 / [2]
69.17 / [3]
Single-scale CNN
55.15
67.37
83.45 / 88.19
70.64 / 54.84
2-scale CNN
(ours)
58.11
69.67
80.19 / 87.58
74.82 / 58.89
3-scale CNN
57.91
70.96
N/A
75.32 / 59.13
Label-inheritable
Yes
Mostly No
Partial
[1] F. S. Khan et al. Painting-91: a large scale database for computational painting categorization. Machine Vision & Applications 14.
[2] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. ECCV14.
[3] Z. Xu et al. Architectural style classification using multinomial latent logistic regression. ECCV14.

24. Is it because of more training data?
What if we train one CNN with images in different scales?
A. Krizhevsky et al. ImageNet classification with deep convolutional neural networks. NIPS12.

25. Additional Results classification accuracy (%) Method \ Task
Artist style classification
Artistic style classification
Caltech-101 object classification
(15 / 30 training examples per class)
Architectural style classification
(10 / 25 classes)
Previous work
(baseline)
53.10 [1]
62.20 [1]
83.80 / [2]
69.17 / [3]
Single-scale CNN
55.15
67.37
83.45 / 88.19
70.64 / 54.84
2-scale CNN
(ours)
58.11
69.67
80.19 / 87.58
74.82 / 58.89
1 CNN +
2-scale images
46.86
61.95
N / A
67.93 / 49.06
Label-inheritable
Yes
Mostly No
Partial
[1] F. S. Khan et al. Painting-91: a large scale database for computational painting categorization. Machine Vision & Applications 14.
[2] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. ECCV14.
[3] Z. Xu et al. Architectural style classification using multinomial latent logistic regression. ECCV14.

26. Conclusion
We proposed Multi-Scale Convolutional Neural Networks (MSCNN) based on Label-Inheritable (LI) property.
Multi-scale features.
MSCNN can outperform the state-of-art performance on datasets where LI property holds or even partially holds.

27. Towards Solving Abstract Tasks
More CNN features to achieve better performance in abstract tasks.
Multi-scale features (ICME15).
Multi-depth features (ICIP15).
Multi-task features (submitted to ICCV15).
K.-C. Peng and T. Chen. A Framework of extracting multi-scale features using multiple convolutional neural networks. ICME15.
K.-C. Peng and T. Chen. Cross-layer features in convolutional neural networks for generic classification tasks. ICIP15.
K.-C. Peng and T. Chen. Toward correlating and solving abstract tasks using convolutional neural networks. Submitted to ICCV15.

28. Q & A