Application of Taylor expansion in reducing the size of convolutional neural networks for classifying Impressionism and Miniature paintings

Document Type : Applied Paper

Author

Faculty of Mathematics and Computer Science, Hakim Sabzevari University, Sabzevar

Abstract

Taylor expansion is one of the methods for approximating functions that are differentiable of any order. The main paradigm in neural networks learning is based on deriving from the objective function and using gradient descent to achieve optimal solutions. Convolutional neural networks are among the most important tools in the field of deep learning. Most of these networks involve models with large sizes, and reducing the volume of these models is a current research topic. The primary method of reducing the size of models is pruning the redundant connections of neural networks, which are usually based on the size of connection weights. One of these methods is using Taylor expansion of the objective function to prioritize connections for removal from the network. In this paper, this method is extensively discussed, and a new application of it in classifying paintings with impressionism and miniature styles is presented. The experimental results show that with the Taylor expansion-based method, 83% of the network connections can be selected and removed without compromising the accuracy of the model in this specific application.

Keywords

Main Subjects


[1] L. Jiao, F. Zhang, F. Liu, S. Yang, L. Li, Z. Feng, and R. Qu, “A survey of deep learning-based object detection,” IEEE Access, vol.7, pp.128837–128868, 2019.
[2] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic seg-mentation,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp.580–587, 2014.
[3] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” IEEE Transactions on Pattern Analysis and Machine Intel-ligence, vol.42, no.2, pp.386–397, 2020.
[4] J. Redmon and A. Farhadi, “Yolo9000: Better, faster, stronger,” CVPR, 2017.
[5] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in International Con-ference on Learning Representations, 2015.
[6] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[7] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems 25 (F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, eds. ), pp.1097–
1105, Curran Associates, Inc., 2012.
[8] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Improving neural networks by preventing co-adaptation of feature detectors,” CoRR, vol.abs/1207.0580, 2012.
[9] L. Wan, M. Zeiler, S. Zhang, Y. L. Cun, and R. Fergus, “Regularization of neural networks using dropconnect,” in Proceedings of the 30th International Conference on Machine Learning (S. Dasgupta and D. McAllester, eds. ), vol.28 of Proceedings of Machine Learning Research, (Atlanta, Georgia, USA), pp.1058–1066, PMLR, 17–19 Jun 2013.
[10] H. Wu and X. Gu, “Towards dropout training for convolutional neural networks,” Neural Networks, vol.71, pp.1 – 10, 2015.
[11] B. Liu, M. Wang, H. Foroosh, M. F. Tappen, and M. Pensky, “Sparse convolutional neural networks.,” in CVPR, pp.806–814, IEEE Computer Society, 2015.
[12] X. Chen, “Escort: Efficient sparse convolutional neural networks on gpus,” CoRR, vol.abs/1802.10280, 2018.
[13] W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li, “Learning structured sparsity in deep neural networks,” in Advances in Neural Information Processing Systems 29 (D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, eds. ), pp.2074–2082, Curran Associates, Inc., 2016.
[14] K. Mitsuno, J. Miyao, and T. Kurita, “Hierarchical group sparse regularization for deep convolutional neural networks,” 2020.
[15] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, “Pruning convolutional neural networks for resource efficient infer-ence,” in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Confer-ence Track Proceedings, OpenReview.net, 2017.
[16] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Neurocomputing: Foundations of research,” chap. Learning Representa-tions by Back-propagating Errors, pp.696–699, Cambridge, MA, USA: MIT Press, 1988.