From classical filters to convolutional neural networks: the mathematical foundations and evolution of neural style transfer in images

Articles in Press

Document Type : Promotional Paper

Author

Computer Science Dept. Faculty of Mathematical Sciences, Ferdowsi University of Mashhad

Abstract

Neural Style Transfer (NST) stands as a remarkable example of the synthesis of mathematics, statistics, deep learning, and art. In this article, we present a historical and analytical exploration of the mathematical evolution of image-processing tools---from classical and derivative-based filters to frequency-domain transforms and multiscale representations---and trace how these concepts ultimately paved the way for convolutional neural networks, which form the foundation for understanding style transfer. We show how these mathematical developments culminated in the formulation of the NST algorithm. We then examine the mathematical structure of style transfer with a focus on the role of Gram matrices, learned feature spaces, and the optimization-based objective function. A central insight of NST is that its loss function becomes meaningful only through the representational power of deep neural networks: the notions of ``style'' and ``content'' are defined in learned feature spaces rather than in pixel space. Grounded in statistical concepts and solved through classical optimization methods, this formulation illustrates the deep and elegant interplay between applied mathematics, statistics, and deep learning in creating an artistic computational technique.

Keywords

Main Subjects

References

[1] M. Amintoosi, M. Fathy and N. Mozayani, Regional varying image super-resolution, In: IEEE international joint conference on computational sciences and optimization, 1, (2009), 913–917.
[2] M. Amintoosi and F. Farbiz, Eigenbackground revisited: can we model the background with eigenvectors?, J. Math. Imaging Vision, 64 (2022), no. 5, 463–477.
[3] M. Amintoosi, Application of Taylor expansion in reducing the size of convolutional neural networks for classifying Impressionism and Miniature paintings, Math. Soc., 5 (2020), no. 1, 1–16. [In Persian]. https://math-sci.ui.ac.ir/article_25351.html.
[4] M. Amintoosi, Fully connected to fully convolutional: road to yesterday, Journal of soft computing and information
technology, 11 (2022), no. 1, 60–72. [In Persian]. https://jscit.nit.ac.ir/article_149453.
[5] M. Amintoosi, Style transfer for data augmentation in convolutional neural networks applied to fire detection, Computational intelligence in electrical engineering, 13 (2022), no. 4, 97–114. [In Persian]. https://isee.ui.ac.ir/article_26042.html.
[6] T. Y. Chiu and D. Gurari, PCA-based knowledge distillation towards lightweight and content-style balanced photorealistic style transfer models, In: 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Los alamitos, CA, USA: IEEE Computer Society, (2022), 7834–7843, https: //doi.ieeecomputersociety.org/10.1109/CVPR52688.2022.00769.
[7] Y. Chen, Y. K. Lai and Y. J. Liu, CartoonGAN: generative adversarial networks for photo cartoonization, In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), (2018), 9465–9474. https://doi.org/10.1109/CVPR.2018.00986.
[8] J. Chung, S. Hyun and J. P. Heo, Style injection in diffusion: a training-free approach for adapting largescale diffusion models for style transfer, In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), (2024), 8795–8805.
[9] L. A. Gatys , A. S. Ecker and M. Bethge, Image style transfer using convolutional neural networks, In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), (2016), 2414–2423.
[10] L. A. Gatys, A. S. Ecker and M. Bethge, Texture synthesis using convolutional neural networks, In: Proceedings of the 29th international conference on neural information processing systems, 1, NIPS’15, Cambridge, MA, USA: MIT Press; (2015), 262–270.
[11] K. He, X. Zhang, S. Ren and J. Sun, Deep residual learning for image recognition, In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), (2016).
[12] X. Huang and S. Belongie, Arbitrary style Transfer in real-time with adaptive instance normalization, In: Proceedings of the IEEE international conference on computer vision (ICCV), (2017), 1510–1519. https://doi.org/10.1109/ICCV.2017.167.
[13] A. Hertzmann, Can computers create art?, Arts, 7 (2018), no. 2, pp. 25. https://www.mdpi.com/2076-0752/7/2/18.
[14] Y. Jing, Y. Yang, Z. Feng, J. Ye, Y. Yu and M. Song, Neural style transfer: a review, IEEE Transactions on Visualization & Computer Graphics, 26 (2020) , no. 11, 3365–3385. https://doi.org/10.48550/arXiv.1705.04058.
[15] Y. Jing, Y. Yang, Z. Feng, J. Ye, Y. Yu and M. Song, Neural style transfer: a review, IEEE transactions on visualization and computer graphics, 26 (2020), no. 11, 3365–3385. https://doi.org/10.1109/TVCG.
[19] Y. T. Kao, H. J. Lin, K. J. Lin and Y. Tokuyama, Content-preserving image style transfer via reversible networks with meta ActNorm, Electronics, 15 (2026), no. 2, 395. https://doi.org/10.3390/electronics15020395.
[20] X. Li, S. Liu, J. Kautz and M. H. Yang, Learning linear transformations for fast image and video style transfer, In: IEEE/CVF conference on computer vision and pattern recognition (CVPR), (2019), 3804–3812.
[21] S. Li, X. Xu, L. Nie and T. S. Chua, Laplacian-steered neural style transfer, In: Proceedings of the 26th international conference on world wide web (WWW), (2017), 1123–1132. https://doi.org/10.1145/3038912.3052617.
[22] J. Redmon and A. Farhadi, YOLO9000: Better, Faster, Stronger, CVPR, 2017.
[23] K. Simonyan and A. Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition, In: International conference on learning representations, (2015).
[24] A. Singh, V. Jaiswal, G. Joshi, A. Sanjeeve, S. Gite and K. Kotecha, Neural style transfer: a critical review, IEEE Access, 9 (2021) ,60297–60311. https://doi.org/10.1109/ACCESS.2021.3112996.
[25] A. Vidhya, The world through the eyes of CNN, 2020. Accessed: (2025). https://medium.com/analytics-vidhya/the-world-through-the-eyes-of-cnn-5a52c034dbeb.
[26] X. Wang and J. Jiang, Illustration image style transfer method design based on improved cyclic consistent adversarial network, PLOS ONE, 20 (2025), no. 1, e0313113.
[27] J. Yoo, Y. Uh, S. Chun, B.Kang and J. W. Ha, Photorealistic style transfer via wavelet transforms, In: IEEE/CVF international conference on computer vision (ICCV), (2019), 9035–9044.
[28] M. D. Zeiler and R. Fergus, Visualizing and understanding convolutional networks, In: European conference on computer vision (ECCV), Springer, (2014), 818–833.
[29] Y. Zheng, C. Yang and A. Merkulov, Breast cancer screening using convolutional neural network and follow-up digital mammography, In: A. Mahalanobis, A. Ashok, L. Tian and J. C. Petruccelli, editors, Conference: Computational Imaging III, 10669, International society for optics and photonics, SPIE, (2018). https://doi.org/10.1117/12.2304564.
[30] Z. Zou, T. Shi, S. Qiu, Y. Yuan and Z. Shi, Stylized neural painting, In: 2021 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Los Alamitos, CA, USA: IEEE Computer Society, (2021), 15684–15693. https://doi.ieeecomputersociety.org/10.1109/CVPR46437.2021.01543.
Mathematics and Society

Articles in Press, Accepted Manuscript
Available Online from 20 June 2026

Files

History

  • Receive Date: 20 November 2025
  • Revise Date: 15 May 2026
  • Accept Date: 20 June 2026
  • Publish Date: 20 June 2026

Share

How to cite

Statistics