In recent years, infrared imaging technology has developed rapidly and has been increasingly used in military reconnaissance, security surveillance, and medical imaging. However, in the process of infrared image imaging or transmission, it is affected by many factors such as environment and equipment. The infrared image often has a low resolution, which greatly reduces the amount of information contained in the infrared image and restricts the application value of the infrared image. Therefore, how to obtain high-resolution and high-information infrared images has become an issue that people urgently need to solve. In recent years, the development of deep learning technology has made rapid progress, and super-resolution methods based on deep learning have begun to appear. However, if A super-resolution reconstruction method of infrared images based on channel attention and transfer learning was proposed to solve the problems of low resolution and low quality of infrared images. In this method, a deep convolutional neural network is designed to enhance the learning ability of the network by introducing the channel attention mechanism, and the residual learning method is used to mitigate the problem of gradient explosion or disappearance and to accelerate the convergence of the network. Because high-quality infrared images are difficult to collect and insufficient in number, so this method is divided into two steps: the first step is to use natural images to pre-train the neural network model, and the second step is to use transfer learning knowledge to fine-tune the pre-trained model’s parameters with a small number of high-quality infrared images to make the model better in reconstructing the infrared image. Finally, a multi-scale detail boosting filter is added to improve the visual effect of the reconstructed infrared image. Experiments on Set5 and Set14 datasets as well as infrared images show that the deepening network depth and introducing channel attention mechanism can improve the effect of super-resolution reconstruction, transfer learning can well solve the problem of insufficient number of infrared image samples, and multi-scale detail boosting filter can improve the details and increase the amount of information of the reconstruction image.these convolutional neural networks are directly applied to the infrared image field, there are some problems: SRCNN, FSRCNN, and ESPCN have fewer network convolutional layers and insufficient network depth, and the learning features will be relatively single, ignoring the differences between image features. The mutual relationship makes it difficult to extract the deep-level information of the infrared image, and SRGAN may generate super-resolution images that are not close to the original image in certain details, which is not conducive to the application of infrared images in military, medical and surveillance. Another problem that needs to be overcome is that it is difficult to collect a sufficient number of high-quality infrared images in real life, and a large number of images of different scenes and targets are required as training samples for common deep learning methods. The infrared images used as training data sets to achieve deep learning methods often fail to achieve the desired effect. In order to solve these problems, this paper proposes a method for super-resolution reconstruction of infrared images based on channel attention and transfer learning. This method first designs a deep convolutional neural network, which integrates the channel attention mechanism to learn the correlation between the channels in the feature space, enhances the learning ability of the network, and uses residual learning to reduce the problem of gradient explosion or disappearance and to speed up network convergence. Then, considering that high-quality infrared images are difficult to collect and insufficient in number, the network training is divided into two steps: the first step uses natural images to pre-train a super-resolution model of natural images, and the second step is to use transfer learning knowledge. Using a small number of high-quality infrared images, the pre-trained model pa-rameters are quickly transferred and fine-tuned to improve the reconstruction effect of the model on the infrared image, thereby obtaining a super-resolution model of the infrared image. Finally, a multi-scale detail boosting (MSDB) module is added to enhance the details and visual effects of the infrared reconstructed image and to increase the amount of in-formation.
Super-resolution reconstruction of infrared image based on channel attention and transfer learning
First published at:Jan 15, 2021
 Su H, Zhou J, Zhang Z H. Survey of super-resolution image reconstruction methods[J]. Acta Autom Sin, 2013, 39(8): 1202–1213.
苏衡, 周杰, 张志浩. 超分辨率图像重建方法综述[J]. 自动化学报, 2013, 39(8): 1202–1213.
 Bätz M, Eichenseer A, Seiler J, et al. Hybrid super-resolution combining example-based single-image and interpolation-based multi-image reconstruction approaches[C]//Proceedings of 2015 IEEE International Conference on Image Processing (ICIP), 2015: 58–62.
 Kim K I, Kwon Y. Single-image super-resolution using sparse regression and natural image prior[J]. IEEE Trans Pattern Anal Mach Intell, 2010, 32(6): 1127–1133.
 Lian Q S, Zhang W. Image super-resolution algorithms based on sparse representation of classified image patches[J]. Acta Electron Sin, 2012, 40(5): 920–925.
练秋生, 张伟. 基于图像块分类稀疏表示的超分辨率重构算法[J]. 电子学报, 2012, 40(5): 920–925.
 Xiao J S, Liu E Y, Zhu L, et al. Improved image super-resolution algorithm based on convolutional neural network[J]. Acta Opt Sin, 2017, 37(3): 0318011.
肖进胜, 刘恩雨, 朱力, 等. 改进的基于卷积神经网络的图像超分辨率算法[J]. 光学学报, 2017, 37(3): 0318011.
 Stark H, Oskoui P. High-resolution image recovery from image-plane arrays, using convex projections[J]. J Opt Soc Am A, 1989, 6(11): 1715–1726.
 Irani M, Peleg S. Improving resolution by image registration[J]. CVGIP: Graph Models Image Process, 1991, 53(3): 231–239.
 Chang H, Yeung D Y, Xiong Y M. Super-resolution through neighbor embedding[C]//Proceedings of 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004: 275–282.
 Yang J C, Wright J, Huang T S, et al. Image super-resolution via sparse representation[J]. IEEE Trans Image Process, 2010, 19(11): 2861–2873.
 Dong C, Loy C C, He K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Trans Pattern Anal Mach Intell, 2016, 38(2): 295–307.
 Dong C, Loy C C, Tang X O. Accelerating the super-resolution convolutional neural network[C]//Proceedings of the 14th European Conference on Computer Vision, 2016: 391–407.
 Shi W Z, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1874–1883.
 Ledig C, Theis L, Huszár F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4681–4690.
 Pan S J, Yang Q. A survey on transfer learning[J]. IEEE Trans Knowl Data Eng, 2010, 22(10): 1345–1359.
 Xu Z, Qu C W, He L Q. SAR target super-resolution based on transfer learning[J]. Acta Aeronaut Astronaut Sin, 2015, 36(6): 1940–1952.
徐舟, 曲长文, 何令琪. 基于迁移学习的SAR目标超分辨重建[J]. 航空学报, 2015, 36(6): 1940–1952.
 Yanai K, Kawano Y. Food image recognition using deep convolutional network with pre–training and fine–tuning[C]//Proceedings of 2015 IEEE International Conference on Multimedia & Expo Workshops, 2015: 1–6.
 Du B, Xiong W, Wu J, et al. Stacked convolutional denoising auto-encoders for feature representation[J]. IEEE Trans Cybern, 2017, 47(4): 1017–1027.
 Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 7132–7141.
 Kim Y, Koh Y J, Lee C, et al. Dark image enhancement based onpairwise target contrast and multi-scale detail boosting[C]//Proceedings of 2015 IEEE International Conference on Image Processing, 2015: 1404–1408.
 Kingma D P, Ba J. Adam: a method for stochastic optimization[Z]. arXiv: 1412.6980, 2014.
 Mittal A, Soundararajan R, Bovik A C. Making a “completely blind” image quality analyzer[J]. IEEE Signal Process Lett, 2013, 20(3): 209–212.
 Shao X, Zeng T Y, Wang Z H. No-reference quality assessment method for printed image based on NIQE[J]. Packaging J, 2016, 8(4): 35–39.
邵雪, 曾台英, 汪祖辉. 一种基于NIQE的印刷图像无参考质量评价方法[J]. 包装学报, 2016, 8(4): 35–39.
National Natural Science Foundation of China (61471154, 61876057) and the Fundamental Research Funds for Central Universities (JZ2018YYPY0287)
Get Citation: Sun Rui, Zhang Han, Cheng Zhikang, et al. Super-resolution reconstruction of infrared image based on channel attention and transfer learning[J]. Opto-Electronic Engineering, 2021, 48(1): 200045.