Super-resolution reconstruction of infrared image based on channel attention and transfer learning

Sun Rui; Zhang Han; Cheng Zhikang; Zhang Xudong

doi:10.12086/oee.2021.200045

Article navigation > Opto-Electronic Engineering > 2021 Vol. 48 > No. 1 > 200045

Next Article Previous Article

Sun R, Zhang H, Cheng Z K, et al. Super-resolution reconstruction of infrared image based on channel attention and transfer learning[J]. Opto-Electron Eng, 2021, 48(1): 200045. doi: 10.12086/oee.2021.200045

Citation:

Sun R, Zhang H, Cheng Z K, et al. Super-resolution reconstruction of infrared image based on channel attention and transfer learning[J]. Opto-Electron Eng, 2021, 48(1): 200045. doi: 10.12086/oee.2021.200045

Super-resolution reconstruction of infrared image based on channel attention and transfer learning

1.
School of Computer and Information, Hefei University of Technology, Hefei, Anhui 230009, China
2.
Anhui Province Key Laboratory of Industry Safety and Emergency Technology, Hefei, Anhui 230009, China

Fund Project: National Natural Science Foundation of China (61471154, 61876057) and the Fundamental Research Funds for Central Universities (JZ2018YYPY0287)

More Information

^*Corresponding author: Zhang Han, E-mail: 1402198087@qq.com

Received Date 11 February 2020

Revised Date 09 May 2020

Published Date 15 January 2021

Abstract

Abstract

A super-resolution reconstruction method of infrared images based on channel attention and transfer learning was proposed to solve the problems of low resolution and low quality of infrared images. In this method, a deep convolutional neural network is designed to enhance the learning ability of the network by introducing the channel attention mechanism, and the residual learning method is used to mitigate the problem of gradient explosion or disappearance and to accelerate the convergence of the network. Because high-quality infrared images are difficult to collect and insufficient in number, so this method is divided into two steps: the first step is to use natural images to pre-train the neural network model, and the second step is to use transfer learning knowledge to fine-tune the pre-trained model's parameters with a small number of high-quality infrared images to make the model better in reconstructing the infrared image. Finally, a multi-scale detail boosting filter is added to improve the visual effect of the reconstructed infrared image. Experiments on Set5 and Set14 datasets as well as infrared images show that the deepening network depth and introducing channel attention mechanism can improve the effect of super-resolution reconstruction, transfer learning can well solve the problem of insufficient number of infrared image samples, and multi-scale detail boosting filter can improve the details and increase the amount of information of the reconstruction image.
- super-resolution /
- infrared image /
- convolutional neural network /
- attention /
- transfer learning

FullText(HTML)

References

[1]	苏衡, 周杰, 张志浩. 超分辨率图像重建方法综述[J]. 自动化学报, 2013, 39(8): 1202–1213. Google Scholar Su H, Zhou J, Zhang Z H. Survey of super-resolution image reconstruction methods[J]. Acta Autom Sin, 2013, 39(8): 1202–1213. Google Scholar
[2]	Bätz M, Eichenseer A, Seiler J, et al. Hybrid super-resolution combining example-based single-image and interpolation-based multi-image reconstruction approaches[C]//Proceedings of 2015 IEEE International Conference on Image Processing (ICIP), 2015: 58–62. Google Scholar
[3]	Kim K I, Kwon Y. Single-image super-resolution using sparse regression and natural image prior[J]. IEEE Trans Pattern Anal Mach Intell, 2010, 32(6): 1127–1133. doi: 10.1109/TPAMI.2010.25 CrossRef Google Scholar
[4]	练秋生, 张伟. 基于图像块分类稀疏表示的超分辨率重构算法[J]. 电子学报, 2012, 40(5): 920–925. Google Scholar Lian Q S, Zhang W. Image super-resolution algorithms based on sparse representation of classified image patches[J]. Acta Electron Sin, 2012, 40(5): 920–925. Google Scholar
[5]	肖进胜, 刘恩雨, 朱力, 等. 改进的基于卷积神经网络的图像超分辨率算法[J]. 光学学报, 2017, 37(3): 0318011. Google Scholar Xiao J S, Liu E Y, Zhu L, et al. Improved image super-resolution algorithm based on convolutional neural network[J]. Acta Opt Sin, 2017, 37(3): 0318011. Google Scholar
[6]	Stark H, Oskoui P. High-resolution image recovery from image-plane arrays, using convex projections[J]. J Opt Soc Am A, 1989, 6(11): 1715–1726. doi: 10.1364/JOSAA.6.001715 CrossRef Google Scholar
[7]	Irani M, Peleg S. Improving resolution by image registration[J]. CVGIP: Graph Models Image Process, 1991, 53(3): 231–239. doi: 10.1016/1049-9652(91)90045-L CrossRef Google Scholar
[8]	Chang H, Yeung D Y, Xiong Y M. Super-resolution through neighbor embedding[C]//Proceedings of 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004: 275–282. Google Scholar
[9]	Yang J C, Wright J, Huang T S, et al. Image super-resolution via sparse representation[J]. IEEE Trans Image Process, 2010, 19(11): 2861–2873. doi: 10.1109/TIP.2010.2050625 CrossRef Google Scholar
[10]	Dong C, Loy C C, He K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Trans Pattern Anal Mach Intell, 2016, 38(2): 295–307. doi: 10.1109/TPAMI.2015.2439281 CrossRef Google Scholar
[11]	Dong C, Loy C C, Tang X O. Accelerating the super-resolution convolutional neural network[C]//Proceedings of the 14th European Conference on Computer Vision, 2016: 391–407. Google Scholar
[12]	Shi W Z, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1874–1883. Google Scholar
[13]	Ledig C, Theis L, Huszár F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4681–4690. Google Scholar
[14]	Pan S J, Yang Q. A survey on transfer learning[J]. IEEE Trans Knowl Data Eng, 2010, 22(10): 1345–1359. doi: 10.1109/TKDE.2009.191 CrossRef Google Scholar
[15]	徐舟, 曲长文, 何令琪. 基于迁移学习的SAR目标超分辨重建[J]. 航空学报, 2015, 36(6): 1940–1952. Google Scholar Xu Z, Qu C W, He L Q. SAR target super-resolution based on transfer learning[J]. Acta Aeronaut Astronaut Sin, 2015, 36(6): 1940–1952. Google Scholar
[16]	Yanai K, Kawano Y. Food image recognition using deep convolutional network with pre–training and fine–tuning[C]//Proceedings of 2015 IEEE International Conference on Multimedia & Expo Workshops, 2015: 1–6. Google Scholar
[17]	Du B, Xiong W, Wu J, et al. Stacked convolutional denoising auto-encoders for feature representation[J]. IEEE Trans Cybern, 2017, 47(4): 1017–1027. doi: 10.1109/TCYB.2016.2536638 CrossRef Google Scholar
[18]	Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 7132–7141. Google Scholar
[19]	Kim Y, Koh Y J, Lee C, et al. Dark image enhancement based onpairwise target contrast and multi-scale detail boosting[C]//Proceedings of 2015 IEEE International Conference on Image Processing, 2015: 1404–1408. Google Scholar
[20]	Kingma D P, Ba J. Adam: a method for stochastic optimization[Z]. arXiv: 1412.6980, 2014. Google Scholar
[21]	Mittal A, Soundararajan R, Bovik A C. Making a "completely blind" image quality analyzer[J]. IEEE Signal Process Lett, 2013, 20(3): 209–212. doi: 10.1109/LSP.2012.2227726 CrossRef Google Scholar
[22]	邵雪, 曾台英, 汪祖辉. 一种基于NIQE的印刷图像无参考质量评价方法[J]. 包装学报, 2016, 8(4): 35–39. doi: 10.3969/j.issn.1674-7100.2016.04.007 CrossRef Google Scholar Shao X, Zeng T Y, Wang Z H. No-reference quality assessment method for printed image based on NIQE[J]. Packaging J, 2016, 8(4): 35–39. doi: 10.3969/j.issn.1674-7100.2016.04.007 CrossRef Google Scholar

Overview

Overview

Overview: In recent years, infrared imaging technology has developed rapidly and has been increasingly used in military reconnaissance, security surveillance, and medical imaging. However, in the process of infrared image imaging or transmission, it is affected by many factors such as environment and equipment. The infrared image often has a low resolution, which greatly reduces the amount of information contained in the infrared image and restricts the application value of the infrared image. Therefore, how to obtain high-resolution and high-information infrared images has become an issue that people urgently need to solve. In recent years, the development of deep learning technology has made rapid progress, and super-resolution methods based on deep learning have begun to appear. However, if these convolutional neural networks are directly applied to the infrared image field, there are some problems: SRCNN, FSRCNN, and ESPCN have fewer network convolutional layers and insufficient network depth, and the learning features will be relatively single, ignoring the differences between image features. The mutual relationship makes it difficult to extract the deep-level information of the infrared image, and SRGAN may generate super-resolution images that are not close to the original image in certain details, which is not conducive to the application of infrared images in military, medical and surveillance. Another problem that needs to be overcome is that it is difficult to collect a sufficient number of high-quality infrared images in real life, and a large number of images of different scenes and targets are required as training samples for common deep learning methods. The infrared images used as training data sets to achieve deep learning methods often fail to achieve the desired effect. In order to solve these problems, this paper proposes a method for super-resolution reconstruction of infrared images based on channel attention and transfer learning. This method first designs a deep convolutional neural network, which integrates the channel attention mechanism to learn the correlation between the channels in the feature space, enhances the learning ability of the network, and uses residual learning to reduce the problem of gradient explosion or disappearance and to speed up network convergence. Then, considering that high-quality infrared images are difficult to collect and insufficient in number, the network training is divided into two steps: the first step uses natural images to pre-train a super-resolution model of natural images, and the second step is to use transfer learning knowledge. Using a small number of high-quality infrared images, the pre-trained model parameters are quickly transferred and fine-tuned to improve the reconstruction effect of the model on the infrared image, thereby obtaining a super-resolution model of the infrared image. Finally, a multi-scale detail boosting (MSDB) module is added to enhance the details and visual effects of the infrared reconstructed image and to increase the amount of information.