基于多尺度特征损失函数的图像超分辨率重建

徐亮, 符冉迪, 金炜, 等. 基于多尺度特征损失函数的图像超分辨率重建[J]. 光电工程, 2019, 46(11): 180419. doi: 10.12086/oee.2019.180419
引用本文: 徐亮, 符冉迪, 金炜, 等. 基于多尺度特征损失函数的图像超分辨率重建[J]. 光电工程, 2019, 46(11): 180419. doi: 10.12086/oee.2019.180419
Xu Liang, Fu Randi, Jin Wei, et al. Image super-resolution reconstruction based on multi-scale feature loss function[J]. Opto-Electronic Engineering, 2019, 46(11): 180419. doi: 10.12086/oee.2019.180419
Citation: Xu Liang, Fu Randi, Jin Wei, et al. Image super-resolution reconstruction based on multi-scale feature loss function[J]. Opto-Electronic Engineering, 2019, 46(11): 180419. doi: 10.12086/oee.2019.180419

基于多尺度特征损失函数的图像超分辨率重建

  • 基金项目:
    国家自然科学基金资助项目(61471212);浙江省自然科学基金资助项目(LY16F010001)
详细信息
    作者简介:
    通讯作者: 符冉迪(1971-),男,副教授,硕士生导师,主要从事压缩感知和数字图像处理等的研究。E-mail: 17346807657@163.com
  • 中图分类号: TP391.41

Image super-resolution reconstruction based on multi-scale feature loss function

  • Fund Project: Supported by National Natural Science Foundation of China (61471212) and Zhejiang Province Natural Science Fund (LY16F010001)
More Information
  • 在图像超分辨率重建问题中,许多基于深度学习的方法大多采用传统的均方误差(MSE)作为损失函数,重建后的图像容易出现细节模糊和过于平滑的问题。针对这一问题,本文对传统的均方误差损失函数进行改进,提出一种基于多尺度特征损失函数的图像超分辨率重建方法。整个网络模型由基于DenseNet的重建模型和一个用来优化多尺度特征损失函数的卷积神经网络串联构成。将重建后得到的图像和对应的原始高清图像作为串联的卷积神经网络的输入,计算重建图像卷积得到的不同尺度特征图与对应的原始高清图像卷积得到的不同尺度特征图的均方误差。实验结果表明,本文提出的方法在主观视觉效果和PSRN、SSIM上均有所提升。

  • Overview: In recent years, with the research and development of deep learning, it has been widely used in image processing. Compared with the traditional shallow learning, which can only extract the features of images simply, deep learning can learn the deeper feature representation, so as to have better performance in image processing. The traditional mean squared error (MSE) as the loss function is mostly adopted in the image super-resolution based on deep learning to obtain better PSNR, such as SRCNN, FSRCNN, and SRDenseNet. However, the reconstructed image is prone to edge blur and may be too smooth. The multi-scale loss function proposed in this paper can improve this problem. Based on the analysis of SRCNN, FSRCNN, SRDenseNet, and other methods, the reconstruction model was built with the DenseNet model as the basic framework, and a three-layer convolutional neural network was connected after the reconstruction model to calculate the multi-scale feature loss function. The reconstruction model consists of four parts: dense connection block, dimension reduction layer, deconvolution layer, and reconstruction layer. Each dense connection block is composed of 4 convolution layers, and 3×3 convolution kernel is adopted. The number of feature maps output by each dense connection block is 256. Since the output of all dense connection blocks is concatenated, the feature map is reduced to 256 by means of 1×1 convolution kernel in the dimension reduction layer to reduce the computational burden. After the deconvolution layer, a single channel image is reconstructed by 3×3 convolution kernel. At last, the reconstructed image and the corresponding original HD image were extracted by the three-layer convolution neural network in series, and the difference between the reconstructed image and the original HD image was compared by calculating the mean square error. This article uses Yang91 and BSD200 dataset that consists of 291 images. Considering that the training of convolution neural network depends on a large number of data samples, the original 291 data sets are extended to ensure sufficient samples. First, the original sample set was flipped from left to right and from top to bottom, and the training sample set was 4 times more than the original one, obtaining 291+(291×4)=1455 training samples. Then, the original sample size is enlarged by 2, 3, and 4 times, respectively, with further 180° mirror transformation. After that, 291×2×3=1746 training samples were obtained, with total samples 1455+1746=3201. Set5, Set14 and BSD100 were selected as the standard evaluation dataset in the field of super-resolution research for the test samples, and objective indicators were evaluated using peak signal to noise ratio (PSNR) and structural similarity (SSIM). The experimental results show that the details of the reconstructed images become richer and the edge blur is improved.

  • 加载中
  • 图 1  SRCNN结构图[13]

    Figure 1.  Structure of SRCNN[13]

    图 2  FSRCNN结构图[20]

    Figure 2.  Structure of FSRCNN[20]

    图 3  SRDenseNet结构图[22]

    Figure 3.  Structure of SRDenseNet[22]

    图 4  基于多尺度特征损失函数的图像超分辨率重建模型

    Figure 4.  Image super-resolution reconstruction model based on multi-scale feature loss function

    图 5  基于DenseNet的重建模型以及传统均方误差作为损失函数的重建效果与本文方法的对比

    Figure 5.  The reconstructed results based on the DenseNet reconstruction model and the traditional mean square error as the loss function are compared with the method in this paper

    图 6  Butterfly放大2倍时各算法重建效果。(a)原图;(b) Bicubic;(c) SRCNN;(d) DnCNN-3;(e)本文方法

    Figure 6.  Reconstruction results of each algorithm for butterfly amplification by 2 times. (a) Original; (b) Bicubic; (c) SRCNN; (d) DnCNN-3; (e) Our method

    图 7  Zebra放大3倍时各算法重建效果。(a)原图;(b) Bicubic;(c) SRCNN;(d) DnCNN-3;(e)本文方法

    Figure 7.  Reconstruction results of each algorithm for zebra amplification by 3 times. (a) Original; (b) Bicubic; (c) SRCNN; (d) DnCNN-3; (e) Our method

    图 8  PPT放大4倍时各算法重建效果。(a)原图;(b) Bicubic;(c) SRCNN;(d) DnCNN-3;(e)本文方法

    Figure 8.  Reconstruction results of each algorithm for PPT amplification by 4 times. (a) Original; (b) Bicubic; (c) SRCNN; (d) DnCNN-3; (e) Our method

    图 9  本文方法在训练集和测试集上的损失函数曲线

    Figure 9.  The loss function curves of our method in training set and test set

    表 1  不同超分辨率算法在数据集Set5、Set14、BSD100上的平均PSNR以及SSIM

    Table 1.  Average PSNR and SSIM of different super-resolution algorithms on datasets Set5, Set14 and BSD100

    Dataset Scale Bicubic SRCNN DnCNN-3 Based on DenseNet Our method
    Set5 33.66/0.9299 36.66/0.9542 37.58/0.9590 37.79/0.9589 37.98/0.9632
    30.39/0.8682 32.75/0.9090 33.75/0.9222 33.99/0.9256 34.36/0.9264
    28.42/0.8104 30.48/0.8628 31.40/0.8845 30.93/0.8754 31.32/0.8819
    Set14 30.24/0.8688 32.42/0.9063 33.03/0.9128 33.48/0.9145 33.69/0.9175
    27.55/0.7742 29.28/0.8209 29.81/0.8321 29.60/0.8281 29.69/0.8296
    26.00/0.7072 27.49/0.7503 28.04/0.7672 28.29/0.7689 28.56/0.7714
    BSD100 29.56/0.8431 31.36/0.8879 31.90/0.8961 32.19/0.8988 32.73/0.9023
    27.21/0.7385 28.41/0.7863 28.85/0.7981 29.18/0.8001 29.43/0.8103
    25.96/0.6675 26.90/0.7101 27.29/0.7253 27.03/0.7203 27.24/0.7232
    下载: 导出CSV
  • [1]

    Glasner D, Bagon S, Irani M. Super-resolution from a single image[C]//Proceedings of the IEEE 12th International Conference on Computer Vision, Kyoto, Japan, 2009: 349-356.

    [2]

    Peled S, Yeshurun Y. Superresolution in MRI: application to human white matter fiber tract visualization by diffusion tensor imaging[J]. Magnetic Resonance in Medicine, 2001, 45(1): 29-35. doi: 10.1002/1522-2594(200101)45:1<29::AID-MRM1005>3.0.CO;2-Z

    [3]

    Shi W Z, Caballero J, Ledig C, et al. Cardiac image super-resolution with global correspondence using multi-atlas patchmatch[C]//Proceedings of the 16th International Conference on Medical Image Computing and Computer-Assisted Intervention, Nagoya, Japan, 2013: 9-16.

    [4]

    Gunturk B K, Batur A U, Altunbasak Y, et al. Eigenface-domain super-resolution for face recognition[J]. IEEE Transactions on Image Processing, 2003, 12(5): 597-606. doi: 10.1109/TIP.2003.811513

    [5]

    Zhang L P, Zhang H Y, Shen H F, et al. A super-resolution reconstruction algorithm for surveillance images[J]. Signal Processing, 2010, 90(3): 848-859. doi: 10.1016/j.sigpro.2009.09.002

    [6]

    Zhou F, Yang W M, Liao Q M. Interpolation-based image super-resolution using multisurface fitting[J]. IEEE Transactions on Image Processing, 2012, 21(7): 3312-3318. doi: 10.1109/TIP.2012.2189576

    [7]

    Zhang L, Wu X L. An edge-guided image interpolation algorithm via directional filtering and data fusion[J]. IEEE Transactions on Image Processing, 2006, 15(8): 2226-2238. doi: 10.1109/TIP.2006.877407

    [8]

    Lin Z C, Shum H Y. Fundamental limits of reconstruction-based superresolution algorithms under local translation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(1): 83-97. doi: 10.1109/TPAMI.2004.1261081

    [9]

    Rasti P, Demirel H, Anbarjafari G. Image resolution enhancement by using interpolation followed by iterative back projection[C]//Proceedings of the 21st Signal Processing and Communications Applications Conference, Haspolat, Turkey, 2013: 1-4.

    [10]

    周靖鸿, 周璀, 朱建军, 等.基于非下采样轮廓波变换遥感影像超分辨重建方法[J].光学学报, 2015, 35(1): 0110001. doi: 10.3788/AOS201535.0110001

    Zhou J H, Zhou C, Zhu J J, et al. A method of super-resolution reconstruction for remote sensing image based on non-subsampled contourlet transform[J]. Acta Optica Sinica, 2015, 35(1): 0110001. doi: 10.3788/AOS201535.0110001

    [11]

    Yang J C, Wright J, Huang T S, et al. Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19(11): 2861-2873. doi: 10.1109/TIP.2010.2050625

    [12]

    Timofte R, De V, van Gool L. Anchored neighborhood regression for fast example-based super-resolution[C]//Proceedings of 2013 IEEE International Conference on Computer Vision, Sydney, NSW, Australia, 2013: 1920-1927.

    [13]

    Dong C, Loy C C, He K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307. doi: 10.1109/TPAMI.2015.2439281

    [14]

    苏衡, 周杰, 张志浩.超分辨率图像重建方法综述[J].自动化学报, 2013, 39(8): 1202-1213. doi: 10.3724/SP.J.1004.2013.01202

    Su H, Zhou J, Zhang Z H. Survey of super-resolution image reconstruction methods[J]. Acta Automatica Sinica, 2013, 39(8): 1202-1213. doi: 10.3724/SP.J.1004.2013.01202

    [15]

    Keys R. Cubic convolution interpolation for digital image processing[J]. IEEE Transactions on Acoustics Speech & Signal Processing, 1981, 29(6): 1153-1160. doi: 10.1109/TASSP.1981.1163711

    [16]

    袁琪, 荆树旭.改进的序列图像超分辨率重建方法[J].计算机应用, 2009, 29(12): 3310-3313. http://d.old.wanfangdata.com.cn/Periodical/jsjyy200912042

    Yuan Q, Jing S X. Improved super resolution reconstruction method for video sequence[J]. Journal of Computer Applications, 2009, 29(12): 3310-3313. http://d.old.wanfangdata.com.cn/Periodical/jsjyy200912042

    [17]

    Chang H, Yeung D Y, Xiong Y. Super-resolution through neighbor embedding[C]//Proceedings of the 2004 Computer Vision and Pattern Recognition.Piscataway, NJ: IEEE, 2004, 1: I-I.

    [18]

    Yang J C, Wright J, Huang T S, et al. Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19(11): 2861-2873. doi: 10.1109/TIP.2010.2050625

    [19]

    Yang J, Wright J, Huang T, et al. Image super-resolution as sparse representation of raw image patches[C]//Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2008: 1-8.

    [20]

    Dong C, Loy C C, Tang X O. Accelerating the super-resolution convolutional neural network[C]//Proceedings of the 14th European Conference on Computer Vision, Amsterdam, The Netherlands, 2016.

    [21]

    Kim J W, Lee J K, Lee K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016: 1646-1654.

    [22]

    Tong T, Li G, Liu X J, et al. Image super-resolution using dense skip connections[C]//Proceedings of 2017 IEEE International Conference on Computer Vision, Venice, Italy, 2017.

    [23]

    Yamanaka J, Kuwashima S, Kurita T. Fast and accurate image super resolution by deep CNN with skip connection and network in network[C]//Proceedings of the 24th International Conference on Neural Information Processing, Guangzhou, China, 2017.

    [24]

    He K M, Zhang X Y, Ren S Q, et al. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification[C]//Proceedings of 2015 IEEE International Conference on Computer Vision, Santiago, Chile, 2015: 1026-1034.

    [25]

    Huang G, Liu Z, van der Maaten L, et al. Densely connected convolutional networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Hawaii, 2017.

  • 加载中

(9)

(1)

计量
  • 文章访问数:  9846
  • PDF下载数:  3007
  • 施引文献:  0
出版历程
收稿日期:  2018-08-04
修回日期:  2019-01-04
刊出日期:  2019-11-01

目录

/

返回文章
返回