In order to obtain better super-resolution reconstruction results of depth images, this paper constructs a multi-scale color image guidance depth image super-resolution reconstruction convolutional neural network. In this paper, the multi-scale fusion method is used to realize the guidance of high resolution (HR) color image features to low resolution (LR) depth image features, which is beneficial to the restoration of image details. In the process of extracting features from LR depth images, a multiple receptive field residual block (MRFRB) is constructed to extract and fuse the features of different receptive fields, and then connect and fuse the features of each MRFRB output to obtain global fusion features. Finally, the HR depth image is obtained through sub-pixel convolution layer and global fusion features. The experimental results show that the super-resolution image obtained by this method alleviates the edge distortion and artifact problems, and has better visual effects.
Color image multi-scale guided depth image super-resolution reconstruction
First published at:Apr 13, 2020
1 Palacios J M, Sagüés C, Montijano E, et al. Human-computer interaction based on hand gestures using RGB-D sensors[J]. Sensors, 2013, 13(9): 11842-11860. DOI:10.3390/s130911842
2 Nguyen T N, Huynh H H, Meunier J. 3D reconstruction with time-of-flight depth camera and multiple mirrors[J]. IEEE Access, 2018, 6: 38106-38114. DOI:10.1109/ACCESS.2018.2854262
3 Yamamoto S. Development of inspection robot for nuclear power plant[C]//Proceedings of 1992 IEEE International Conference on Robotics and Automation, Nice, France, 1992: 1559-1566.
4 Kolb A, Barth E, Koch R, et al. Time-of-flight cameras in computer graphics[J]. Computer Graphics Forum, 2010, 29(1): 141-159.
5 Xie J, Feris R S, Yu S S, et al. Joint super resolution and denoising from a single depth image[J]. IEEE Transactions on Multimedia, 2015, 17(9): 1525-1537. DOI:10.1109/TMM.2015.2457678
6 Mandal S, Bhavsar A, Sao A K. Noise adaptive super-resolution from single image via non-local mean and sparse representation[J]. Signal Processing, 2017, 132: 134-149. DOI:10.1016/j.sigpro.2016.09.017
7 Aodha O M, Campbell N D F, Nair A, et al. Patch based synthesis for single depth image super-resolution[C]//Proceedings of the 12th European Conference on Computer Vision, Florence, Italy, 2012: 71-84.
8 Li J, Lu Z C, Zeng G, et al. Similarity-aware patchwork assembly for depth image super-resolution[C]//Proceedings of 2014 IEEEConference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 2014: 3374-3381.
9 Xie J, Feris R S, Sun M T. Edge-guided single depth image super resolution[J]. IEEE Transactions on Image Processing, 2016, 25(1): 428-438.
10 Chen B L, Jung C. Single depth image super-resolution using convolutional neural networks[C]//Proceedings of2018 IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, AB, Canada, 2018: 1473-1477.
11 Liu W, Chen X G, Yang J, et al. Robust color guided depth map restoration[J]. IEEE Transactions on Image Processing, 2017, 26(1): 315-327. DOI:10.1109/TIP.2016.2612826
12 Kiechle M, Hawe S, Kleinsteuber M. A joint intensity and depth co-sparse analysis model for depth map super-resolution[C]//Proceedings of 2013 International Conference on Computer Vision, Sydney, NSW, Australia, 2013: 1545-1552.
13 Li Y, Min D B, Do M N, et al. Fast guided global interpolation for depth and motion[C]//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands, 2016: 717-733.
14 Park J, Kim H, Tai Y W, et al. High quality depth map upsampling for 3D-tof cameras[C]//Proceedings of 2011 IEEE International Conference on Computer Vision, Barcelona, Spain, 2011: 1623-1630.
15 Li W, Zhang X D. Depth image super-resolution reconstruction based on convolution neural network[J]. Journal of Electronic Measurement and Instrumentation, 2017, 31(12): 1918-1928.
李伟, 张旭东.基于卷积神经网络的深度图像超分辨率重建方法[J].电子测量与仪器学报, 2017, 31(12): 1918-1928.
16 Xiao Y, Cao X, Zhu X Y, et al. Joint convolutional neural pyramid for depth map super-resolution[Z]. arXiv: 1801.00968, 2018.
17 Lecun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324. DOI:10.1109/5.726791
18 Scharstein D, Szeliski R, Zabih R. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms[C]//Proceedings of 2001 IEEE Workshop on Stereo and Multi-Baseline Vision, Kauai, HI, USA, 2001: 131-140.
19 Richardt C, Stoll C, Dodgson N A, et al. Coherent spatiotemporal filtering, upsampling and rendering of rgbz videos[J]. Computer Graphics Forum, 2012, 31(2): 247-256.
20 Lu S, Ren X F, Liu F. Depth enhancement via low-rank matrix completion[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 2014: 3390-3397.
21 Handa A, Whelan T, McDonald J, et al. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM[C]//Proceedings of 2014 IEEE International Conference on Robotics and Automation, Hong Kong, China, 2014: 1524-1531.
22 He K M, Zhang X Y, Ren S Q, et al. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification[C]//Proceedings of 2015 IEEE International Conference on Computer Vision, Santiago, Chile, 2015: 1026-1034.
23 Dong C, Loy C C, He K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307. DOI:10.1109/TPAMI.2015.2439281
24 Hui T W, Loy C C, Tang X O. Depth map super-resolution by deep multi-scale guidance[C]//Proceedings of the 14th European Conference on Computer Vision, Amsterdam, The Netherlands, 2016: 353-369.
25 Kim J, Lee J K, Lee K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016: 1646-1654.
26 Lai W S, Huang J B, Ahuja N, et al. Deep Laplacian pyramid networks for fast and accurate super-resolution[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017: 624-632.
National Natural Science Foundation of China (61876057)
Get Citation: Yu Shuxia, Hu Liangmei, Zhang Xudong, et al. Color image multi-scale guided depth image super-resolution reconstruction[J]. Opto-Electronic Engineering, 2020, 47(4): 190260.
Previous: Measurement of optical fiber geometry parameters by gray distribution fitting with Gaussian function
Next: Characteristic analysis of orbital angular momentum of vortex beam propagating in atmospheric turbulent