Light field image super-resolution network based on angular difference enhancement

Lv Tianqi; Wu Yingchun; Zhao Xianling

doi:10.12086/oee.2023.220185

Article navigation > Opto-Electronic Engineering > 2023 Vol. 50 > No. 2 > 220185

Next Article Previous Article

Lv T Q, Wu Y C, Zhao X L. Light field image super-resolution network based on angular difference enhancement[J]. Opto-Electron Eng, 2023, 50(2): 220185. doi: 10.12086/oee.2023.220185

Citation:

Lv T Q, Wu Y C, Zhao X L. Light field image super-resolution network based on angular difference enhancement[J]. Opto-Electron Eng, 2023, 50(2): 220185. doi: 10.12086/oee.2023.220185

Light field image super-resolution network based on angular difference enhancement

School of Electronic and Information Engineering, Taiyuan University of Science and Technology, Taiyuan, Shanxi 030024, China

Fund Project: National Natural Science Foundation of China (61601318), the Basic Research Project of Shanxi Province (202103021224278), and Research Project Supported by Shanxi Scholarship Council of China (2020-128)

More Information

^*Corresponding author: yingchunwu3030@foxmail.com

Received Date 28 July 2022

Revised Date 17 October 2022

Accepted Date 21 October 2022

Available Online 16 February 2023

Published Date 25 February 2023

Abstract

Abstract

Based on the advanced imaging technology, light field camera can obtain the spatial information and the angular information of the scene synchronously. It achieves higher dimensional scene representation by sacrificing the spatial resolution. In order to improve the spatial resolution of the light field image, a light field super-resolution reconstruction network based on angle difference enhancement is built in this paper. In the proposed network, eight multi-branch residual blocks are used to extract shallow features. Then, four enhanced angular deformable alignment modules are used to extract deep features. Finally six simplified residual feature distillation modules and pixel shuffle modules are used to complete data reconstruction. The proposed network takes advantage of the angle difference of the light field to complete the spatial information super-resolution. In order to obtain more features difference between different views, the own feature of the single view is emphasized during the feature extraction. The performance of the proposed network is verified on five public light field data sets. The proposed algorithm obtains high-resolution light field sub-aperture images with higher PSNR and SSIM.
- light field camera /
- super-resolution reconstruction /
- residual /
- deformable convolution /
- residual feature distillation

FullText(HTML)

References

[1]	Ng R, Levoy M, Brédif M, et al. Light field photography with a hand-held plenoptic camera[J]. Comput Sci Tech Rep CSTR, 2005, 2(11): 1−11. Google Scholar
[2]	Tao M W, Hadap S, Malik J, et al. Depth from combining defocus and correspondence using light-field Cameras[C]//Proceedings of 2013 IEEE International Conference on Computer Vision, Sydney, 2013: 673–680. https://doi.org/10.1109/ICCV.2013.89. Google Scholar
[3]	Levoy M, Hanrahan P. Light field rendering[C]//Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, New York, 1996: 31–42. https://doi.org/10.1145/237170.237199. Google Scholar
[4]	Wang Y Q, Wang L G, Yang J G, et al. Spatial-angular interaction for light field image super-resolution[C]//Proceedings of the 16th European Conference on Computer Vision, Glasgow, 2020: 290–308. https://doi.org/10.1007/978-3-030-58592-1_18. Google Scholar
[5]	Wang Y Q, Yang J G, Wang L G, et al. Light field image super-resolution using deformable convolution[J]. IEEE Trans Image Process, 2020, 30: 1057−1071. doi: 10.1109/TIP.2020.3042059 CrossRef Google Scholar
[6]	Mo Y, Wang Y Q, Xiao C, et al. Dense dual-attention network for light field image super-resolution[J]. IEEE Trans Circuits Syst Video Technol, 2022, 32(7): 4431−4443. doi: 10.1109/TCSVT.2021.3121679 CrossRef Google Scholar
[7]	Yeung H W F, Hou J H, Chen X M, et al. Light field spatial super-resolution using deep efficient spatial-angular separable convolution[J]. IEEE Trans Image Process, 2019, 28(5): 2319−2330. doi: 10.1109/TIP.2018.2885236 CrossRef Google Scholar
[8]	Zhang S, Lin Y F, Sheng H. Residual networks for light field image super-resolution[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, 2019: 11046–11055. https://doi.org/10.1109/CVPR.2019.01130. Google Scholar
[9]	Jin J, Hou J H, Chen J, et al. Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, 2020: 2260−2269. https://doi.org/10.1109/CVPR42600.2020.00233. Google Scholar
[10]	Liang Z Y, Wang Y Q, Wang L G, et al. Angular-flexible network for light field image super-resolution[J]. Electron Lett, 2021, 57(24): 921−924. doi: 10.1049/ell2.12312 CrossRef Google Scholar
[11]	赵圆圆, 施圣贤. 融合多尺度特征的光场图像超分辨率方法[J]. 光电工程, 2020, 47(12): 200007. doi: 10.12086/oee.2020.200007 CrossRef Google Scholar Zhao Y Y, Shi S X. Light-field image super-resolution based on multi-scale feature fusion[J]. Opto-Electron Eng, 2020, 47(12): 200007. doi: 10.12086/oee.2020.200007 CrossRef Google Scholar
[12]	Liu J, Tang J, Wu G S. Residual feature distillation network for lightweight image super-resolution[C]//Proceedings of the European Conference on Computer Vision, Glasgow, 2020: 41–55. https://doi.org/10.1007/978-3-030-67070-2_2. Google Scholar
[13]	Rerabek M, Ebrahimi T. New light field image dataset[C]//Proceedings of the 8th International Conference on Quality of Multimedia Experience, Lisbon, 2016. Google Scholar
[14]	Honauer K, Johannsen O, Kondermann D, et al. A dataset and evaluation methodology for depth estimation on 4D light fields[C]//Proceedings of the 13th Asian Conference on Computer Vision, Cham, 2016: 19–34. https://doi.org/10.1007/978-3-319-54187-7_2. Google Scholar
[15]	Wanner S, Meister S, Goldluecke B. Datasets and benchmarks for densely sampled 4D light fields[M]//Bronstein M, Favre J, Hormann K. Vision, Modeling and Visualization. Eurographics Association, 2013: 225–226. https://doi.org/10.2312/PE.VMV.VMV13.225-226. Google Scholar
[16]	Le Pendu M, Jiang X R, Guillemot C. Light field inpainting propagation via low rank matrix completion[J]. IEEE Trans Image Process, 2018, 27(4): 1981−1993. doi: 10.1109/TIP.2018.2791864 CrossRef Google Scholar
[17]	Vaish V, Adams A. The (new) Stanford light field archive, computer graphics laboratory, Stanford University[EB/OL]. 2008. http://lightfield.stanford.edu Google Scholar
[18]	Kingma D P, Ba J. Adam: a method for stochastic optimization[C]//Proceedings of the 3rd International Conference on Learning Representations, San Diego, 2015. Google Scholar
[19]	He K M, Zhang X Y, Ren S Q, et al. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification[C]//Proceedings of 2015 International Conference on Computer Vision, Santiago, 2015: 1026–1034. https://doi.org/10.1109/ICCV.2015.123. Google Scholar
[20]	Lim B, Son S, Kim H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, 2017: 136−144. https://doi.org/10.1109/CVPRW.2017.151. Google Scholar
[21]	Zhang Y L, Li K P, Li K, et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the 15th European Conference on Computer Vision, Munich, 2018: 294–310. https://doi.org/10.1007/978-3-030-01234-2_18. Google Scholar

Overview

Overview

Based on advanced imaging technology, light field cameras can obtain the spatial information and the angular information of the scene synchronously. It achieves higher dimensional scene representation by sacrificing spatial resolution. To improve the spatial resolution of the light field image, a light field super-resolution reconstruction network based on angle difference enhancement is built in this paper. In the proposed network, eight multi-branch residual blocks are used to extract shallow features. Then, four enhanced angular deformable alignment modules are used to extract deep features. Finally, six simplified residual feature distillation modules and pixel shuffle modules are used to complete data reconstruction. The proposed network takes advantage of the angle difference of the light field to complete the spatial information super-resolution. To obtain more features difference between different views, the own feature of the single view is emphasized during the feature extraction. The design ideas of different modules are as follows.

a) Shallow feature extraction based on multi-branch residual (MBR) block. The convolution blocks of 1×1 are used to complete the channel map first, then eight multi-branch residual blocks are used to extract shallow features. By using the skip connection and atrous convolution in the multi-branch residual blocks, the network degradation is suppressed, more hierarchical information is extracted and the receptive field of the feature is expanded.

b) Deep feature extraction based on enhanced angular deformable alignment (EADA) block. To take full advantage of the angle information correlation of multi-view images in the 4D light field, the ADA block is replaced by the EADA block to complete the feature collection and feature distribution. Compared with the ADA block, EADA adds a branch in front of the two DCB blocks in the feature collection. Two atrous convolutions are added in branch 1, and one ordinary convolution and one atrous convolution are added in branch 2. In the feature distribution, one ordinary convolution and two atrous convolutions are added before DCB blocks. By using the EADA block, the own feature of each side-view is fully extracted, and the more accurate disparity is obtained. So the super-resolution performance of the network can be improved.

c) Data reconstruction based on the simplified residual feature distillation (SRFD) block. The RFD block is adopted in the proposed light field super-resolution network to complete the data reconstruction. To reduce the parameters of the network, the RFD block is simplified to obtain SRFD the block by omitting the contrast-aware channel attention (CCA) layer. A convolution and Relu activation are added at the front of the SRFD block, so the input feature is refined; In the shallow residual block, the activation function is swapped with convolution. In this way, the feature of the current layer is fed directly into the next distillation block without activation, which makes the shallow information and the deep information get a more direct combination.

The experiments are carried out from the aspects of validity verification of each network module, comparison of subjective visual effects, comparison of quantitative evaluation results, and algorithm complexity. The performance of the proposed network is verified on five public light field data sets. The proposed algorithm obtains high-resolution light field sub-aperture images with higher PSNR and SSIM.