Light field image super-resolution network based on angular difference enhancement

Lv Tianqi; Wu Yingchun; Zhao Xianling

doi:10.12086/oee.2023.220185

Article navigation > Opto-Electronic Engineering > 2023 Vol. 50 > No. 2 > 220185

Next Article Previous Article

Lv T Q, Wu Y C, Zhao X L. Light field image super-resolution network based on angular difference enhancement[J]. Opto-Electron Eng, 2023, 50(2): 220185. doi: 10.12086/oee.2023.220185

Citation:

Lv T Q, Wu Y C, Zhao X L. Light field image super-resolution network based on angular difference enhancement[J]. Opto-Electron Eng, 2023, 50(2): 220185. doi: 10.12086/oee.2023.220185

Light field image super-resolution network based on angular difference enhancement

School of Electronic and Information Engineering, Taiyuan University of Science and Technology, Taiyuan, Shanxi 030024, China

Fund Project: National Natural Science Foundation of China (61601318), the Basic Research Project of Shanxi Province (202103021224278), and Research Project Supported by Shanxi Scholarship Council of China (2020-128)

More Information

Corresponding author: yingchunwu3030@foxmail.com

Received Date 28 July 2022

Revised Date 17 October 2022

Accepted Date 21 October 2022

Available Online 16 February 2023

Published Date 25 February 2023

Abstract

Abstract

Based on the advanced imaging technology, light field camera can obtain the spatial information and the angular information of the scene synchronously. It achieves higher dimensional scene representation by sacrificing the spatial resolution. In order to improve the spatial resolution of the light field image, a light field super-resolution reconstruction network based on angle difference enhancement is built in this paper. In the proposed network, eight multi-branch residual blocks are used to extract shallow features. Then, four enhanced angular deformable alignment modules are used to extract deep features. Finally six simplified residual feature distillation modules and pixel shuffle modules are used to complete data reconstruction. The proposed network takes advantage of the angle difference of the light field to complete the spatial information super-resolution. In order to obtain more features difference between different views, the own feature of the single view is emphasized during the feature extraction. The performance of the proposed network is verified on five public light field data sets. The proposed algorithm obtains high-resolution light field sub-aperture images with higher PSNR and SSIM.
- light field camera /
- super-resolution reconstruction /
- residual /
- deformable convolution /
- residual feature distillation

FullText(HTML)

References

[1]	Ng R, Levoy M, Brédif M, et al. Light field photography with a hand-held plenoptic camera[J]. Comput Sci Tech Rep CSTR, 2005, 2(11): 1−11. Google Scholar
[2]	Tao M W, Hadap S, Malik J, et al. Depth from combining defocus and correspondence using light-field Cameras[C]//Proceedings of 2013 IEEE International Conference on Computer Vision, Sydney, 2013: 673–680. https://doi.org/10.1109/ICCV.2013.89. Google Scholar
[3]	Levoy M, Hanrahan P. Light field rendering[C]//Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, New York, 1996: 31–42. https://doi.org/10.1145/237170.237199. Google Scholar
[4]	Wang Y Q, Wang L G, Yang J G, et al. Spatial-angular interaction for light field image super-resolution[C]//Proceedings of the 16th European Conference on Computer Vision, Glasgow, 2020: 290–308. https://doi.org/10.1007/978-3-030-58592-1_18. Google Scholar
[5]	Wang Y Q, Yang J G, Wang L G, et al. Light field image super-resolution using deformable convolution[J]. IEEE Trans Image Process, 2020, 30: 1057−1071. doi: 10.1109/TIP.2020.3042059 CrossRef Google Scholar
[6]	Mo Y, Wang Y Q, Xiao C, et al. Dense dual-attention network for light field image super-resolution[J]. IEEE Trans Circuits Syst Video Technol, 2022, 32(7): 4431−4443. doi: 10.1109/TCSVT.2021.3121679 CrossRef Google Scholar
[7]	Yeung H W F, Hou J H, Chen X M, et al. Light field spatial super-resolution using deep efficient spatial-angular separable convolution[J]. IEEE Trans Image Process, 2019, 28(5): 2319−2330. doi: 10.1109/TIP.2018.2885236 CrossRef Google Scholar
[8]	Zhang S, Lin Y F, Sheng H. Residual networks for light field image super-resolution[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, 2019: 11046–11055. https://doi.org/10.1109/CVPR.2019.01130. Google Scholar
[9]	Jin J, Hou J H, Chen J, et al. Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, 2020: 2260−2269. https://doi.org/10.1109/CVPR42600.2020.00233. Google Scholar
[10]	Liang Z Y, Wang Y Q, Wang L G, et al. Angular-flexible network for light field image super-resolution[J]. Electron Lett, 2021, 57(24): 921−924. doi: 10.1049/ell2.12312 CrossRef Google Scholar
[11]	赵圆圆, 施圣贤. 融合多尺度特征的光场图像超分辨率方法[J]. 光电工程, 2020, 47(12): 200007. doi: 10.12086/oee.2020.200007 CrossRef Google Scholar Zhao Y Y, Shi S X. Light-field image super-resolution based on multi-scale feature fusion[J]. Opto-Electron Eng, 2020, 47(12): 200007. doi: 10.12086/oee.2020.200007 CrossRef Google Scholar
[12]	Liu J, Tang J, Wu G S. Residual feature distillation network for lightweight image super-resolution[C]//Proceedings of the European Conference on Computer Vision, Glasgow, 2020: 41–55. https://doi.org/10.1007/978-3-030-67070-2_2. Google Scholar
[13]	Rerabek M, Ebrahimi T. New light field image dataset[C]//Proceedings of the 8th International Conference on Quality of Multimedia Experience, Lisbon, 2016. Google Scholar
[14]	Honauer K, Johannsen O, Kondermann D, et al. A dataset and evaluation methodology for depth estimation on 4D light fields[C]//Proceedings of the 13th Asian Conference on Computer Vision, Cham, 2016: 19–34. https://doi.org/10.1007/978-3-319-54187-7_2. Google Scholar
[15]	Wanner S, Meister S, Goldluecke B. Datasets and benchmarks for densely sampled 4D light fields[M]//Bronstein M, Favre J, Hormann K. Vision, Modeling and Visualization. Eurographics Association, 2013: 225–226. https://doi.org/10.2312/PE.VMV.VMV13.225-226. Google Scholar
[16]	Le Pendu M, Jiang X R, Guillemot C. Light field inpainting propagation via low rank matrix completion[J]. IEEE Trans Image Process, 2018, 27(4): 1981−1993. doi: 10.1109/TIP.2018.2791864 CrossRef Google Scholar
[17]	Vaish V, Adams A. The (new) Stanford light field archive, computer graphics laboratory, Stanford University[EB/OL]. 2008. http://lightfield.stanford.edu Google Scholar
[18]	Kingma D P, Ba J. Adam: a method for stochastic optimization[C]//Proceedings of the 3rd International Conference on Learning Representations, San Diego, 2015. Google Scholar
[19]	He K M, Zhang X Y, Ren S Q, et al. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification[C]//Proceedings of 2015 International Conference on Computer Vision, Santiago, 2015: 1026–1034. https://doi.org/10.1109/ICCV.2015.123. Google Scholar
[20]	Lim B, Son S, Kim H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, 2017: 136−144. https://doi.org/10.1109/CVPRW.2017.151. Google Scholar
[21]	Zhang Y L, Li K P, Li K, et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the 15th European Conference on Computer Vision, Munich, 2018: 294–310. https://doi.org/10.1007/978-3-030-01234-2_18. Google Scholar