Light field depth estimation using weighted side window angular coherence

Ma Shuai; Wang Ning; Zhu Licheng; Wang Shuai; Yang Ping; Xu Bing

doi:10.12086/oee.2021.210405

Article navigation > Opto-Electronic Engineering > 2021 Vol. 48 > No. 12 > 210405

Next Article Previous Article

Ma S, Wang N, Zhu L C, et al. Light field depth estimation using weighted side window angular coherence[J]. Opto-Electron Eng, 2021, 48(12): 210405. doi: 10.12086/oee.2021.210405

Citation:

Ma S, Wang N, Zhu L C, et al. Light field depth estimation using weighted side window angular coherence[J]. Opto-Electron Eng, 2021, 48(12): 210405. doi: 10.12086/oee.2021.210405

Light field depth estimation using weighted side window angular coherence

1.
Key Laboratory of Adaptive Optics, Chinese Academy of Sciences, Chengdu, Sichuan 610209, China
2.
Institute of Optics and Electronics, Chinese Academy of Sciences, Chengdu, Sichuan 610209, China
3.
University of Chinese Academy of Sciences, Beijing 100049, China

Fund Project:

More Information

Corresponding author: Xu Bing, E-mail: bing_xu_ioe@163.com

Received Date 22 November 2021

Revised Date 05 December 2021

Published Date 30 December 2021

Abstract

Abstract

Light field imaging recodes not only the intensity information of the rays, but also its direction information, and has the ability to estimate the depth of the scene. However, the accuracy of the depth estimation is easily influenced by light field occlusion. This paper proposes a method of weighted side window angular coherence to deal with different types of occlusions. Firstly, the angular patch is divided into four side window subsets, and the coherence of the pixels in these subsets is measured to construct four cost volumes to solve different types of occlusion. Secondly, the weighted fusion strategy is proposed to fuse the four cost volumes to further enhance the robustness of the algorithm and retain the anti-occlusion of the algorithm. Finally, the fused cost volume is optimized by the guided filter to further improve the accuracy of depth estimation. Experimental results show that the proposed method is superior to the existing methods in the quantitative index and can achieve high-precision measurement in the absolute depth measurement experiment.
- side window angular coherence /
- weighted fusion /
- light field imaging /
- depth estimation

FullText(HTML)

References

[1]	Levoy M, Hanrahan P. Light field rendering[C]//Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, New York, 1996: 31–42. Google Scholar
[2]	Ng R. Fourier slice photography[J]. ACM Trans Graph, 2005, 24(3): 735–744. doi: 10.1145/1073204.1073256 CrossRef Google Scholar
[3]	赵圆圆, 施圣贤. 融合多尺度特征的光场图像超分辨率方法[J]. 光电工程, 2020, 47(12): 200007. doi: 10.12086/oee.2020.200007 CrossRef Google Scholar Zhao Y Y, Shi S X. Light-field image super-resolution based on multi-scale feature fusion[J]. Opto-Electron Eng, 2020, 47(12): 200007. doi: 10.12086/oee.2020.200007 CrossRef Google Scholar
[4]	CDATA[Piao Y R, Rong Z K, Xu S, et al. DUT-LFSaliency: versatile dataset and light field-to-RGB saliency detection[Z]. arXiv preprint arXiv: 2012.15124, 2020. Google Scholar
[5]	吴迪, 张旭东, 范之国, 等. 基于光场内联遮挡处理的噪声场景深度获取[J]. 光电工程, 2021, 48(7): 200422. doi: 10.12086/oee.2021.200422 CrossRef Google Scholar Wu D, Zhang X D, Fan Z G, et al. Depth acquisition of noisy scene based on inline occlusion handling of light field[J]. Opto-Electron Eng, 2021, 48(7): 200422. doi: 10.12086/oee.2021.200422 CrossRef Google Scholar
[6]	Wang T C, Efros A A, Ramamoorthi R. Depth estimation with occlusion modeling using light-field cameras[J]. IEEE Trans Pattern Anal Mach Intell, 2016, 38(11): 2170–2181. doi: 10.1109/TPAMI.2016.2515615 CrossRef Google Scholar
[7]	Jeon H G, Park J, Choe G, et al. Accurate depth map estimation from a lenslet light field camera[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, 2015: 1547–1555. Google Scholar
[8]	Johannsen O, Sulc A, Goldluecke B. What sparse light field coding reveals about scene structure[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, 2016: 3262–3270. Google Scholar
[9]	Chen C, Lin H T, Yu Z, et al. Light field stereo matching using bilateral statistics of surface cameras[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, 2014: 1518–1525. Google Scholar
[10]	Zhang S, Sheng H, Li C, et al. Robust depth estimation for light field via spinning parallelogram operator[J]. Comput Vis Image Underst, 2016, 145: 148–159. doi: 10.1016/j.cviu.2015.12.007 CrossRef Google Scholar
[11]	Wang W K, Lin Y F, Zhang S. Enhanced spinning parallelogram operator combining color constraint and histogram integration for robust light field depth estimation[J]. IEEE Signal Process Lett, 2021, 28: 1080–1084. doi: 10.1109/LSP.2021.3079844 CrossRef Google Scholar
[12]	Zhu H, Wang Q, Yu J Y. Occlusion-model guided antiocclusion depth estimation in light field[J]. IEEE J Sel Top Signal Process, 2017, 11(7): 965–978. doi: 10.1109/JSTSP.2017.2730818 CrossRef Google Scholar
[13]	Ng R, Levoy M, Brédif M, et al. Light field photography with a hand-held plenoptic camera[R]. Computer Science Technical Report CSTR, 2005. Google Scholar
[14]	Ma S, Guo Z H, Wu J L, et al. Occlusion-aware light field depth estimation using side window angular coherence[J]. Appl Opt, 2021, 60(2): 392–404. doi: 10.1364/AO.411070 CrossRef Google Scholar
[15]	Boykov Y, Veksler O, Zabih R. Fast approximate energy minimization via graph cuts[J]. IEEE Trans Pattern Anal Mach Intell, 2001, 23(11): 1222–1239. doi: 10.1109/34.969114 CrossRef Google Scholar
[16]	He K M, Sun J, Tang X O. Guided image filtering[J]. IEEE Trans Pattern Anal Mach Intell, 2013, 35(6): 1397–1409. doi: 10.1109/TPAMI.2012.213 CrossRef Google Scholar
[17]	Honauer K, Johannsen O, Kondermann D, et al. A dataset and evaluation methodology for depth estimation on 4d light fields[C]//Proceedings of the 13th Asian Conference on Computer Vision, Taiwan, China, 2016: 19–34. Google Scholar
[18]	Strecke M, Alperovich A, Goldluecke B. Accurate depth and normal maps from occlusion-aware focal stack symmetry[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, 2017: 2529–2537. Google Scholar
[19]	石梦迪, 张旭东, 董运流, 等. 一种双引导滤波的光场去马赛克方法[J]. 光电工程, 2019, 46(12): 180539. doi: 10.12086/oee.2019.180539 CrossRef Google Scholar Shi M D, Zhang X D, Dong Y L, et al. A light field demosaicing method with double guided filtering[J]. Opto-Electron Eng, 2019, 46(12): 180539. doi: 10.12086/oee.2019.180539 CrossRef Google Scholar
[20]	Dansereau D G, Pizarro O, Williams S B. Decoding, calibration and rectification for lenselet-based plenoptic cameras[C]//Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, 2013: 1027–1034. Google Scholar
[21]	Williem, Park I K, Lee K M. Robust light field depth estimation using occlusion-noise aware data costs[J]. IEEE Trans Pattern Anal Mach Intell, 2017, 40(10): 2484–2497. Google Scholar
[22]	Bok Y, Jeon H G, Kweon I S. Geometric calibration of micro-lens-based light field cameras using line features[J]. IEEE Transn Pattern Anal Mach Intell, 2017, 39(2): 287–300. doi: 10.1109/TPAMI.2016.2541145 CrossRef Google Scholar

Overview

Overview

Overview: Compared with traditional imaging techniques, the light field imaging technique can recode the intensity information and direction information of rays at the same time, which is favored by both scientific research and commercial fields. Meanwhile, the equipment of recoding light field has also been greatly developed, such as programmable aperture light field, camera array, the gantry system, and micro-lens-array based light field cameras. Due to the characteristics of light field imaging, it is also applied to many fields, such as refocus, 3D reconstruction, super-resolution, object detection, light field edit, and depth estimation. Among them, depth estimation is a key step in the application of light field to high-dimensional vision, such as 3D reconstruction. However, the accuracy of depth estimation is easily influenced by the light field occlusion.

In this paper, we proposed a method of weighted side window angular coherence to solve different types of occlusion problems. Firstly, the angular patch image is divided into four different patterns of side window subsets and the pixels in these subsets are measured respectively to construct four cost volumes. The side window subset which only contains the pixels from the occluded point will exhibit the photo consistency when the depth label represents the true depth label. Then, the true depth can be obtained by the cost volume corresponding to the subsets only containing the occluded pixels, which can deal with the occlusion problem well. Secondly, we proposed a weighted strategy to fuse four cost volumes into one cost volume. According to the characteristics of four side window subsets, the cost volume corresponding to four side window subsets is given by different weighted values, which can enhance the robustness of the proposed algorithm and retain its ability to resist different types of occlusions. Finally, the fused cost volume is optimized by the guided filter to further improve the quality of the depth map. Experimental results in both synthetics and real scenes show the proposed method can handle the occlusion problem well and outperform the existing methods, especially near occlusion boundaries. In synthetic scenes, the proposed method is nearly as good as the other methods in the quantitative index of some scenes, which further proves the effectiveness of our method. In the real scene captured by our light field camera, combined with the calibration parameters of the light field camera, our method can accurately measure the surface size of the standard part, which proves the robustness of our method.