Depth acquisition of noisy scene based on inline occlusion handling of light field

Wu Di; Zhang Xudong; Fan Zhiguo; Sun Rui

doi:10.12086/oee.2021.200422

Article navigation > Opto-Electronic Engineering > 2021 Vol. 48 > No. 7 > 200422

Next Article Previous Article

Wu D, Zhang X D, Fan Z G, et al. Depth acquisition of noisy scene based on inline occlusion handling of light field[J]. Opto-Electron Eng, 2021, 48(7): 200422. doi: 10.12086/oee.2021.200422

Citation:

Wu D, Zhang X D, Fan Z G, et al. Depth acquisition of noisy scene based on inline occlusion handling of light field[J]. Opto-Electron Eng, 2021, 48(7): 200422. doi: 10.12086/oee.2021.200422

Depth acquisition of noisy scene based on inline occlusion handling of light field

School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, Anhui 230601, China

Fund Project: National Natural Science Foundation of China (61876057, 61971177)

More Information

^*Corresponding author: Zhang Xudong, E-mail: xudong@hfut.edu.cn

Received Date 16 November 2020

Revised Date 12 May 2021

Published Date 15 July 2021

Abstract

Abstract

A light field camera can simultaneously sample a scene from multiple viewpoints with a single exposure, which has unique advantages in portability and depth accuracy over other depth sensors. Noise is a challenging issue for light field depth estimation. Most of the traditional depth estimation methods for noisy scenes are only suitable for non-occluded scenes, and cannot handle the noisy scenes with occluded regions. To solve this problem, we present a light field depth estimation method based on inline occlusion handling. The proposed method integrates the occlusion handling into the anti-noise cost volume, which can improve the anti-occlusion capability while maintaining the anti-noise performance. After the cost volume is constructed, we propose a multi-template filtering algorithm to smooth the data cost while preserving the edge structure. Experimental results show that the proposed method has better performance over other state-of-the-art depth estimation methods in high noise scenes, and can better handle the occlusion problem of depth estimation in noisy scenes.
- light field /
- depth estimation /
- defocus cue /
- noise suppression /
- occlusion handling

FullText(HTML)

References

[1]	Ng R, Levoy M, Brédif M, et al. Light field photography with a hand-held plenoptic camera[R]. Computer Science Technical Report CSTR2, 2005: 1-11. Google Scholar
[2]	Zhang S, Sheng H, Li C, et al. Robust depth estimation for light field via spinning parallelogram operator[J]. Comput Vis Image Underst, 2016, 145: 148-159. doi: 10.1016/j.cviu.2015.12.007 CrossRef Google Scholar
[3]	Wanner S, Goldluecke B. Globally consistent depth labeling of 4D light fields[C]//2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012. Google Scholar
[4]	Chen J, Hou J H, Ni Y, et al. Accurate light field depth estimation with superpixel regularization over partially occluded regions[J]. IEEE Trans Image Process, 2018, 27(10): 4889-4900. doi: 10.1109/TIP.2018.2839524 CrossRef Google Scholar
[5]	Jeon H G, Park J, Choe G, et al. Accurate depth map estimation from a lenslet light field camera[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015. Google Scholar
[6]	Williem, Park I K, Lee K M. Robust light field depth estimation using occlusion-noise aware data costs[J]. IEEE Trans Pattern Anal Mach Intell, 2018, 40(10): 2484-2497. doi: 10.1109/TPAMI.2017.2746858 CrossRef Google Scholar
[7]	Williem W, Park I K. Robust light field depth estimation for noisy scene with occlusion[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016. Google Scholar
[8]	Tao M W, Hadap S, Malik J, et al. Depth from combining defocus and correspondence using light-field cameras[C]//Proceedings of the IEEE International Conference on Computer Vision, 2013. Google Scholar
[9]	Wang T C, Efros A, Ramamoorthi R. Occlusion-aware depth estimation using light-field cameras[C]//Proceedings of the IEEE International Conference on Computer Vision, 2015. Google Scholar
[10]	Sheng H, Zhao P, Zhang S, et al. Occlusion-aware depth estimation for light field using multi-orientation EPIs[J]. Pattern Recognit, 2018, 74: 587-599. doi: 10.1016/j.patcog.2017.09.010 CrossRef Google Scholar
[11]	Guo Z H, Wu J L, Chen X F, et al. Accurate light field depth estimation using multi-orientation partial angular coherence[J]. IEEE Access, 2019, 7: 169123-169132. doi: 10.1109/ACCESS.2019.2954892 CrossRef Google Scholar
[12]	Strecke M, Goldluecke B. Sublabel-accurate convex relaxation with total generalized variation regularization[C]//German Conference on Pattern Recognition, 2018. Google Scholar
[13]	Lin H T, Chen C, Kang S B, et al. Depth recovery from light field using focal stack symmetry[C]//Proceedings of the IEEE International Conference on Computer Vision, 2015. Google Scholar
[14]	Sheng H, Zhang S, Cao X C, et al. Geometric occlusion analysis in depth estimation using integral guided filter for light-field image[J]. IEEE Trans Image Process, 2017, 26(12): 5758-5771. doi: 10.1109/TIP.2017.2745100 CrossRef Google Scholar
[15]	Honauer K, Johannsen O, Kondermann D, et al. A dataset and evaluation methodology for depth estimation on 4d light fields[C]//Asian Conference on Computer Vision, 2016. Google Scholar

Overview

Overview

Overview: Depth estimation from multiple images is a central task in computer vision. Reliable depth information provides an effective source for visual tasks, such as target detection, image segmentation, and special effects for movies. As one of the new multi-view image acquisition devices, the light field camera makes it more convenient to acquire multiple images data. A light field camera can simultaneously sample a scene from multiple viewpoints with a single exposure, which has unique advantages in portability and depth accuracy over other depth sensors. Noise is a challenging issue for light field depth estimation. Especially for high-noise scenes containing occlusion, the simultaneous presence of occlusion and noise makes depth acquisition more difficult. For this problem, we present a light field depth estimation algorithm that is robust to occlusion and noise. The proposed method uses an inline occlusion handling framework. By integrating the occlusion handling into the anti-noise cost volume, the anti-occlusion ability of the proposed method is improved while maintaining the anti-noise performance. For the construction of the anti-noise cost volume, a focal stack matching measure based on the double-directions defocusing is proposed, which increases the defocus direction of the traditional focal stack and introduces more samples for a single match. More samples allow the algorithm to select the sample with the lowest matching cost, thereby improving the anti-noise performance. For occlusion handling, the occlusion mode in noisy scenes has greater computational difficulty. To eliminate the influence of occlusion on the focal stack and not be interfered by noise, the proposed algorithm designs view masks for different occlusion modes and constructs the cost volume respectively, and then adaptively selects the best volume according to the matching cost. After the cost volume is constructed, we use the filter-based algorithm to further smooth the cost volume. Because of the problem that traditional filtering methods cannot preserve the occlusion boundary, we design a multi-template filtering strategy. This strategy designs filters for occlusion in different directions and can better preserve the edge structure of the scene. Experiments are conducted on the HCI synthetic dataset and Stanford Lytro Illum dataset for real scenes. For quantitative evaluation, we use the percentage of bad pixels and the mean square error to measure the pros and cons of every algorithm. Experimental results show that the proposed method achieves better performance than other state-of-the-art methods for scenes where occlusion and noise exist at the same time.