Multi-resolution feature fusion for point cloud classification and segmentation network

Tao Zhiyong; Li Heng; Dou Miaosen; Lin Sen

doi:10.12086/oee.2023.230166

Article navigation > Opto-Electronic Engineering > 2023 Vol. 50 > No. 10 > 230166

Next Article Previous Article

Tao Z Y, Li H, Dou M S, et al. Multi-resolution feature fusion for point cloud classification and segmentation network[J]. Opto-Electron Eng, 2023, 50(10): 230166. doi: 10.12086/oee.2023.230166

Citation:

Tao Z Y, Li H, Dou M S, et al. Multi-resolution feature fusion for point cloud classification and segmentation network[J]. Opto-Electron Eng, 2023, 50(10): 230166. doi: 10.12086/oee.2023.230166

Multi-resolution feature fusion for point cloud classification and segmentation network

1.
School of Electronic and Information Engineering, Liaoning Technical University, Huludao, Liaoning 125100, China
2.
School of Automation and Electrical Engineering, Shenyang Ligong University, Shenyang, Liaoning 110159, China

Fund Project: Project supported by Department of Science & Technology of Liaoning Province Application Fundamental Research Project(2022JH2/101300274)

More Information

^*Corresponding author: PaperLH@163.com

Received Date 07 July 2023

Revised Date 13 September 2023

Accepted Date 20 September 2023

Published Date 25 October 2023

Abstract

Abstract

To address the problem that existing networks find it difficult to learn local geometric information of point cloud effectively, a graph convolutional network that fuses multi-resolution features of point cloud is proposed. First, the local graph structure of the point cloud is constructed by the k-nearest neighbor algorithm to better represent the local geometric structure of the point cloud. Second, a parallel channel branch is proposed based on the farthest point sampling algorithm, which obtains point clouds with different resolutions by downsampling them and then groups them. To overcome the sparse characteristics of the point cloud, a geometric mapping module is proposed to perform normalization operations on the grouped point cloud. Finally, a feature fusion module is proposed to aggregate graph features and multi-resolution features to obtain global features more effectively. Experiments are evaluated using ModelNet40, ScanObjectNN, and ShapeNet Part datasets. The experimental results show that the proposed network has state-of-the-art classification and segmentation performance.
- point cloud /
- graph convolutional network /
- multi-resolution point cloud /
- feature fusion module

FullText(HTML)

References

[1]	张昕怡, 陈茂霖, 刘祥江, 等. 顾及点密度与未知角分辨率的地面点云分类[J]. 激光技术, 2023, 47(1): 59−66. doi: 10.7510/jgjs.issn.1001-3806.2023.01.009 CrossRef Google Scholar Zhang X Y, Chen M L, Liu X J, et al. Classification of terrestrial point cloud considering point density and unknown angular resolution[J]. Laser Technol, 2023, 47(1): 59−66. doi: 10.7510/jgjs.issn.1001-3806.2023.01.009 CrossRef Google Scholar
[2]	李佳男, 王泽, 许廷发. 基于点云数据的三维目标检测技术研究进展[J]. 光学学报, 2023, 43(15): 1515001. doi: 10.3788/AOS230745 CrossRef Google Scholar Li J N, Wang Z, Xu T F. Three-dimensional object detection technology based on point cloud data[J]. Acta Opt Sin, 2023, 43(15): 1515001. doi: 10.3788/AOS230745 CrossRef Google Scholar
[3]	陆康亮, 薛俊, 陶重犇. 融合空间掩膜预测与点云投影的多目标跟踪[J]. 光电工程, 2022, 49(9): 220024. doi: 10.12086/oee.2022.220024 CrossRef Google Scholar Lu K L, Xue J, Tao C B. Multi target tracking based on spatial mask prediction and point cloud projection[J]. Opto-Electron Eng, 2022, 49(9): 220024. doi: 10.12086/oee.2022.220024 CrossRef Google Scholar
[4]	陈仲生, 李潮林, 左旺, 等. 双重下采样增强的点云改进配准算法研究[J]. 汽车工程, 2023, 45(4): 572−578. doi: 10.19562/j.chinasae.qcgc.2023.04.005 CrossRef Google Scholar Chen Z S, Li C L, Zuo W, et al. Study on improved point cloud registration algorithm enhanced by double down-sampling[J]. Automot Eng, 2023, 45(4): 572−578. doi: 10.19562/j.chinasae.qcgc.2023.04.005 CrossRef Google Scholar
[5]	李美佳, 于泽宽, 刘晓, 等. 点云算法在医学领域的研究进展[J]. 中国图象图形学报, 2020, 25(10): 2013−2023. doi: 10.11834/jig.200253 CrossRef Google Scholar Li M J, Yu Z K, Liu X, et al. Progress of point cloud algorithm in medical field[J]. J Image Graphics, 2020, 25(10): 2013−2023. doi: 10.11834/jig.200253 CrossRef Google Scholar
[6]	夏金泽, 孙浩铭, 胡盛辉, 等. 基于图像信息约束的三维激光点云聚类方法[J]. 光电工程, 2023, 50(2): 220148. doi: 10.12086/oee.2023.220148 CrossRef Google Scholar Xia J Z, Sun H M, Hu S H, et al. 3D laser point cloud clustering method based on image information constraints[J]. Opto-Electron Eng, 2023, 50(2): 220148. doi: 10.12086/oee.2023.220148 CrossRef Google Scholar
[7]	柏宏强, 夏永华, 杨明龙, 等. 基于三维激光点云特征线提取的溶洞多分辨率三维重建方法研究[J]. 激光与光电子学进展, 2020, 57(20): 202802. doi: 10.3788/LOP57.202802 CrossRef Google Scholar Bai H Q, Xia Y H, Yang M L, et al. Multi-resolution 3D reconstruction of karst caves based on the feature line extraction of 3D laser point cloud[J]. Laser Optoelectron Prog, 2020, 57(20): 202802. doi: 10.3788/LOP57.202802 CrossRef Google Scholar
[8]	Maturana D, Scherer S. Voxnet: a 3D convolutional neural network for real-time object recognition[C]//Proceedings of 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2015: 922–928. https://doi.org/10.1109/IROS.2015.7353481. Google Scholar
[9]	Riegler G, Osman Ulusoy A, Geiger A. OctNet: learning deep 3D representations at high resolutions[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6620–6629. https://doi.org/10.1109/CVPR.2017.701. Google Scholar
[10]	肖正涛, 高健, 吴东庆, 等. 一种基于体素网格的三维点云均匀降采样方法[J]. 机械设计与制造, 2023(8): 180−184. doi: 10.19356/j.cnki.1001-3997.20230310.025 CrossRef Google Scholar Xiao Z T, Gao J, Wu D Q, et al. A uniform downsampling method for three-dimensional point clouds based on voxel grids[J]. Mach Des Manuf, 2023(8): 180−184. doi: 10.19356/j.cnki.1001-3997.20230310.025 CrossRef Google Scholar
[11]	Su H, Maji S, Kalogerakis E, et al. Multi-view convolutional neural networks for 3D shape recognition[C]//Proceedings of 2015 IEEE International Conference on Computer Vision, 2015: 945–953. https://doi.org/10.1109/ICCV.2015.114. Google Scholar
[12]	Charles R Q, Su H, Mo K C, et al. PointNet: deep learning on point sets for 3D classification and segmentation[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 77–85. https://doi.org/10.1109/CVPR.2017.16 Google Scholar
[13]	Qi C R, Yi L, Su H, et al. PointNet++: Deep hierarchical feature learning on point sets in a metric space[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 5105–5114. https://doi.org/10.5555/3295222.3295263. Google Scholar
[14]	Simonovsky M, Komodakis N. Dynamic edge-conditioned filters in convolutional neural networks on graphs[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 29–38. https://doi.org/10.1109/CVPR.2017.11. Google Scholar
[15]	Wang Y, Sun Y B, Liu Z W, et al. Dynamic graph CNN for learning on point clouds[J]. ACM Trans Graphics, 2019, 38(5): 146. doi: 10.1145/3326362 CrossRef Google Scholar
[16]	Zhang K G, Hao M, Wang J, et al. Linked dynamic graph CNN: learning on point cloud via linking hierarchical features[Z]. arXiv: 1904.10014, 2019. https://doi.org/10.48550/arXiv.1904.10014. Google Scholar
[17]	刘斌, 樊云超. 基于改进动态图卷积的点云分类模型[J]. 中国科技论文, 2022, 17(11): 1230−1235, 1266. doi: 10.3969/j.issn.2095-2783.2022.11.009 CrossRef Google Scholar Liu B, Fan Y C. A point cloud classification model based on improved dynamic graph convolution[J]. China Sci, 2022, 17(11): 1230−1235, 1266. doi: 10.3969/j.issn.2095-2783.2022.11.009 CrossRef Google Scholar
[18]	Sun Q, Liu H Y, He J, et al. DAGC: employing dual attention and graph convolution for point cloud based place recognition[C]//Proceedings of 2020 International Conference on Multimedia Retrieval, 2020: 224–232. https://doi.org/10.1145/3372278.3390693. Google Scholar
[19]	Guo M H, Cai J X, Liu Z N, et al. PCT: point cloud transformer[J]. Comput Vis Med, 2021, 7(2): 187−199. doi: 10.1007/s41095-021-0229-5 CrossRef Google Scholar
[20]	Liu H, Tian S H. Deep 3D point cloud classification and segmentation network based on GateNet[J]. Vis Comput, 2023. https://doi.org/10.1007/s00371-023-02826-w. Google Scholar
[21]	Wu Z R, Song S R, Khosla A, et al. 3D shapeNets: a deep representation for volumetric shapes[C]/Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition, 2015: 1912–1920. https://doi.org/10.1109/CVPR.2015.7298801. Google Scholar
[22]	Uy M A, Pham Q H, Hua B S, et al. Revisiting point cloud classification: a new benchmark dataset and classification model on real-world data[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision, 2019: 1588–1597. https://doi.org/10.1109/ICCV.2019.00167. Google Scholar
[23]	Yi L, Kim V G, Ceylan D, et al. A scalable active framework for region annotation in 3D shape collections[J]. ACM Trans Graphics, 2016, 35(6): 210. doi: 10.1145/2980179.2980238 CrossRef Google Scholar
[24]	王子璇, 任明武. DST-Pointnet++: 基于Pointnet++改进的点云分类网络[J]. 计算机与数字工程, 2022, 50(11): 2497−2501. doi: 10.3969/j.issn.1672-9722.2022.11.026 CrossRef Google Scholar Wang Z X, Ren M W. DST-Pointnet++: A novel point cloud classification network based on pointnet++[J]. Comput Digit Eng, 2022, 50(11): 2497−2501. doi: 10.3969/j.issn.1672-9722.2022.11.026 CrossRef Google Scholar
[25]	王本杰, 农丽萍, 张文辉, 等. 基于Spider卷积的三维点云分类与分割网络[J]. 计算机应用, 2020, 40(6): 1607−1612. doi: 10.11772/j.issn.1001-9081.2019101879 CrossRef Google Scholar Wang B J, Nong L P, Zhang W H, et al. 3D point cloud classification and segmentation network based on Spider convolution[J]. J Comput Appl, 2020, 40(6): 1607−1612. doi: 10.11772/j.issn.1001-9081.2019101879 CrossRef Google Scholar
[26]	Lin Z H, Huang S Y, Wang Y C F. Convolution in the cloud: learning deformable kernels in 3D graph convolution networks for point cloud analysis[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 1797–1806. https://doi.org/10.1109/CVPR42600.2020.00187. Google Scholar
[27]	Chen L F, Zhang Q. DDGCN: graph convolution network based on direction and distance for point cloud learning[J]. Vis Comput, 2023, 39(3): 863−873. doi: 10.1007/s00371-021-02351-8 CrossRef Google Scholar
[28]	Qiu S, Anwar S, Barnes N. Dense-resolution network for point cloud classification and segmentation[C]//Proceedings of 2021 IEEE Winter Conference on Applications of Computer Vision, 2021: 3812–3821. https://doi.org/10.1109/WACV48630.2021.00386. Google Scholar
[29]	Wan J, Xie Z, Xu Y Y, et al. DGANet: a dilated graph attention-based network for local feature extraction on 3D point clouds[J]. Remote Sens, 2021, 13(17): 3484. doi: 10.3390/rs13173484 CrossRef Google Scholar
[30]	张润梅, 程婷, 尹蕾, 等. 一种注意力融合的多尺度点云分类网络[J]. 淮北师范大学学报(自然科学版), 2023, 44(1): 70−75. doi: 10.3969/j.issn.2096-8248.2023.01.012 CrossRef Google Scholar Zhang R M, Cheng T, Yin L, et al. Attention fusion and multi-scale point cloud classification network[J]. J Huaibei Normal Univ (Natl Sci), 2023, 44(1): 70−75. doi: 10.3969/j.issn.2096-8248.2023.01.012 CrossRef Google Scholar
[31]	国玉恩, 任明武. 基于PointConv改进的点云分类网络[J]. 计算机与数字工程, 2022, 50(12): 2737−2740, 2764. doi: 10.3969/j.issn.1672-9722.2022.12.026 CrossRef Google Scholar Guo Y E, Ren M W. Improved point cloud classification network based on PointConv[J]. Comput Digit Eng, 2022, 50(12): 2737−2740, 2764. doi: 10.3969/j.issn.1672-9722.2022.12.026 CrossRef Google Scholar
[32]	Qiu S, Anwar S, Barnes N. Geometric back-projection network for point cloud classification[J]. IEEE Trans Multimedia, 2021, 24: 1943−1955. doi: 10.1109/TMM.2021.3074240 CrossRef Google Scholar
[33]	Cheng S L, Chen X W, He X W, et al. PRA-Net: point relation-aware network for 3D point cloud analysis[J]. IEEE Trans Image Process, 2021, 30: 4436−4448. doi: 10.1109/TIP.2021.3072214 CrossRef Google Scholar
[34]	宋巍, 蔡万源, 何盛琪, 等. 结合动态图卷积和空间注意力的点云分类与分割[J]. 中国图象图形学报, 2021, 26(11): 2691−2702. doi: 10.11834/jig.200550 CrossRef Google Scholar Song W, Cai W Y, He S Q, et al. Dynamic graph convolution with spatial attention for point cloud classification and segmentation[J]. J Image Graphics, 2021, 26(11): 2691−2702. doi: 10.11834/jig.200550 CrossRef Google Scholar
[35]	Zhou H R, Feng Y D, Fang M S, et al. Adaptive graph convolution for point cloud analysis[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision, 2021: 4945–4954. https://doi.org/10.1109/ICCV48922.2021.00492. Google Scholar

Overview

Overview

In recent years, 3D point cloud analysis has become a hot topic in computer vision and been widely used in mapping, medical imaging, and autonomous driving. As a 3D image representation, point cloud contains rich geometric information. With the development of 3D scanning technologies such as LiDAR, the acquisition of point clouds is becoming more accessible. Since convolutional neural networks (CNN) have greatly improved the results of computer vision tasks, neural networks are becoming a mainstream approach in image processing. Traditional 2D images comprise regular and dense pixels, and CNNs apply to 2D image processing. However, point cloud data are sparse and disordered; each point does not contain additional information (e.g., RGB). Using traditional CNNs for point cloud learning tasks is a challenging task. The graph-like structure can effectively represent non-Euclidean data like point clouds, and this method largely solves the problem of difficulty in learning the local features of point clouds. Since the graph structure construction process is generally based on the k-nearest neighbor algorithm (kNN), the size of the predefined neighborhood limits the effectiveness of the local graph structure. If the value of k is too small, it will lead to an incomplete representation of local information. At the same time, too large a value of k will introduce information redundancy, leading to performance degradation. To this end, we propose a multi-resolution graph convolutional network to perform the point cloud analysis task. The network learns the local features of point clouds by constructing graph structures and then downsamples the point clouds using the farthest point sampling method (FPS) to obtain multi-resolution point cloud data, followed by feature learning for point clouds at different resolutions. To overcome the effect of predefined neighborhoods, we compensate local features with multi-resolution features and efficiently aggregate point cloud features by the feature fusion module. To verify the classification and segmentation performance of the model, we perform classification experiments on ModelNet40 and ScanObjectNN datasets and part segmentation experiments on ShapeNet Part dataset. It is experimentally verified that the compensation of point cloud local graph structure information with multi-resolution features can enhance point clouds' local feature learning ability. The multi-resolution graph convolutional network proposed in this paper can effectively capture the local features of point clouds and achieve good results in classification and segmentation tasks. A large number of ablation experiments verify the effectiveness and robustness of the model.