Breast tumor grading network based on adaptive fusion and microscopic imaging

Huang Pan; He Peng; Yang Xing; Luo Jiayang; Xiao Hualiang; Tian Sukun; Feng Peng

doi:10.12086/oee.2023.220158

Article navigation > Opto-Electronic Engineering > 2023 Vol. 50 > No. 1 > 220158

Next Article Previous Article

Huang P, He P, Yang X, et al. Breast tumor grading network based on adaptive fusion and microscopic imaging[J]. Opto-Electron Eng, 2023, 50(1): 220158. doi: 10.12086/oee.2023.220158

Citation:

Huang P, He P, Yang X, et al. Breast tumor grading network based on adaptive fusion and microscopic imaging[J]. Opto-Electron Eng, 2023, 50(1): 220158. doi: 10.12086/oee.2023.220158

Breast tumor grading network based on adaptive fusion and microscopic imaging

1.
Key Laboratory of Optoelectronic Technology & Systems (Ministry of Education), College of Optoelectronic Engineering, Chongqing University, Chongqing 400044, China
2.
College of Computer and Network Security, Chengdu University of Technology, Chengdu, Sichuan 610000, China
3.
Daping Hospital, Department of Pathology, Army Military Medical University, Chongqing 400037, China
4.
School of Mechanical Engineering, Shandong University, Jinan, Shandong 250000, China

Fund Project:

More Information

^*Corresponding authors: dpbl_xhl@163.com; sukhum169@hotmail.com; coe-fp@cqu.edu.cn

Received Date 08 July 2022

Revised Date 06 September 2022

Accepted Date 06 September 2022

Available Online 28 December 2022

Published Date 19 January 2023

Abstract

Abstract

Tumor grading based on microscopic imaging is critical for the diagnosis and prognosis of breast cancer, which demands excellent accuracy and interpretability. Deep networks with CNN blocks combined with attention currently offer better induction bias capabilities but low interpretability. In comparison, the deep network based on ViT blocks has stronger interpretability but less induction bias capabilities. To that end, we present an end-to-end adaptive model fusion for deep networks that combine ViT and CNN blocks with integrated attention. However, the existing model fusion methods suffer from negative fusion. Because there is no guarantee that both the ViT blocks and the CNN blocks with integrated attention have acceptable feature representation capabilities, and secondly, the great similarity between the two feature representations results in a lot of redundant information, resulting in a poor model fusion capability. For that purpose, the adaptive model fusion approach suggested in this study consists of multi-objective optimization, an adaptive feature representation metric, and adaptive feature fusion, thereby significantly boosting the model's fusion capabilities. The accuracy of this model is 95.14%, which is 9.73% better than that of ViT-B/16, and 7.6% better than that of FABNet; secondly, the visualization map of our model is more focused on the regions of nuclear heterogeneity (e.g., mega nuclei, polymorphic nuclei, multi-nuclei, and dark nuclei), which is more consistent with the regions of interest to pathologists. Overall, the proposed model outperforms other state-of-the-art models in terms of accuracy and interpretability.
- microscopic imaging /
- interpretability /
- deep learning /
- adaptive fusion /
- breast cancer /
- tumor grading

FullText(HTML)

References

[1]	Parkin D M, Bray F, Ferlay J, et al. Estimating the world cancer burden: globocan 2000[J]. Int J Cancer, 2001, 94(2): 153−156. doi: 10.1002/ijc.1440 CrossRef Google Scholar
[2]	Robertson S, Azizpour H, Smith K, et al. Digital image analysis in breast pathology—from image processing techniques to artificial intelligence[J]. Transl Res, 2018, 194: 19−35. doi: 10.1016/j.trsl.2017.10.010 CrossRef Google Scholar
[3]	Desai S B, Moonim M T, Gill A K, et al. Hormone receptor status of breast cancer in India: a study of 798 tumours[J]. Breast, 2000, 9(5): 267−270. doi: 10.1054/brst.2000.0134 CrossRef Google Scholar
[4]	Zafrani B, Aubriot M H, Mouret E, et al. High sensitivity and specificity of immunohistochemistry for the detection of hormone receptors in breast carcinoma: comparison with biochemical determination in a prospective study of 793 cases[J]. Histopathology, 2000, 37(6): 536−545. doi: 10.1046/j.1365-2559.2000.01006.x CrossRef Google Scholar
[5]	Fuqua S A W, Schiff R, Parra I, et al. Estrogen receptor β protein in human breast cancer: correlation with clinical tumor parameters[J]. Cancer Res, 2003, 63(10): 2434−2439. Google Scholar
[6]	Vagunda V, Šmardová J, Vagundová M, et al. Correlations of breast carcinoma biomarkers and p53 tested by FASAY and immunohistochemistry[J]. Pathol Res Pract, 2003, 199(12): 795−801. doi: 10.1078/0344-0338-00498 CrossRef Google Scholar
[7]	Baqai T, Shousha S. Oestrogen receptor negativity as a marker for high-grade ductal carcinoma in situ of the breast[J]. Histopathology, 2003, 42(5): 440−447. doi: 10.1046/j.1365-2559.2003.01612.x CrossRef Google Scholar
[8]	Sofi G N, Sofi J N, Nadeem R, et al. Estrogen receptor and progesterone receptor status in breast cancer in relation to age, histological grade, size of lesion and lymph node involvement[J]. Asian Pac J Cancer Prev, 2012, 13(10): 5047−5052. doi: 10.7314/apjcp.2012.13.10.5047 CrossRef Google Scholar
[9]	Azizun-Nisa, Bhurgri Y, Raza F, et al. Comparison of ER, PR and HER-2/neu (C-erb B 2) reactivity pattern with histologic grade, tumor size and lymph node status in breast cancer[J]. Asian Pac J Cancer Prev, 2008, 9(4): 553−556. Google Scholar
[10]	Gurcan M N, Boucheron L E, Can A, et al. Histopathological image analysis: a review[J]. IEEE Rev Biomed Eng, 2009, 2: 147−171. doi: 10.1109/RBME.2009.2034865 CrossRef Google Scholar
[11]	Yang H, Kim J Y, Kim H, et al. Guided soft attention network for classification of breast cancer histopathology images[J]. IEEE Trans Med Imaging, 2020, 39(5): 1306−1315. doi: 10.1109/TMI.2019.2948026 CrossRef Google Scholar
[12]	BenTaieb A, Hamarneh G. Predicting cancer with a recurrent visual attention model for histopathology images[C]//21st International Conference on Medical Image Computing and Computer Assisted Intervention, 2018: 129–137. https://doi.org/10.1007/978-3-030-00934-2_15. Google Scholar
[13]	Tomita N, Abdollahi B, Wei J, et al. Attention-based deep neural networks for detection of cancerous and precancerous esophagus tissue on histopathological slides[J]. JAMA Netw Open, 2019, 2(11): e1914645. doi: 10.1001/jamanetworkopen.2019.14645 CrossRef Google Scholar
[14]	Yao J W, Zhu X L, Jonnagaddala J, et al. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks[J]. Med Image Anal, 2020, 65: 101789. doi: 10.1016/j.media.2020.101789 CrossRef Google Scholar
[15]	Huang P, Tan X H, Zhou X L, et al. FABNet: Fusion attention block and transfer learning for laryngeal cancer tumor grading in p63 IHC histopathology images[J]. IEEE J Biomed Health Inform, 2022, 26(4): 1696−1707. doi: 10.1109/JBHI.2021.3108999 CrossRef Google Scholar
[16]	Zhou X L, Tang C W, Huang P, et al. LPCANet: classification of laryngeal cancer histopathological images using a CNN with position attention and channel attention mechanisms[J]. Interdiscip Sci, 2021, 13(4): 666−682. doi: 10.1007/s12539-021-00452-5 CrossRef Google Scholar
[17]	刘敬璇, 樊金宇, 汪权, 等. SS-OCTA对黑色素瘤皮肤结构和血管的成像实验[J]. 光电工程, 2020, 47(2): 190239. doi: 10.12086/oee.2020.190239 CrossRef Google Scholar Liu J X, Fan J Y, Wang Q, et al. Imaging of skin structure and vessels in melanoma by swept source optical coherence tomography angiography[J]. Opto-Electron Eng, 2020, 47(2): 190239. doi: 10.12086/oee.2020.190239 CrossRef Google Scholar
[18]	Abnar S, Zuidema W. Quantifying attention flow in transformers[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 4190–4197. https://doi.org/10.18653/v1/2020.acl-main.385. Google Scholar
[19]	Chefer H, Gur S, Wolf L. Transformer interpretability beyond attention visualization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 782–791. https://doi.org/10.1109/CVPR46437.2021.00084. Google Scholar
[20]	梁礼明, 周珑颂, 陈鑫, 等. 鬼影卷积自适应视网膜血管分割算法[J]. 光电工程, 2021, 48(10): 210291. doi: 10.12086/oee.2021.210291 CrossRef Google Scholar Liang L M, Zhou L S, Chen X, et al. Ghost convolution adaptive retinal vessel segmentation algorithm[J]. Opto-Electron Eng, 2021, 48(10): 210291. doi: 10.12086/oee.2021.210291 CrossRef Google Scholar
[21]	刘侠, 甘权, 刘晓, 等. 基于超像素的联合能量主动轮廓CT图像分割方法[J]. 光电工程, 2020, 47(1): 190104. doi: 10.12086/oee.2020.190104 CrossRef Google Scholar Liu X, Gan Q, Liu X, et al. Joint energy active contour CT image segmentation method based on super-pixel[J]. Opto-Electron Eng, 2020, 47(1): 190104. doi: 10.12086/oee.2020.190104 CrossRef Google Scholar
[22]	Gao Z Y, Hong B Y, Zhang X L, et al. Instance-based vision transformer for subtyping of papillary renal cell carcinoma in histopathological image[C]//24th International Conference on Medical Image Computing and Computer Assisted Intervention, 2021: 299–308. https://doi.org/10.1007/978-3-030-87237-3_29. Google Scholar
[23]	Wang X Y, Yang S, Zhang J, et al. TransPath: Transformer-based self-supervised learning for histopathological image classification[C]//24th International Conference on Medical Image Computing and Computer Assisted Intervention, 2021: 186–195. https://doi.org/10.1007/978-3-030-87237-3_18. Google Scholar
[24]	Li H, Yang F, Zhao Y, et al. DT-MIL: deformable transformer for multi-instance learning on histopathological image[C]//24th International Conference on Medical Image Computing and Computer Assisted Intervention, 2021: 206–216. https://doi.org/10.1007/978-3-030-87237-3_20. Google Scholar
[25]	Chen H Y, Li C, Wang G, et al. GasHis-transformer: a multi-scale visual transformer approach for gastric histopathological image detection[J]. Pattern Recognit, 2022, 130: 108827. doi: 10.1016/j.patcog.2022.108827 CrossRef Google Scholar
[26]	Zou Y, Chen S N, Sun Q L, et al. DCET-Net: dual-stream convolution expanded transformer for breast cancer histopathological image classification[C]//2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2021: 1235–1240. https://doi.org/10.1109/BIBM52615.2021.9669903. Google Scholar
[27]	Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: transformers for image recognition at scale[C]//9th International Conference on Learning Representations, 2021. Google Scholar
[28]	Breiman L. Bagging predictors[J]. Mach Learn, 1996, 24(2): 123−140. doi: 10.1023/A:1018054314350 CrossRef Google Scholar
[29]	Schapire R E. The boosting approach to machine learning: an overview[M]//Denison D D, Hansen M H, Holmes C C, et al. Nonlinear Estimation and Classification. New York: Springer, 2003: 149–171. https://doi.org/10.1007/978-0-387-21579-2_9. Google Scholar
[30]	Ting K M, Witten I H. Stacking bagged and Dagged models[C]//Proceedings of the Fourteenth International Conference on Machine Learning, 1997: 367–375. https://doi.org/10.5555/645526.657147. Google Scholar
[31]	Ke W C, Fan F, Liao P X, et al. Biological gender estimation from panoramic dental X-ray images based on multiple feature fusion model[J]. Sens Imaging, 2020, 21(1): 54. doi: 10.1007/s11220-020-00320-4 CrossRef Google Scholar
[32]	周志华. 机器学习[M]. 北京: 清华大学出版社, 2016: 171–196. Google Scholar Zhou Z H. Machine Learning[M]. Beijing: Tsinghua University Press, 2016: 171–196. Google Scholar
[33]	Selvaraju R R, Cogswell M, Das A, et al. Grad-CAM: visual explanations from deep networks via gradient-based localization[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, 2017: 618–626. Google Scholar
[34]	Selvaraju R R, Cogswell M, Das A, et al. Grad-CAM: visual explanations from deep networks via gradient-based localization[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 618–626. https://doi.org/10.1109/ICCV.2017.74. Google Scholar
[35]	Kostopoulos S, Glotsos D, Cavouras D, et al. Computer-based association of the texture of expressed estrogen receptor nuclei with histologic grade using immunohistochemically-stained breast carcinomas[J]. Anal Quant Cytol Histol, 2009, 31(4): 187−196. Google Scholar
[36]	Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016: 2818–2826. https://doi.org/10.1109/CVPR.2016.308. Google Scholar
[37]	Chollet F. Xception: deep learning with depthwise separable convolutions[C]//30th IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1800–1807. https://doi.org/10.1109/CVPR.2017.195. Google Scholar
[38]	He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770–778. https://doi.org/10.1109/CVPR.2016.90. Google Scholar
[39]	Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2261–2269. https://doi.org/10.1109/CVPR.2017.243. Google Scholar
[40]	Wang X L, Girshick R, Gupta A, et al. Non-local neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 7794–7803. https://doi.org/10.1109/CVPR.2018.00813. Google Scholar
[41]	Hu J, Shen L, Albanie S, et al. Squeeze-and-excitation networks[J]. IEEE Trans Pattern Anal Mach Intell, 2020, 42(8): 2011−2023. doi: 10.1109/TPAMI.2019.2913372 CrossRef Google Scholar
[42]	Woo S, Park J, Lee J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of the 15th European Conference on Computer Vision, 2018: 3–19. https://doi.org/10.1007/978-3-030-01234-2_1. Google Scholar
[43]	Sun H, Zeng X X, Xu T, et al. Computer-aided diagnosis in histopathological images of the endometrium using a convolutional neural network and attention mechanisms[J]. IEEE J Biomed Health Inform, 2020, 24(6): 1664−1676. doi: 10.1109/JBHI.2019.2944977 CrossRef Google Scholar
[44]	Glotsos D, Kalatzis I, Spyridonos P, Kostopoulos S, Daskalakis A, Athanasiadis E, Ravazoula P, Nikiforidis G, and Cavouras D. Improving accuracy in astrocytomas grading by integrating a robust least squares mapping driven support vector machine classifier into a two level grade classification scheme[J]. Comput Methods Programs Biomed, 2008, 90(3): 251−261. doi: 10.1016/j.cmpb.2008.01.006 CrossRef Google Scholar

Overview

Overview

Breast cancer is the most common cancer among women. Tumor grading based on microscopic imaging is important for the diagnosis and prognosis of breast cancer, and the results need to be highly accurate and interpretable. Breast cancer tumor grading relies on pathologists to assess the morphological status of tissues and cells in the microscopic images of tissue sections, such as tissue differentiation, nuclear isotypes, and mitotic counts. A strong statistical correlation between the hematoxylin-Eosin (HE) stained microscopic imaging samples and the progesterone Receptor (ER) immunohistochemically (IHC) stained microscopic imaging samples has been documented, i.e., the ER status is strongly correlated with the tumor tissue grading. Therefore, it is a meaningful task to use deep learning models to research the breast tumor grading in ER IHC pathology images exploratively. At present, the CNN module integrating attention has a strong ability of induction bias but poor interpretability, while the Vision Transformer (ViT) block-based deep network has better interpretability but weaker ability of induction bias. In this paper, we propose an end-to-end deep network with adaptive model fusion by fusing ViT blocks and CNN blocks with integrated attention. Due to the negative fusion phenomenon of the existing model fusion methods, while it is impossible to guarantee that ViT blocks and CNN blocks with integrated attention have good feature representation capability at the same time; in addition, the high similarity and redundant information between the two feature representations lead to a poor model fusion capability. To this end, this paper proposes an adaptive model fusion method that includes multi-objective optimization, adaptive feature representation metric, and adaptive feature fusion, which effectively improves the fusion ability of the model. The accuracy of the model is 95.14%, which is 9.73% better than that of ViT-B/16 and 7.6% better than that of FABNet. The visualization of the model focuses more on the regions of nuclear heterogeneity (e.g., giant nuclei, polymorphic nuclei, multinuclei and dark nuclei), which is more consistent with the regions of interest to pathologists. Overall, the proposed model outperforms the current state-of-the-art model in terms of accuracy and interpretability.