Journal of Shanghai Jiao Tong University (Medical Science) >
Comparative study on methods for colon polyp endoscopic image segmentation and classification based on deep learning
Received date: 2024-01-01
Accepted date: 2024-03-15
Online published: 2024-06-28
Supported by
Suzhou City's 23rd Science and Technology Development Plan(SLT2023006);Changshu City Key Laboratory of Medical Artificial Intelligence and Big Data Capability Enhancement Project(CYZ202301);Scientific Research Project of Suzhou Nursing Association(SZHL-B-202407)
Objective ·To compare the performance of various deep learning methods in the segmentation and classification of colorectal polyp endoscopic images, and identify the most effective approach. Methods ·Four colorectal polyp datasets were collected from three hospitals, encompassing 1 534 static images and 15 videos. All samples were pathologically validated and categorized into two types: serrated lesions and adenomatous polyps. Polygonal annotations were performed by using the LabelMe tool, and the annotated results were converted into integer mask formats. These data were utilized to train various architectures of deep neural networks, including convolutional neural network (CNN), Transformers, and their fusion, aiming to develop an effective semantic segmentation model. Multiple performance indicators for automatic diagnosis of colon polyps by different architecture models were compared, including mIoU, aAcc, mAcc, mDice, mFscore, mPrecision and mRecall. Results ·Four different architectures of semantic segmentation models were developed, including two deep CNN architectures (Fast-SCNN and DeepLabV3plus), one Transformer architecture (Segformer), and one hybrid architecture (KNet). In a comprehensive performance evaluation of 291 test images, KNet achieved the highest mIoU of 84.59%, significantly surpassing Fast-SCNN (75.32%), DeepLabV3plus (78.63%), and Segformer (80.17%). Across the categories of “background”, “serrated lesions” and “adenomatous polyps” , KNet's intersection over union (IoU) were 98.91%, 74.12%, and 80.73%, respectively, all exceeding other models. Additionally, KNet performed excellently in key performance metrics, with aAcc, mAcc, mDice, mFscore, and mRecall reaching 98.59%, 91.24%, 91.31%, 91.31%, and 91.24%, respectively, all superior to other models. Although its mPrecision of 91.46% was not the most outstanding, KNet's overall performance remained leading. In inference testing on 80 external test images, KNet maintained an mIoU of 81.53%, demonstrating strong generalization capabilities. Conclusion ·The semantic segmentation model of colorectal polyp endoscopic images constructed by deep neural network based on KNet hybrid architecture, exhibits superior predictive performance, demonstrating its potential as an efficient tool for detecting colorectal polyps.
Jian CHEN , Zhenni WANG , Kaijian XIA , Ganhong WANG , Luojie LIU , Xiaodan XU . Comparative study on methods for colon polyp endoscopic image segmentation and classification based on deep learning[J]. Journal of Shanghai Jiao Tong University (Medical Science), 2024 , 44(6) : 762 -772 . DOI: 10.3969/j.issn.1674-8115.2024.06.012
1 | GUNTER M J, ALHOMOUD S, ARNOLD M, et al. Meeting report from the joint IARC-NCI international cancer seminar series: a focus on colorectal cancer[J]. Ann Oncol, 2019, 30(4): 510-519. |
2 | 中华人民共和国国家卫生健康委员会, 中华医学会肿瘤学分会. 中国结直肠癌诊疗规范(2023年版)[J]. 中华外科杂志, 2023, 61(8):617-644. |
2 | National Health Commission of the People's Republic of China, Chinese Society of Oncology. Chinese protocol of diagnosis and treatment of colorectal cancer (2023 edition)[J]. Chinese Journal of Surgery, 2023, 61(8): 617-644. |
3 | GUPTA S, LIEBERMAN D, ANDERSON J C, et al. Recom-mendations for follow-up after colonoscopy and polypectomy: a consensus update by the US multi-society task force on colorectal cancer[J]. Gastrointest Endosc, 2020, 91(3): 463-485.e5. |
4 | LIEBERMAN D A, REX D K, WINAWER S J, et al. Guidelines for colonoscopy surveillance after screening and polypectomy: a consensus update by the US Multi-Society Task Force on Colorectal Cancer[J]. Gastroenterology, 2012, 143(3): 844-857. |
5 | BRETTHAUER M, KALAGER M, ADAMI H O. Do's and don'ts in evaluation of endoscopic screening for gastrointestinal cancers[J]. Endoscopy, 2016, 48(1): 75-80. |
6 | MORI Y, KUDO S E, BERZIN T M, et al. Computer-aided diagnosis for colonoscopy[J]. Endoscopy, 2017, 49(8): 813-819. |
7 | LEUFKENS A M, VAN OIJEN M G, VLEGGAAR F P, et al. Factors influencing the miss rate of polyps in a back-to-back colonoscopy study[J]. Endoscopy, 2012, 44(5): 470-475. |
8 | KIM N H, JUNG Y S, JEONG W S, et al. Miss rate of colorectal neoplastic polyps and risk factors for missed polyps in consecutive colonoscopies[J]. Intest Res, 2017, 15(3): 411-418. |
9 | RUTTER M D, BEINTARIS I, VALORI R, et al. World endoscopy organization consensus statements on post-colonoscopy and post-imaging colorectal cancer[J]. Gastroenterology, 2018, 155(3): 909-925.e3. |
10 | GONG E J, BANG C S, LEE J J, et al. No-code platform-based deep-learning models for prediction of colorectal polyp histology from white-light endoscopy images: development and performance verification[J]. J Pers Med, 2022, 12(6): 963. |
11 | GARCíA-RODRíGUEZ A, TUDELA Y, CóRDOVA H, et al. In vivo computer-aided diagnosis of colorectal polyps using white light endoscopy[J]. Endosc Int Open, 2022, 10(9): E1201-E1207. |
12 | WANG P, LIU P, GLISSEN BROWN J R, et al. Lower adenoma miss rate of computer-aided detection-assisted colonoscopy vs routine white-light colonoscopy in a prospective tandem study[J]. Gastroenterology, 2020, 159(4): 1252-1261.e5. |
13 | HASSAN C, EAST J, RADAELLI F, et al. Bowel preparation for colonoscopy: European Society of Gastrointestinal Endoscopy (ESGE) Guideline-Update 2019[J]. Endoscopy, 2019, 51(8): 775-794. |
14 | RUSSELL B C, TORRALBA A, MURPHY K P, et al. LabelMe: a database and web-based tool for image annotation[J]. Int J Comput Vis, 2008, 77(1): 157-173. |
15 | URBAN G, TRIPATHI P, ALKAYALI T, et al. Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy[J]. Gastroenterology, 2018, 155(4): 1069-1078.e8. |
16 | CHEUNG T H, YEUNG D Y. A survey of automated data augmentation for image classification: learning to compose, mix, and generate[J]. IEEE Trans Neural Netw Learn Syst, 2023. DOI: 10. 1109/TNNLS.2023.3282258. |
17 | SHIN H C, ROTH H R, GAO M, et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning[J]. IEEE Trans Med Imaging, 2016, 35(5): 1285-1298. |
18 | LESLIE A, CAREY F A, PRATT N R, et al. The colorectal adenoma-carcinoma sequence[J]. Br J Surg, 2002, 89(7): 845-860. |
19 | ZHOU Y J, LU X F, CHEN H, et al. Single-cell transcriptomics reveals early molecular and immune alterations underlying the serrated neoplasia pathway toward colorectal cancer[J]. Cell Mol Gastroenterol Hepatol, 2023, 15(2): 393-424. |
20 | GOTO H, ODA Y, MURAKAMI Y, et al. Proportion of de novo cancers among colorectal cancers in Japan[J]. Gastroenterology, 2006, 131(1): 40-46. |
21 | KIM K M, LEE E J, HA S, et al. Molecular features of colorectal hyperplastic polyps and sessile serrated adenoma/polyps from Korea[J]. Am J Surg Pathol, 2011, 35(9): 1274-1286. |
22 | LUI R N, SUNG J J Y. Sessile serrated adenoma/polyps: why we should be working flat out to understand more about these flat lesions?[J]. J Gastroenterol Hepatol, 2019, 34(10): 1667-1668. |
23 | NAGTEGAAL I D, ODZE R D, KLIMSTRA D, et al. The 2019 WHO classification of tumours of the digestive system[J]. Histopathology, 2020, 76(2): 182-188. |
24 | JHA D, SMEDSRUD P, RIEGLER M, et al. ResUNet++: an advanced architecture for medical image segmentation[R]. San Diego: IEEE International Symposium on Multimedia, 2019. |
25 | CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Trans Pattern Anal Mach Intell, 2018, 40(4): 834-848. |
26 | HSU C M, HSU C C, HSU Z M, et al. Colorectal polyp image detection and classification through grayscale images and deep learning[J]. Sensors (Basel), 2021, 21(18): 5995. |
27 | LO C M, YEH Y H, TANG J H, et al. Rapid polyp classification in colonoscopy using textural and convolutional features[J]. Healthcare (Basel), 2022, 10(8): 1494. |
28 | KRENZER A, HEIL S, FITTING D, et al. Automated classification of polyps using deep learning architectures and few-shot learning[J]. BMC Med Imaging, 2023, 23(1): 59. |
29 | LIU Z, LV Q, YANG Z, et al. Recent progress in transformer-based medical image analysis[J]. Comput Biol Med, 2023, 164: 107268. |
30 | ZHANG W, PANG J, CHEN K, et al. K-Net: towards unified image segmentation [R]. Montreal: Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021), 2021. |
/
〈 |
|
〉 |