基于深度学习的结直肠息肉内镜图像分割和分类方法比较

doi:10.3969/j.issn.1674-8115.2024.06.012

上海交通大学学报（医学版） ›› 2024, Vol. 44 ›› Issue (6): 762-772.doi: 10.3969/j.issn.1674-8115.2024.06.012

• 论著 · 技术与方法 • 上一篇下一篇

基于深度学习的结直肠息肉内镜图像分割和分类方法比较

陈健¹(), 王珍妮¹, 夏开建², 王甘红³, 刘罗杰¹(), 徐晓丹¹()

^1.江苏省常熟市第一人民医院（苏州大学附属常熟医院）消化内科，常熟 215500
^2.江苏省常熟市医学人工智能与大数据重点实验室，常熟 215500
^3.江苏省常熟市中医院消化内科，常熟 215500

收稿日期:2024-01-01 接受日期:2024-03-15 出版日期:2024-06-28 发布日期:2024-06-28
通讯作者: 刘罗杰,徐晓丹 E-mail:szcsdocter@gmail.com;Luojie13542@163.com;xxddocter@gmail.com
作者简介:陈健（1987—），男，副主任医师，硕士；电子信箱：szcsdocter@gmail.com。
基金资助:
苏州市第二十三批科技发展计划项目(SLT2023006);常熟市医学人工智能与大数据重点实验室能力提升项目(CYZ202301);苏州市护理学会科研项目(SZHL-B-202407)

Comparative study on methods for colon polyp endoscopic image segmentation and classification based on deep learning

CHEN Jian¹(), WANG Zhenni¹, XIA Kaijian², WANG Ganhong³, LIU Luojie¹(), XU Xiaodan¹()

^1.Department of Gastroenterology, Changshu No. 1 People's Hospital (Changshu Hospital Affiliated to Soochow University), Jiangsu Province, Changshu 215500, China
^2.Changshu Key Laboratory of Medical Artificial Intelligence and Big Data, Jiangsu Province, Changshu 215500, China
^3.Department of Gastroenterology, Changshu Traditional Chinese Medicine Hospital, Jiangsu Province, Changshu 215500, China

Received:2024-01-01 Accepted:2024-03-15 Online:2024-06-28 Published:2024-06-28
Contact: LIU Luojie,XU Xiaodan E-mail:szcsdocter@gmail.com;Luojie13542@163.com;xxddocter@gmail.com
Supported by:
Suzhou City's 23rd Science and Technology Development Plan(SLT2023006);Changshu City Key Laboratory of Medical Artificial Intelligence and Big Data Capability Enhancement Project(CYZ202301);Scientific Research Project of Suzhou Nursing Association(SZHL-B-202407)

摘要/Abstract

摘要：

目的·比较不同深度学习方法在结直肠息肉内镜图像分割和分类任务中的性能，以确定最优方法。方法·从3家医院采集4个结肠息肉数据集，涵盖1 534个静态图像及15个肠镜视频。所有样本均经病理学验证，分为锯齿状病变和腺瘤性息肉2类。使用LabelMe工具进行多边形标注，将标注结果转换为整数掩膜格式。数据用于训练不同架构的深度神经网络，包括卷积神经网络、Transformer以及这2种技术的融合，建立有效的语义分割模型。对比不同架构模型自动诊断结肠息肉的多项性能指标，包括平均交并比（mIoU）、整体准确率（aAcc）、平均准确率（mAcc）、平均Dice系数（mDice）、平均F分数（mFscore）、平均精确率（mPrecision）和平均召回率（mRecall）。结果·开发了4种不同架构的语义分割模型，包括2种深度卷积神经网络架构（Fast-SCNN和DeepLabV3plus）、1种Transformer架构（Segformer）以及1种混合架构（KNet）。在对291张测试图像进行综合性能评估中，KNet最高mIoU为84.59%，显著优于Fast-SCNN（75.32%）、DeepLabV3plus（78.63%）和Segformer（80.17%）。在“背景”“锯齿状病变”和“腺瘤性息肉”3个类别上，KNet的交并比（IoU）分别为98.91%、74.12%和80.73%，均超越其他模型。KNet在关键性能指标上也表现优异，其中aAcc、mAcc、mDice、mFscore和mRecall分别达到98.59%、91.24%、91.31%、91.31%和91.24%，均优于其他模型。尽管在mPrecision上，91.46%并非最突出，但KNet的整体性能仍领先。在80张外部测试图像的推理测试中，KNet保持了81.53%的mIoU，展现出良好的泛化能力。结论·利用基于KNet混合架构的深度神经网络构建的结直肠息肉内镜图像语义分割模型表现出优异的预测性能，具有成为检测结直肠息肉高效工具的潜力。

关键词: 深度学习, 结直肠息肉, 卷积神经网络, Transformer, 图像分割

Abstract:

Objective ·To compare the performance of various deep learning methods in the segmentation and classification of colorectal polyp endoscopic images, and identify the most effective approach. Methods ·Four colorectal polyp datasets were collected from three hospitals, encompassing 1 534 static images and 15 videos. All samples were pathologically validated and categorized into two types: serrated lesions and adenomatous polyps. Polygonal annotations were performed by using the LabelMe tool, and the annotated results were converted into integer mask formats. These data were utilized to train various architectures of deep neural networks, including convolutional neural network (CNN), Transformers, and their fusion, aiming to develop an effective semantic segmentation model. Multiple performance indicators for automatic diagnosis of colon polyps by different architecture models were compared, including mIoU, aAcc, mAcc, mDice, mFscore, mPrecision and mRecall. Results ·Four different architectures of semantic segmentation models were developed, including two deep CNN architectures (Fast-SCNN and DeepLabV3plus), one Transformer architecture (Segformer), and one hybrid architecture (KNet). In a comprehensive performance evaluation of 291 test images, KNet achieved the highest mIoU of 84.59%, significantly surpassing Fast-SCNN (75.32%), DeepLabV3plus (78.63%), and Segformer (80.17%). Across the categories of “background”， “serrated lesions” and “adenomatous polyps” , KNet's intersection over union (IoU) were 98.91%, 74.12%, and 80.73%, respectively, all exceeding other models. Additionally, KNet performed excellently in key performance metrics, with aAcc, mAcc, mDice, mFscore, and mRecall reaching 98.59%, 91.24%, 91.31%, 91.31%, and 91.24%, respectively, all superior to other models. Although its mPrecision of 91.46% was not the most outstanding, KNet's overall performance remained leading. In inference testing on 80 external test images, KNet maintained an mIoU of 81.53%, demonstrating strong generalization capabilities. Conclusion ·The semantic segmentation model of colorectal polyp endoscopic images constructed by deep neural network based on KNet hybrid architecture, exhibits superior predictive performance, demonstrating its potential as an efficient tool for detecting colorectal polyps.

Key words: deep learning, colorectal polyp, convolutional neural network, Transformer, image segmentation

中图分类号:

R574.6

陈健, 王珍妮, 夏开建, 王甘红, 刘罗杰, 徐晓丹. 基于深度学习的结直肠息肉内镜图像分割和分类方法比较[J]. 上海交通大学学报（医学版）, 2024, 44(6): 762-772.

CHEN Jian, WANG Zhenni, XIA Kaijian, WANG Ganhong, LIU Luojie, XU Xiaodan. Comparative study on methods for colon polyp endoscopic image segmentation and classification based on deep learning[J]. Journal of Shanghai Jiao Tong University (Medical Science), 2024, 44(6): 762-772.

图/表 9

图1 数据集中息肉图像示例Note： A. Serrated lesions. B. Adenomatous polyps.

Fig 1 Examples of polyp images from the datasets

图2 数据集图像分布情况Note： A. Distribution of image sizes in the dataset. B. Distribution of images across various categories.

Fig 2 Distribution of images in the datasets

图3 4种分割模型在训练集上的性能指标随训练进程的变化Note： A/B.Accuracy (A) and loss function (B) fluctuation of the Fast-SCNN model. C/D.Variations in accuracy (C) and loss function (D) in the DeepLabV3plus model. E/F. Segformer model's accuracy (E) and loss function (F) changes. G/H. KNet model's accuracy evolution (G) and loss function (H)alterations. All figures encapsulate the decoder segmentation accuracy (decode.acc_seg) and auxiliary classifier segmentation accuracy (aux.acc_seg).

Fig 3 Changes in performance metrics for four segmentation models in the training set throughout the training process

图4 4种分割模型在测试集上的性能指标随训练进程的变化Note： A. Fast-SCNN model. B. DeepLabV3plus model. C. Segformer model. D. KNet model.

Fig 4 Changes in performance metrics for four segmentation models in the test set throughout the training process

图5 不同模型在外部测试集上的预测结果对比Note： A/B. Adenomatous polyps. C/D. Serrated lesions. Green represents the predicted areas for adenomatous polyps, and red represents the predicted areas for serrated lesions.

Fig 5 Comparison of prediction results from different models in an external test set

图6 测试集上4种结肠息肉分割模型的平均分割性能指标对比

Fig 6 Comparative metrics of the average performance of four colon polyp segmentation models in the test set

表1 4种模型在测试集上的分类预测性能指标（%）

Tab 1 Performance metrics for classification predictions of the four models in the test dataset （%）

Model	Category	IoU	Acc	Dice	Fscore	Precision	Recall
Fast-SCNN	Background	97.74	99.02	98.86	98.86	98.70	99.02
	Serrated lesion	59.08	71.13	74.27	74.27	77.70	71.13
	Adenomatous polyp	69.15	81.24	81.76	81.76	82.30	81.24
DeepLabV3plus	Background	98.26	98.96	99.12	99.12	99.28	98.96
	Serrated lesion	63.63	83.38	77.78	77.78	72.88	83.38
	Adenomatous polyp	74.00	84.02	85.06	85.06	86.12	84.02
Segformer	Background	98.20	99.6	99.09	99.09	98.60	99.60
	Serrated lesion	67.22	71.95	80.39	80.39	91.09	71.95
	Adenomatous polyp	75.09	82.57	85.77	85.77	89.23	82.57
KNet	Background	98.91	99.50	99.45	99.45	99.40	99.50
	Serrated lesion	74.12	87.33	85.14	85.14	83.05	87.33
	Adenomatous polyp	80.73	86.88	89.34	89.34	91.94	86.88

图7 KNet-ColoSeg 模型在外部测试集上对新息肉图像的预测结果Note： This figure consists of four columns, including the original polyp images (A, E and I), the model's prediction output (B, F and J), a view of the model's predictions superimposed on the original images (C, G and K), and the confusion matrix for the prediction of individual images (D, H and L).

Fig 7 Prediction results of KNet-ColoSeg model for new polyp images in an external test dataset

图8 KNet-ColoSeg模型对一例视频案例的实时预测过程Note： The image sequence from left to right demonstrates a serrated lesion case successfully segmented by the AI model after irrigation with physiological saline.

Fig 8 KNet-ColoSeg model for real-time prediction of a video case

参考文献 30

1	GUNTER M J, ALHOMOUD S, ARNOLD M, et al. Meeting report from the joint IARC-NCI international cancer seminar series: a focus on colorectal cancer[J]. Ann Oncol, 2019, 30(4): 510-519.
2	中华人民共和国国家卫生健康委员会, 中华医学会肿瘤学分会. 中国结直肠癌诊疗规范(2023年版)[J]. 中华外科杂志, 2023, 61(8):617-644.
	National Health Commission of the People's Republic of China, Chinese Society of Oncology. Chinese protocol of diagnosis and treatment of colorectal cancer (2023 edition)[J]. Chinese Journal of Surgery, 2023, 61(8): 617-644.
3	GUPTA S, LIEBERMAN D, ANDERSON J C, et al. Recom-mendations for follow-up after colonoscopy and polypectomy: a consensus update by the US multi-society task force on colorectal cancer[J]. Gastrointest Endosc, 2020, 91(3): 463-485.e5.
4	LIEBERMAN D A, REX D K, WINAWER S J, et al. Guidelines for colonoscopy surveillance after screening and polypectomy: a consensus update by the US Multi-Society Task Force on Colorectal Cancer[J]. Gastroenterology, 2012, 143(3): 844-857.
5	BRETTHAUER M, KALAGER M, ADAMI H O. Do's and don'ts in evaluation of endoscopic screening for gastrointestinal cancers[J]. Endoscopy, 2016, 48(1): 75-80.
6	MORI Y, KUDO S E, BERZIN T M, et al. Computer-aided diagnosis for colonoscopy[J]. Endoscopy, 2017, 49(8): 813-819.
7	LEUFKENS A M, VAN OIJEN M G, VLEGGAAR F P, et al. Factors influencing the miss rate of polyps in a back-to-back colonoscopy study[J]. Endoscopy, 2012, 44(5): 470-475.
8	KIM N H, JUNG Y S, JEONG W S, et al. Miss rate of colorectal neoplastic polyps and risk factors for missed polyps in consecutive colonoscopies[J]. Intest Res, 2017, 15(3): 411-418.
9	RUTTER M D, BEINTARIS I, VALORI R, et al. World endoscopy organization consensus statements on post-colonoscopy and post-imaging colorectal cancer[J]. Gastroenterology, 2018, 155(3): 909-925.e3.
10	GONG E J, BANG C S, LEE J J, et al. No-code platform-based deep-learning models for prediction of colorectal polyp histology from white-light endoscopy images: development and performance verification[J]. J Pers Med, 2022, 12(6): 963.
11	GARCÍA-RODRÍGUEZ A, TUDELA Y, CÓRDOVA H, et al. In vivo computer-aided diagnosis of colorectal polyps using white light endoscopy[J]. Endosc Int Open, 2022, 10(9): E1201-E1207.
12	WANG P, LIU P, GLISSEN BROWN J R, et al. Lower adenoma miss rate of computer-aided detection-assisted colonoscopy vs routine white-light colonoscopy in a prospective tandem study[J]. Gastroenterology, 2020, 159(4): 1252-1261.e5.
13	HASSAN C, EAST J, RADAELLI F, et al. Bowel preparation for colonoscopy: European Society of Gastrointestinal Endoscopy (ESGE) Guideline-Update 2019[J]. Endoscopy, 2019, 51(8): 775-794.
14	RUSSELL B C, TORRALBA A, MURPHY K P, et al. LabelMe: a database and web-based tool for image annotation[J]. Int J Comput Vis, 2008, 77(1): 157-173.
15	URBAN G, TRIPATHI P, ALKAYALI T, et al. Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy[J]. Gastroenterology, 2018, 155(4): 1069-1078.e8.
16	CHEUNG T H, YEUNG D Y. A survey of automated data augmentation for image classification: learning to compose, mix, and generate[J]. IEEE Trans Neural Netw Learn Syst, 2023. DOI: 10. 1109/TNNLS.2023.3282258.
17	SHIN H C, ROTH H R, GAO M, et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning[J]. IEEE Trans Med Imaging, 2016, 35(5): 1285-1298.
18	LESLIE A, CAREY F A, PRATT N R, et al. The colorectal adenoma-carcinoma sequence[J]. Br J Surg, 2002, 89(7): 845-860.
19	ZHOU Y J, LU X F, CHEN H, et al. Single-cell transcriptomics reveals early molecular and immune alterations underlying the serrated neoplasia pathway toward colorectal cancer[J]. Cell Mol Gastroenterol Hepatol, 2023, 15(2): 393-424.
20	GOTO H, ODA Y, MURAKAMI Y, et al. Proportion of de novo cancers among colorectal cancers in Japan[J]. Gastroenterology, 2006, 131(1): 40-46.
21	KIM K M, LEE E J, HA S, et al. Molecular features of colorectal hyperplastic polyps and sessile serrated adenoma/polyps from Korea[J]. Am J Surg Pathol, 2011, 35(9): 1274-1286.
22	LUI R N, SUNG J J Y. Sessile serrated adenoma/polyps: why we should be working flat out to understand more about these flat lesions?[J]. J Gastroenterol Hepatol, 2019, 34(10): 1667-1668.
23	NAGTEGAAL I D, ODZE R D, KLIMSTRA D, et al. The 2019 WHO classification of tumours of the digestive system[J]. Histopathology, 2020, 76(2): 182-188.
24	JHA D, SMEDSRUD P, RIEGLER M, et al. ResUNet++: an advanced architecture for medical image segmentation[R]. San Diego: IEEE International Symposium on Multimedia, 2019.
25	CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Trans Pattern Anal Mach Intell, 2018, 40(4): 834-848.
26	HSU C M, HSU C C, HSU Z M, et al. Colorectal polyp image detection and classification through grayscale images and deep learning[J]. Sensors (Basel), 2021, 21(18): 5995.
27	LO C M, YEH Y H, TANG J H, et al. Rapid polyp classification in colonoscopy using textural and convolutional features[J]. Healthcare (Basel), 2022, 10(8): 1494.
28	KRENZER A, HEIL S, FITTING D, et al. Automated classification of polyps using deep learning architectures and few-shot learning[J]. BMC Med Imaging, 2023, 23(1): 59.
29	LIU Z, LV Q, YANG Z, et al. Recent progress in transformer-based medical image analysis[J]. Comput Biol Med, 2023, 164: 107268.
30	ZHANG W, PANG J, CHEN K, et al. K-Net: towards unified image segmentation [R]. Montreal: Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021), 2021.

[1]	赵珂珂, 蒋蓓蓓, 张璐, 王凌云, 张亚平, 解学乾. 超低剂量平扫CT深度学习图像重建评价肺部病灶的可行性[J]. 上海交通大学学报（医学版）, 2022, 42(8): 1062-1069.
[2]	林嘉希, 汪盛嘉, 赵鑫, 高欣, 殷民月, 朱锦舟. 基于深度卷积神经网络的Barrett食管内镜图片分类模型的建立[J]. 上海交通大学学报（医学版）, 2022, 42(5): 653-659.
[3]	陈立奇, 薛卓维, 吴氢凯. 基于磁共振成像的女性盆底器官三维数字模型重建的研究进展[J]. 上海交通大学学报（医学版）, 2022, 42(3): 381-386.
[4]	王晓峰, 周璐, 姚乐宇, 何凡, 彭海霞, 杨大明, 黄晓霖. 深度卷积神经网络模型辅助下结直肠息肉检测系统对初级医师结直肠小息肉检出率的影响[J]. 上海交通大学学报(医学版), 2022, 42(2): 205-210.
[5]	李小敏, 曲扬, 张少霆, 赵亮, 刘畅, 谢帅宁, 戴尅戎, 艾松涛. 人工智能技术在骨肌系统影像学方面的应用[J]. 上海交通大学学报(医学版), 2021, 41(2): 262-266.
[6]	陆言巧，沈兰，何奔. 人工智能在心血管疾病的辅助诊疗中的应用[J]. 上海交通大学学报（医学版）, 2020, 40(2): 259-.
[7]	蒋蓓蓓，张亚平，张琳，刘桂雪，解学乾. 深度卷积神经网络对≤ 3 cm的亚实性肺腺癌 CT图像病理学分型预测的可视化研究[J]. 上海交通大学学报（医学版）, 2019, 39(9): 1045-.

基于深度学习的结直肠息肉内镜图像分割和分类方法比较

Comparative study on methods for colon polyp endoscopic image segmentation and classification based on deep learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 30

相关文章 7

编辑推荐

Metrics