基于深度卷积神经网络的Barrett食管内镜图片分类模型的建立

doi:10.3969/j.issn.1674-8115.2022.05.014

上海交通大学学报（医学版） ›› 2022, Vol. 42 ›› Issue (5): 653-659.doi: 10.3969/j.issn.1674-8115.2022.05.014

• 论著 · 技术与方法 • 上一篇下一篇

基于深度卷积神经网络的Barrett食管内镜图片分类模型的建立

林嘉希¹^,²(), 汪盛嘉³, 赵鑫³, 高欣¹^,², 殷民月¹^,², 朱锦舟¹^,²()

^1.苏州大学附属第一医院消化内科，苏州 215006
^2.江苏省苏州市消化病临床医学中心，苏州 215006
^3.苏州大学附属第一医院普外科，苏州 215006

收稿日期:2022-01-07 接受日期:2022-03-25 出版日期:2022-05-28 发布日期:2022-05-07
通讯作者: 朱锦舟 E-mail:jxlin@stu.suda.edu.cn;jzzhu@zju.edu.cn
作者简介:林嘉希（1999—），男，本科生；电子信箱：jxlin@stu.suda.edu.cn。
基金资助:
国家自然科学基金(82000540);苏州市“科教兴卫”青年科技项目(KJXW2019001)

Development of endoscopic image classification models of Barrett's esophagus based on deep convolutional neural networks

LIN Jiaxi¹^,²(), WANG Shengjia³, ZHAO Xin³, GAO Xin¹^,², YIN Minyue¹^,², ZHU Jinzhou¹^,²()

^1.Department of Gastroenterology, The First Affiliated Hospital of Soochow University, Suzhou 215006, China
^2.Suzhou Clinical Center of Digestive Disease, Jiangsu Province, Suzhou 215006, China
^3.Department of General Surgery, The First Affiliated Hospital of Soochow University, Suzhou 215006, China

Received:2022-01-07 Accepted:2022-03-25 Online:2022-05-28 Published:2022-05-07
Contact: ZHU Jinzhou E-mail:jxlin@stu.suda.edu.cn;jzzhu@zju.edu.cn
Supported by:
National Natural Science Foundation of China(82000540);Youth Program of Suzhou Health Committee(KJXW2019001)

摘要/Abstract

摘要：

目的·利用深度卷积神经网络算法，构建Barrett食管内镜图片分类模型并评估其分类能力。方法·收集苏州大学附属第一医院消化内镜中心及HyperKvasir数据库中的内镜下食管图片共806张，其中正常食管图片412张、Barrett食管图片394张。随机将所有图片分为训练集（85%）与验证集（15%）。利用于ImageNet数据库进行预训练的4种深度卷积神经网络［Xception、NASNet Large（NASNetL）、ResNet50V2（ResNet）及BigTransfer（BiT）］，分别在训练集中进行迁移学习，建立Barrett食管内镜图片分类模型，并使用梯度加权分类激活映射对该4个模型的分类结果进行可视化解释。随后，于验证集中评价模型的分类能力。同时，收集高、低年资医师对验证集数据的分类结果，将其与模型的分类结果进行对比，进一步评估模型的分类能力。结果·成功构建了基于深度卷积神经网络的Barrett食管内镜图片的4个分类模型。利用梯度加权分类激活映射，以热力图形式实现了对模型分类结果的可视化解释。在验证集数据中，各模型均拥有较高的分类准确性与精确性，其平均分类准确性为0.852，平均分类精确性为0.846。NASNetL模型相较其余3种模型，拥有最高分类准确性（0.873）及最高分类精确性（0.867），是表现最优的模型。该模型对Barrett食管内镜图片的分类能力近似高年资医师，其分类准确性略低于高年资医师（0.881）而高于低年资医师（0.855）；同时，该模型与高年资医师（Kappa=0.712，P=0.000）、低年资医师（Kappa=0.695，P=0.000）均具有较好的分类一致性。结论·利用深度卷积神经网络迁移学习构建的Barrett食管内镜图片分类模型具有较好的分类能力。

关键词: Barrett食管, 深度学习, 迁移学习, 消化内镜, 卷积神经网络

Abstract:

Objective·To develop classification models for endoscopic images of Barrett's esophagus using deep convolutional neural network, and evaluate its classification performance.

Methods·A total of 806 esophageal images were collected from Gastrointestinal Endoscopy Center of the First Affiliated Hospital of Soochow University and the HyperKvasir database, including 412 normal esophageal images and 394 Barrett's esophagus images. All the images were randomly divided into training set (85%) and validation set (15%). Four deep convolutional neural networks [Xception, NASNet Large (NASNetL), ResNet50V2 (ResNet) and BigTransfer (BiT)], which had been pre-trained in the ImageNet database, were used to perform transfer learning in the training set and develop endoscopic image classification models of Barrett's esophagus, respectively. The gradient-weighted class activation mapping (Grad-CAM) was used to visually interpret the classification results of the four models. In the validation set, the models' classification abilities were evaluated. Besides, to further evaluate the classification abilities of the models, the classification results of the validation set by senior and junior physicians were compared with the models'.

Results·The 4 endoscopic image classification models of Barrett's esophagus based on deep convolutional neural networks were successfully developed. The visual interpretation of the models' classification results was presented in the form of a heat map by using Grad-CAM. In the validation set, all models presented high classification accuracy (average accuracy = 0.852) and high classification precision (average precision = 0.846). Compared with the other three models, the NASNetL model had the highest classification accuracy (0.873) and the highest classification precision (0.867), and was the best model. The NASNetL model showed similar classification ability to the senior physician. Its classification accuracy was slightly lower than that of senior physician (0.881), and higher than that of junior physician (0.855). Meanwhile, it had good classification consistency with both senior physician (Kappa=0.712, P=0.000) and junior physician (Kappa=0.695, P=0.000).

Conclusion·The endoscopic image classification models of Barrett's esophagus, developed by deep convolutional neural networks transfer learning, show good classification ability.

Key words: Barrett's esophagus, deep learning, transfer learning, gastrointestinal endoscopy, convolutional neural network

中图分类号:

R571

林嘉希, 汪盛嘉, 赵鑫, 高欣, 殷民月, 朱锦舟. 基于深度卷积神经网络的Barrett食管内镜图片分类模型的建立[J]. 上海交通大学学报（医学版）, 2022, 42(5): 653-659.

LIN Jiaxi, WANG Shengjia, ZHAO Xin, GAO Xin, YIN Minyue, ZHU Jinzhou. Development of endoscopic image classification models of Barrett's esophagus based on deep convolutional neural networks[J]. Journal of Shanghai Jiao Tong University (Medical Science), 2022, 42(5): 653-659.

图/表 5

图 1 Barrett食管内镜图片分类模型的迁移学习示意图Note：Gray solid nodes represent deep convolutional neural networks' feature extraction layers which retain the ImageNet pre-training weights; Blue solid nodes represent deep convolutional neural networks' full connected layers with 1 000 classified results; Red solid nodes represent deep convolutional neural networks' full connected layers with 2 classified results.

Fig 1 Schematic diagram of transfer learning of endoscopic image classification models of Barrett's esophagus

表 1 Barrett食管内镜图片分类模型在验证集中的分类能力分析

Tab 1 Analysis of classification ability of endoscopic image classification models of Barrett's esophagus in the validation set

Model	Confusion matrix				Evaluation Parameter
Xception			Model predicted label		Accuracy	0.805
	True label		Normal	Barrett	Recall	0.814
		Normal	47	12	Precision	0.800
					F1-Score	0.807
		Barrett	11	48	AUC	0.805
NASNetL			Model predicted label		Accuracy	0.873
	True label		Normal	Barrett	Recall	0.881
		Normal	51	8	Precision	0.867
					F1-Score	0.874
		Barrett	7	52	AUC	0.873
ResNet			Model predicted label		Accuracy	0.864
	True label		Normal	Barrett	Recall	0.864
		Normal	51	8	Precision	0.864
					F1-Score	0.864
		Barrett	8	51	AUC	0.864
BiT			Model predicted label		Accuracy	0.864
	True label		Normal	Barrett	Recall	0.881
		Normal	50	9	Precision	0.852
					F1-Score	0.867
		Barrett	7	52	AUC	0.864

表 2 高、低年资医师在验证集中的分类能力分析

Tab 2 Analysis of classification ability of the senior and junior physicians in the validation set

Physician	Confusion matrix				Evaluation Parameter
Junior physician			Physician predicted label		Accuracy	0.855
	True label		Normal	Barrett	Recall	0.847
		Normal	49	10	Precision	0.850
					F1-Score	0.857
		Barrett	9	50	AUC	0.855
Senior physician			Physician predicted label		Accuracy	0.881
	True label		Normal	Barrett	Recall	0.898
		Normal	51	8	Precision	0.869
					F1-Score	0.883
		Barrett	6	53	AUC	0.881

表 3 Barrett食管内镜图片分类模型与高、低年资医师于验证集中的分类结果的一致性检验

Tab 3 Consistency test of classification results between of endoscopic image classification models of Barrett's esophagus with senior and junior physicians in the validation set

Variable	Kappa value
Variable	Junior physician	Senior physician	Xception	NASNetL	ResNet	BiT
Junior physician	1.000	0.847	0.763	0.695	0.610	0.644
Senior physician	‒	1.000	0.780	0.712	0.661	0.661
Xception	‒	‒	1.000	0.695	0.678	0.712
NASNetL	‒	‒	‒	1.000	0.746	0.712
ResNet	‒	‒	‒	‒	1.000	0.763
BiT	‒	‒	‒	‒	‒	1.000

图 2 Barrett食管内镜图片分类模型的可视化解释Note：A. Endoscopic manifestation of Barrett's esophagus. B. Classification activation map of Barrett's esophagus. Red marked parts represent the area of Barrett's esophageal lesion judged by models; Blue marked parts represent the area of normal esophageal tissue judged by models.

Fig 2 Visual interpretation of endoscopic image classification models of Barrett's esophagus

参考文献 17

1	郑荣寿, 孙可欣, 张思维, 等. 2015年中国恶性肿瘤流行情况分析[J]. 中华肿瘤杂志, 2019, 41(1): 19-28.
	ZHENG R S, SUN K X, ZHANG S W, et al. Report of cancer epidemiology in China, 2015[J]. Chin J Oncol, 2019, 41(1): 19-28.
2	NUCCI D, MARINO A, REALDON S, et al. Lifestyle, WCRF/AICR recommendations, and esophageal adenocarcinoma risk: a systematic review of the literature[J]. Nutrients, 2021, 13(10): 3525.
3	ISHIHARA R, GODA K, OYAMA T. Endoscopic diagnosis and treatment of esophageal adenocarcinoma: introduction of Japan Esophageal Society classification of Barrett's esophagus[J]. J Gastroenterol, 2019, 54(1): 1-9.
4	HAMADE N, SHARMA P. Artificial intelligence in Barrett's esophagus[J]. Ther Adv Gastrointest Endosc, 2021, 14: 26317745211049964.
5	SHAHEEN N J, FALK G W, IYER P G, et al. ACG clinical guideline: diagnosis and management of Barrett's esophagus[J]. Am J Gastroenterol, 2016, 111(1): 30-50.
6	MANN R, GAJENDRAN M, PERISETTI A, et al. Advanced endoscopic imaging and interventions in GERD: an update and future directions[J]. Front Med (Lausanne), 2021, 8: 728696.
7	PAN W, LI X J, WANG W J, et al. Identification of Barrett's esophagus in endoscopic images using deep learning[J]. BMC Gastroenterol, 2021, 21(1): 479.
8	BORGLI H, THAMBAWITA V, SMEDSRUD P H, et al. HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy[J]. Sci Data, 2020, 7(1): 283.
9	CHOLLET F. Xception: deep learning with depthwise separable convolutions[J/OL]. (2017-09-09)[2021-12-28]. http://arxiv.org/abs/1610.02357.
10	HE K, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[J/OL]. (2015-12-10)[2021-12-28]. http://arxiv.org/abs/1512.03385.
11	KOLESNIKOV A, BEYER L, ZHAI X H, et al. Big transfer (BiT): general visual representation learning[J/OL]. (2020-05-05)[2021-12-28]. http://arxiv.org/abs/1912.11370.
12	ZOPH B, VASUDEVAN V, SHLENS J, et al. Learning transferable architectures for scalable image recognition[J/OL]. (2017-12-07)[2021-12-30]. http://arxiv.org/abs/1707.07012v4.
13	HONG J S, PARK B Y, PARK H. Convolutional neural network classifier for distinguishing Barrett's esophagus and neoplasia endomicroscopy images[J]. Annu Int Conf IEEE Eng Med Biol Soc, 2017, 2017: 2892-2895.
14	DE GROOF A J, STRUYVENBERG M R, VAN DER PUTTEN J, et al. Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking[J]. Gastroenterology, 2020, 158(4): 915-929.e4.
15	ALOM M Z, TAHA T M, YAKOPCIC C, et al. The history began from AlexNet: a comprehensive survey on deep learning approaches[J/OL]. (2018-09-12)[2021-12-27]. http://arxiv.org/abs/1803.01164.
16	SHIN H C, ROTH H R, GAO M C, et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning[J]. IEEE Trans Med Imaging, 2016, 35(5): 1285-1298.
17	SAMALA R K, CHAN H P, HADJIISKI L, et al. Breast cancer diagnosis in digital breast tomosynthesis: effects of training sample size on multi-stage transfer learning using deep neural nets[J]. IEEE Trans Med Imaging, 2019, 38(3): 686-696.

[1]	陈立奇, 薛卓维, 吴氢凯. 基于磁共振成像的女性盆底器官三维数字模型重建的研究进展[J]. 上海交通大学学报(医学版), 2022, 42(3): 381-386.
[2]	王晓峰, 周璐, 姚乐宇, 何凡, 彭海霞, 杨大明, 黄晓霖. 深度卷积神经网络模型辅助下结直肠息肉检测系统对初级医师结直肠小息肉检出率的影响[J]. 上海交通大学学报(医学版), 2022, 42(2): 205-210.
[3]	李小敏, 曲扬, 张少霆, 赵亮, 刘畅, 谢帅宁, 戴尅戎, 艾松涛. 人工智能技术在骨肌系统影像学方面的应用[J]. 上海交通大学学报(医学版), 2021, 41(2): 262-266.
[4]	陆言巧，沈兰，何奔. 人工智能在心血管疾病的辅助诊疗中的应用[J]. 上海交通大学学报（医学版）, 2020, 40(2): 259-.
[5]	蒋蓓蓓，张亚平，张琳，刘桂雪，解学乾. 深度卷积神经网络对≤ 3 cm的亚实性肺腺癌 CT图像病理学分型预测的可视化研究[J]. 上海交通大学学报（医学版）, 2019, 39(9): 1045-.
[6]	沈煜枫 1，戴军 1，李晓波 2，戈之铮 2. 非放大内镜下窄带成像技术结合 NICE分型在诊断结直肠病变中的应用[J]. 上海交通大学学报（医学版）, 2019, 39(2): 176-.
[7]	尹霞, 徐艳丽, 周隽, 等. Barrett食管症状、内镜下分型与病理特征探讨[J]. , 2013, 33(1): 50-.

基于深度卷积神经网络的Barrett食管内镜图片分类模型的建立

Development of endoscopic image classification models of Barrett's esophagus based on deep convolutional neural networks

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 5

参考文献 17

相关文章 7

编辑推荐

Metrics