Development of endoscopic image classification models of Barrett's esophagus based on deep convolutional neural networks

doi:10.3969/j.issn.1674-8115.2022.05.014

Abstract

Abstract: Objective

·To develop classification models for endoscopic images of Barrett's esophagus using deep convolutional neural network, and evaluate its classification performance.

Methods

·A total of 806 esophageal images were collected from Gastrointestinal Endoscopy Center of the First Affiliated Hospital of Soochow University and the HyperKvasir database, including 412 normal esophageal images and 394 Barrett's esophagus images. All the images were randomly divided into training set (85%) and validation set (15%). Four deep convolutional neural networks [Xception, NASNet Large (NASNetL), ResNet50V2 (ResNet) and BigTransfer (BiT)], which had been pre-trained in the ImageNet database, were used to perform transfer learning in the training set and develop endoscopic image classification models of Barrett's esophagus, respectively. The gradient-weighted class activation mapping (Grad-CAM) was used to visually interpret the classification results of the four models. In the validation set, the models' classification abilities were evaluated. Besides, to further evaluate the classification abilities of the models, the classification results of the validation set by senior and junior physicians were compared with the models'.

Results

·The 4 endoscopic image classification models of Barrett's esophagus based on deep convolutional neural networks were successfully developed. The visual interpretation of the models' classification results was presented in the form of a heat map by using Grad-CAM. In the validation set, all models presented high classification accuracy (average accuracy = 0.852) and high classification precision (average precision = 0.846). Compared with the other three models, the NASNetL model had the highest classification accuracy (0.873) and the highest classification precision (0.867), and was the best model. The NASNetL model showed similar classification ability to the senior physician. Its classification accuracy was slightly lower than that of senior physician (0.881), and higher than that of junior physician (0.855). Meanwhile, it had good classification consistency with both senior physician (Kappa=0.712, P=0.000) and junior physician (Kappa=0.695, P=0.000).

Conclusion

·The endoscopic image classification models of Barrett's esophagus, developed by deep convolutional neural networks transfer learning, show good classification ability.

Key words: Barrett's esophagus, deep learning, transfer learning, gastrointestinal endoscopy, convolutional neural network

CLC Number:

R571

LIN Jiaxi, WANG Shengjia, ZHAO Xin, GAO Xin, YIN Minyue, ZHU Jinzhou. Development of endoscopic image classification models of Barrett's esophagus based on deep convolutional neural networks[J]. Journal of Shanghai Jiao Tong University (Medical Science), 2022, 42(5): 653-659.

Figures/Tables 5

TrendMD

References 17

1	郑荣寿, 孙可欣, 张思维, 等. 2015年中国恶性肿瘤流行情况分析[J]. 中华肿瘤杂志, 2019, 41(1): 19-28.
	ZHENG R S, SUN K X, ZHANG S W, et al. Report of cancer epidemiology in China, 2015[J]. Chin J Oncol, 2019, 41(1): 19-28.
2	NUCCI D, MARINO A, REALDON S, et al. Lifestyle, WCRF/AICR recommendations, and esophageal adenocarcinoma risk: a systematic review of the literature[J]. Nutrients, 2021, 13(10): 3525.
3	ISHIHARA R, GODA K, OYAMA T. Endoscopic diagnosis and treatment of esophageal adenocarcinoma: introduction of Japan Esophageal Society classification of Barrett's esophagus[J]. J Gastroenterol, 2019, 54(1): 1-9.
4	HAMADE N, SHARMA P. Artificial intelligence in Barrett's esophagus[J]. Ther Adv Gastrointest Endosc, 2021, 14: 26317745211049964.
5	SHAHEEN N J, FALK G W, IYER P G, et al. ACG clinical guideline: diagnosis and management of Barrett's esophagus[J]. Am J Gastroenterol, 2016, 111(1): 30-50.
6	MANN R, GAJENDRAN M, PERISETTI A, et al. Advanced endoscopic imaging and interventions in GERD: an update and future directions[J]. Front Med (Lausanne), 2021, 8: 728696.
7	PAN W, LI X J, WANG W J, et al. Identification of Barrett's esophagus in endoscopic images using deep learning[J]. BMC Gastroenterol, 2021, 21(1): 479.
8	BORGLI H, THAMBAWITA V, SMEDSRUD P H, et al. HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy[J]. Sci Data, 2020, 7(1): 283.
9	CHOLLET F. Xception: deep learning with depthwise separable convolutions[J/OL]. (2017-09-09)[2021-12-28]. http://arxiv.org/abs/1610.02357.
10	HE K, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[J/OL]. (2015-12-10)[2021-12-28]. http://arxiv.org/abs/1512.03385.
11	KOLESNIKOV A, BEYER L, ZHAI X H, et al. Big transfer (BiT): general visual representation learning[J/OL]. (2020-05-05)[2021-12-28]. http://arxiv.org/abs/1912.11370.
12	ZOPH B, VASUDEVAN V, SHLENS J, et al. Learning transferable architectures for scalable image recognition[J/OL]. (2017-12-07)[2021-12-30]. http://arxiv.org/abs/1707.07012v4.
13	HONG J S, PARK B Y, PARK H. Convolutional neural network classifier for distinguishing Barrett's esophagus and neoplasia endomicroscopy images[J]. Annu Int Conf IEEE Eng Med Biol Soc, 2017, 2017: 2892-2895.
14	DE GROOF A J, STRUYVENBERG M R, VAN DER PUTTEN J, et al. Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking[J]. Gastroenterology, 2020, 158(4): 915-929.e4.
15	ALOM M Z, TAHA T M, YAKOPCIC C, et al. The history began from AlexNet: a comprehensive survey on deep learning approaches[J/OL]. (2018-09-12)[2021-12-27]. http://arxiv.org/abs/1803.01164.
16	SHIN H C, ROTH H R, GAO M C, et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning[J]. IEEE Trans Med Imaging, 2016, 35(5): 1285-1298.
17	SAMALA R K, CHAN H P, HADJIISKI L, et al. Breast cancer diagnosis in digital breast tomosynthesis: effects of training sample size on multi-stage transfer learning using deep neural nets[J]. IEEE Trans Med Imaging, 2019, 38(3): 686-696.

Model	Confusion matrix				Evaluation Parameter
Xception			Model predicted label		Accuracy	0.805
	True label		Normal	Barrett	Recall	0.814
		Normal	47	12	Precision	0.800
					F1-Score	0.807
		Barrett	11	48	AUC	0.805
NASNetL			Model predicted label		Accuracy	0.873
	True label		Normal	Barrett	Recall	0.881
		Normal	51	8	Precision	0.867
					F1-Score	0.874
		Barrett	7	52	AUC	0.873
ResNet			Model predicted label		Accuracy	0.864
	True label		Normal	Barrett	Recall	0.864
		Normal	51	8	Precision	0.864
					F1-Score	0.864
		Barrett	8	51	AUC	0.864
BiT			Model predicted label		Accuracy	0.864
	True label		Normal	Barrett	Recall	0.881
		Normal	50	9	Precision	0.852
					F1-Score	0.867
		Barrett	7	52	AUC	0.864

Model	Confusion matrix				Evaluation Parameter
Xception			Model predicted label		Accuracy	0.805
	True label		Normal	Barrett	Recall	0.814
		Normal	47	12	Precision	0.800
					F1-Score	0.807
		Barrett	11	48	AUC	0.805
NASNetL			Model predicted label		Accuracy	0.873
	True label		Normal	Barrett	Recall	0.881
		Normal	51	8	Precision	0.867
					F1-Score	0.874
		Barrett	7	52	AUC	0.873
ResNet			Model predicted label		Accuracy	0.864
	True label		Normal	Barrett	Recall	0.864
		Normal	51	8	Precision	0.864
					F1-Score	0.864
		Barrett	8	51	AUC	0.864
BiT			Model predicted label		Accuracy	0.864
	True label		Normal	Barrett	Recall	0.881
		Normal	50	9	Precision	0.852
					F1-Score	0.867
		Barrett	7	52	AUC	0.864

Physician	Confusion matrix				Evaluation Parameter
Junior physician			Physician predicted label		Accuracy	0.855
	True label		Normal	Barrett	Recall	0.847
		Normal	49	10	Precision	0.850
					F1-Score	0.857
		Barrett	9	50	AUC	0.855
Senior physician			Physician predicted label		Accuracy	0.881
	True label		Normal	Barrett	Recall	0.898
		Normal	51	8	Precision	0.869
					F1-Score	0.883
		Barrett	6	53	AUC	0.881

Physician	Confusion matrix				Evaluation Parameter
Junior physician			Physician predicted label		Accuracy	0.855
	True label		Normal	Barrett	Recall	0.847
		Normal	49	10	Precision	0.850
					F1-Score	0.857
		Barrett	9	50	AUC	0.855
Senior physician			Physician predicted label		Accuracy	0.881
	True label		Normal	Barrett	Recall	0.898
		Normal	51	8	Precision	0.869
					F1-Score	0.883
		Barrett	6	53	AUC	0.881

Variable	Kappa value
Variable	Junior physician	Senior physician	Xception	NASNetL	ResNet	BiT
Junior physician	1.000	0.847	0.763	0.695	0.610	0.644
Senior physician	‒	1.000	0.780	0.712	0.661	0.661
Xception	‒	‒	1.000	0.695	0.678	0.712
NASNetL	‒	‒	‒	1.000	0.746	0.712
ResNet	‒	‒	‒	‒	1.000	0.763
BiT	‒	‒	‒	‒	‒	1.000