机器学习在抑郁症患者面部特征研究中的应用进展

doi:10.3969/j.issn.1674-8115.2022.01.019

摘要/Abstract

摘要：

抑郁症是一种严重影响生活质量的精神疾病，会伴随面部表情和行为的变化。目前的抑郁症诊断评估主要依赖于自我报告和医师观察，存在主观误差，缺乏客观有效的自动化抑郁症检测方法。面部表情可呈现重要的非语言信息，研究人员开始通过面部特征来辅助识别和诊断抑郁症。而机器学习作为人工智能的核心，在图像特征提取和分类方面有着突出的优势。该文以IEEE Xplore数据库为数据来源，梳理了2016—2021年基于机器学习的抑郁症患者面部特征研究，并对未来研究方向进行展望，以期为日后抑郁症临床智能化诊断和跟踪提供参考。

关键词: 抑郁症, 机器学习, 面部特征, 面部表情

Abstract:

Major depressive disorder (MDD) is a mental illness that severely affects the quality of life, accompanied by changes in facial expressions and other behaviors. The current diagnosis for MDD mainly relies on self-reports and observations from doctors, which has subjective errors. There is a lack of objective and effective automated MDD detection methods. Facial expressions are important nonverbal behaviors, and the researchers have begun to use facial features to assist in identifying and diagnosing depression. As the core of artificial intelligence, machine learning has outstanding advantages in image feature extraction and classification. Taking IEEE Xplore database as the data source, this article sorts out the researches on the facial features of MDD patients based on machine learning from 2016 to 2021, and prospects the future research directions, to provide reference for clinical intelligent diagnosis and tracking of MDD in the future.

Key words: major depressive disorder (MDD), machine learning, facial feature, facial expression

中图分类号:

R749.4

李欣, 范青. 机器学习在抑郁症患者面部特征研究中的应用进展[J]. 上海交通大学学报(医学版), 2022, 42(1): 124-129.

Xin LI, Qing FAN. Application progress of machine learning in the study of facial features of patients with depression[J]. JOURNAL OF SHANGHAI JIAOTONG UNIVERSITY (MEDICAL SCIENCE), 2022, 42(1): 124-129.

参考文献 49

1	AGARWAL S, MUKHERJEE D P. Facial expression recognition through adaptive learning of local motion descriptor[J]. Multimed Tools Appl, 2017, 76(1): 1073-1099.
2	WHO. The world health report 2001. Mental health: new understanding, new hope[R/OL]. [2021-03-22]. https://www.who.int/whr/2001/en/whr01_en.pdf.
3	BHATIA S, HAYAT M, BREAKSPEAR M, et al. A video-based facial behaviour analysis approach to melancholia[C]//2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017). Washington, DC, USA: IEEE, 2017: 754-761.
4	ELLGRING H. Non-verbal communication in depression[M]. Cambridge: Cambridge University Press, 2007.
5	ROTTENBERG J, GROSS J J, GOTLIB I H. Emotion context insensitivity in major depressive disorder[J]. J Abnorm Psychol, 2005, 114(4): 627-639.
6	BYLSMA L M, MORRIS B H, ROTTENBERG J. A meta-analysis of emotional reactivity in major depressive disorder[J]. Clin Psychol Rev, 2008, 28(4): 676-691.
7	PAMPOUCHIDOU A, SIMANTIRAKI O, VAZAKOPOULOU C M, et al. Facial geometry and speech analysis for depression detection[C]//2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Jeju, Korea (South): IEEE, 2017: 1433-1436.
8	CHOLLET F. Xception: deep learning with depthwise separable convolutions[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017: 1800-1807.
9	MOLLAHOSSEINI A, CHAN D, MAHOOR M H. Going deeper in facial expression recognition using deep neural networks[C]//2016 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Placid, NY, USA: IEEE, 2016: 1-10.
10	YADAV Y, KUMAR V, RANGA V, et al. Analysis of facial sentiments: a deep-learning way[C]//2020 International Conference on Electronics and Sustainable Communication Systems (ICESC). Coimbatore, India: IEEE, 2020: 541-545.
11	JIA C, LI C L, YING Z. Facial expression recognition based on the ensemble learning of CNNs[C]//2020 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC). Macao, China: IEEE, 2020: 1-5.
12	TANG Y, ZHANG X M, WANG H. Geometric-convolutional feature fusion based on learning propagation for facial expression recognition[J]. IEEE Access, 2018, 6: 42532-42540.
13	WANG X, WANG Y, ZHOU M, et al. Identifying psychological symptoms based on facial movements[J]. Front Psychiatry, 2020, 11: 607890.
14	JONITTA MERYL C, DHARSHINI K, SUJITHA JULIET D, et al. Deep learning based facial expression recognition for psychological health analysis[C]//2020 International Conference on Communication and Signal Processing (ICCSP). Chennai, India: IEEE, 2020: 1155-1158.
15	JIANG Z, HARATI S, CROWELL A, et al. Classifying major depressive disorder and response to deep brain stimulation over time by analyzing facial expressions[J]. IEEE Trans Biomed Eng, 2021, 68(2): 664-672.
16	MULAY A, DHEKNE A, WANI R, et al. Automatic depression level detection through visual input[C]//2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability WorldS4. London, UK: IEEE, 2020: 19-22.
17	LI J, LIU Z, DING Z, et al. A novel study for MDD detection through task-elicited facial cues[C]//2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Madrid, Spain: IEEE, 2018: 1003-1008.
18	GOGATE U, PARATE A, SAH S, et al. Real time emotion recognition and gender classification[C]//2020 International Conference on Smart Innovations in Design, Environment, Management, Planning and Computing (ICSIDEMPC). Aurangabad, India: IEEE, 2020: 138-143.
19	JAN A, MENG H, GAUS Y F B A, et al. Artificial intelligent system for automatic depression level analysis through visual and vocal expressions[J]. IEEE Trans Cogn Dev Syst, 2018, 10(3): 668-680.
20	CARNEIRO DE MELO W, GRANGER E, LOPEZ M B. Encoding temporal information for automatic depression recognition from facial analysis[C]//ICASSP 2020‒2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Barcelona, Spain: IEEE, 2020: 1080-1084.
21	CARNEIRO DE MELO W, GRANGER E, HADID A. A deep multiscale spatiotemporal network for assessing depression from facial dynamics[J]. IEEE Trans Affect Comput, 2020. DOI: 10.1109/TAFFC.2020.3021755.
22	ZHOU X, JIN K, SHANG Y, et al. Visually interpretable representation learning for depression recognition from facial images[J]. IEEE Trans Affect Comput, 2020, 11(3): 542-552.
23	SWAMY P M, JANARDHAN KURAPOTHULA P, MURTHY S V, et al. Voice assistant and facial analysis based approach to screen test clinical depression[C]//2019 1st International Conference on Advances in Information Technology (ICAIT). Chikmagalur, India: IEEE, 2019: 39-44.
24	CARNEIRO DE MELO W, GRANGER E, HADID A. Combining global and local convolutional 3D networks for detecting depression from facial expressions[C]//2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). Lille, France: IEEE, 2019: 1-8.
25	YANG L. Multi-modal depression detection and estimation[C]//2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). Cambridge, UK: IEEE, 2019: 26-30.
26	LI X, GUO W, YANG H. Depression severity prediction from facial expression based on the DRR_DepressionNet network[C]//2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Seoul, Korea (South): IEEE, 2020: 2757-2764.
27	KUMAR G A R, KUMAR R K, SANYAL G. Facial emotion analysis using deep convolution neural network[C]//2017 International Conference on Signal Processing and Communication (ICSPC). Coimbatore, India: IEEE, 2017: 369-374.
28	CARNEIRO DE MELO W, GRANGER E, HADID A. Depression detection based on deep distribution learning[C]//2019 IEEE International Conference on Image Processing (ICIP). Taipei, Taiwan, China: IEEE, 2019: 4544-4548.
29	JAZAERY MAL, GUO G. Video-based depression level analysis by encoding deep spatiotemporal features[J]. IEEE Trans Affect Comput, 2021, 12(1): 262-268.
30	WANG Y, MA J, HAO B, et al. Automatic depression detection via facial expressions using multiple instance learning[C]//2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). Iowa City, IA, USA: IEEE, 2020: 1933-1936.
31	SU M, WU C, HUANG K, et al. Exploring microscopic fluctuation of facial expression for mood disorder classification[C]//2017 International Conference on Orange Technologies (ICOT). Singapore: IEEE, 2017: 65-69.
32	GUO W, YANG H, LIU Z. Deep neural networks for depression recognition based on facial expressions caused by stimulus tasks[C]//2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). Cambridge, UK: IEEE, 2019: 133-139.
33	DIBEKLIOĞLU H, HAMMAL Z, COHN J F. Dynamic multimodal measurement of depression severity using deep autoencoding[J]. IEEE J Biomed Health Inform, 2018, 22(2): 525-536.
34	WEN L, LI X, GUO G, et al. Automated depression diagnosis based on facial dynamic analysis and sparse coding[J]. IEEE Trans Inf Forensics Secur, 2015, 10(7): 1432-1441.
35	安昳, 曲珍, 许宁, 等. 面部动态特征描述的抑郁症识别[J]. 中国图象图形学报, 2020, 25(11): 2415-2427.
36	HARATI S, CROWELL A, MAYBERG H, et al. Discriminating clinical phases of recovery from major depressive disorder using the dynamics of facial expression[J]. Annu Int Conf IEEE Eng Med Biol Soc, 2016, 2016: 2254-2257.
37	HARATI S, CROWELL A, HUANG Y, et al. Classifying depression severity in recovery from major depressive disorder via dynamic facial features[J]. IEEE J Biomed Health Informatics, 2020, 24(3): 815-824.
38	TRÉMEAU F, MALASPINA D, DUVAL F, et al. Facial expressiveness in patients with schizophrenia compared to depressed patients and nonpatient comparison subjects[J]. Am J Psychiatry, 2005, 162(1): 92-101.
39	GIRARD J M, COHN J F, MAHOOR M H, et al. Social risk and depression: evidence from manual and automatic facial expression analysis[J]. Proc Int Conf Autom Face Gesture Recognit, 2013: 1-8.
40	ALGHOWINEM S, GOECKE R, WAGNER M, et al. Multimodal depression detection: fusion analysis of paralinguistic, head pose and eye gaze behaviors[J]. IEEE Trans Affect Comput, 2018, 9(4): 478-490.
41	TOTO E, TLACHAC M, STEVENS F L, et al. Audio-based depression screening using sliding window Sub-clip pooling[C]//2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA). Miami, FL, USA: IEEE, 2020: 791-796.
42	YALAMANCHILI B, KOTA N S, ABBARAJU M S, et al. Real-time acoustic based depression detection using machine learning techniques[C]//2020 International Conference on Emerging Trends in Information Technology and Engineering (ic-ETITE). Vellore, India: IEEE, 2020: 1-6.
43	WANG Z, CHEN L, WANG L, et al. Recognition of audio depression based on convolutional neural network and generative antagonism network model[J]. IEEE Access, 2020, 8: 101181-101191.
44	FANG J, WANG T, LI C, et al. Depression prevalence in postgraduate students and its association with gait abnormality[J]. IEEE Access, 2019, 7: 174425-174437.
45	GAO S, CALHOUN V D, SUI J. Machine learning in major depression: from classification to treatment outcome prediction[J]. CNS Neurosci Ther, 2018, 24(11): 1037-1052.
46	FAN Y, YU R, LI J, et al. EEG-based mild depression recognition using multi-kernel convolutional and spatial-temporal Feature[C]//2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Seoul, Korea (South): IEEE, 2020: 1777-1784.
47	YANG L, JIANG D, SAHLI H. Integrating deep and shallow models for multi-modal depression analysis: hybrid architectures[J]. IEEE Trans Affect Comput, 2021, 12(1): 239-253.
48	STEPANOV E A, LATHUILIÈRE S, CHOWDHURY S A, et al. Depression severity estimation from multiple modalities[C]//2018 IEEE 20th International Conference on e-Health Networking, Applications and Services (Healthcom). Ostrava, Czech Republic: IEEE, 2018: 1-6.
49	HAQUE A, GUO M, MINER A S, et al. Measuring depression symptom severity from spoken language and 3D facial expressions[Z/OL]. [2021-03-22]. https://arxiv.org/pdf/1811.08592.pdf.

[1]	黄昕, 刘家辉, 叶敬文, 钱文莉, 许万星, 王琳. 基于机器学习的小细胞肺癌代谢分子诊断模型的建立和临床应用[J]. 上海交通大学学报（医学版）, 2025, 45(8): 1009-1016.
[2]	王思睿, 孔盖, 李惠, 钱禛颖, 崔慧茹, 唐莺莹. 经颅磁刺激治疗对抑郁症患者杏仁核及海马亚区体积的影响[J]. 上海交通大学学报（医学版）, 2025, 45(4): 434-442.
[3]	禹志远, 董海平, 高楠, 马柯. 背根神经节吗啡耐受核心基因筛选与机制研究：加权基因共表达网络分析和机器学习的转录组学整合策略[J]. 上海交通大学学报（医学版）, 2025, 45(10): 1308-1319.
[4]	吴其蓁, 刘启明, 柴烨子, 陶政宇, 王依楠, 郭欣宁, 姜萌, 卜军. 机器学习预测乳腺癌新辅助治疗后炎症代谢状态改变的模型评价[J]. 上海交通大学学报（医学版）, 2024, 44(9): 1169-1181.
[5]	许万星, 王琳, 郭巧梅, 王薛庆, 娄加陶. 多模态肺结节诊断模型的临床验证及应用价值探索[J]. 上海交通大学学报（医学版）, 2024, 44(8): 1030-1036.
[6]	陈深册, 陈依明, 王凡, 张梦珂, 杨惟杰, 吕洞宾, 洪武. 饮食干预治疗抑郁相关症状的研究进展[J]. 上海交通大学学报（医学版）, 2024, 44(8): 1050-1055.
[7]	孙晨寅, 吴百川, 张慧凤, 方贻儒, 彭代辉. 体动记录仪评估抑郁症昼夜节律：一项系统综述和meta分析[J]. 上海交通大学学报（医学版）, 2024, 44(5): 606-616.
[8]	杨瑞君, 吕书红, 刘志业, 张新, 刘志浩. 中国5省初中生视屏时间和饮食行为与抑郁症状的关联[J]. 上海交通大学学报（医学版）, 2024, 44(3): 358-364.
[9]	马文琳, 林元杰, 金婷婷, 石薇, 蒋莉华, 赵莉. 初中生自我中心主义与非自杀性自伤的关系研究[J]. 上海交通大学学报（医学版）, 2023, 43(8): 971-976.
[10]	马奔, 赵成, 束翌俊, 董平. CT影像组学在胃肠道间质瘤中的应用进展[J]. 上海交通大学学报（医学版）, 2023, 43(7): 923-930.
[11]	李偲媛, 和申, 李华芳. 重度抑郁症中自噬通路及其关键标志物的研究进展[J]. 上海交通大学学报（医学版）, 2023, 43(10): 1324-1331.
[12]	吴侠霏, 方婕, 漆洪波, 余昕烊. 妊娠期糖尿病对C57BL/6J子代成年鼠神经精神功能的影响[J]. 上海交通大学学报（医学版）, 2022, 42(4): 422-432.
[13]	沈梦婷, 张选红, 钱禛颖, 李惠, 盛建华, 王继军, 唐莺莹. 精神分裂症与抑郁症失匹配负波异常的对照研究[J]. 上海交通大学学报(医学版), 2021, 41(8): 1041-1045.
[14]	耿瑞杰, 姚琳, 黄欣欣, 禹顺英, 苑成梅, 洪武, 吕钦谕, 王庆中, 易正辉, 方贻儒. 基于加权基因共表达网络分析识别抑郁症的差异表达基因模块[J]. 上海交通大学学报(医学版), 2021, 41(6): 724-731.
[15]	吴静, 李学义, 陈京红, 王泽剑. 抑郁模型小鼠海马中胆汁酸受体变化的研究[J]. 上海交通大学学报(医学版), 2021, 41(12): 1628-1634.