机器学习在抑郁症患者面部特征研究中的应用进展

doi:10.3969/j.issn.1674-8115.2022.01.019

摘要/Abstract

摘要：

抑郁症是一种严重影响生活质量的精神疾病，会伴随面部表情和行为的变化。目前的抑郁症诊断评估主要依赖于自我报告和医师观察，存在主观误差，缺乏客观有效的自动化抑郁症检测方法。面部表情可呈现重要的非语言信息，研究人员开始通过面部特征来辅助识别和诊断抑郁症。而机器学习作为人工智能的核心，在图像特征提取和分类方面有着突出的优势。该文以IEEE Xplore数据库为数据来源，梳理了2016—2021年基于机器学习的抑郁症患者面部特征研究，并对未来研究方向进行展望，以期为日后抑郁症临床智能化诊断和跟踪提供参考。

关键词: 抑郁症, 机器学习, 面部特征, 面部表情

Abstract:

Major depressive disorder (MDD) is a mental illness that severely affects the quality of life, accompanied by changes in facial expressions and other behaviors. The current diagnosis for MDD mainly relies on self-reports and observations from doctors, which has subjective errors. There is a lack of objective and effective automated MDD detection methods. Facial expressions are important nonverbal behaviors, and the researchers have begun to use facial features to assist in identifying and diagnosing depression. As the core of artificial intelligence, machine learning has outstanding advantages in image feature extraction and classification. Taking IEEE Xplore database as the data source, this article sorts out the researches on the facial features of MDD patients based on machine learning from 2016 to 2021, and prospects the future research directions, to provide reference for clinical intelligent diagnosis and tracking of MDD in the future.

Key words: major depressive disorder (MDD), machine learning, facial feature, facial expression

中图分类号:

R749.4

李欣, 范青. 机器学习在抑郁症患者面部特征研究中的应用进展[J]. 上海交通大学学报(医学版), 2022, 42(1): 124-129.

Xin LI, Qing FAN. Application progress of machine learning in the study of facial features of patients with depression[J]. JOURNAL OF SHANGHAI JIAOTONG UNIVERSITY (MEDICAL SCIENCE), 2022, 42(1): 124-129.

参考文献 49

1	AGARWAL S, MUKHERJEE D P. Facial expression recognition through adaptive learning of local motion descriptor[J]. Multimed Tools Appl, 2017, 76(1): 1073-1099.
2	WHO. The world health report 2001. Mental health: new understanding, new hope[R/OL]. [2021-03-22]. https://www.who.int/whr/2001/en/whr01_en.pdf.
3	BHATIA S, HAYAT M, BREAKSPEAR M, et al. A video-based facial behaviour analysis approach to melancholia[C]//2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017). Washington, DC, USA: IEEE, 2017: 754-761.
4	ELLGRING H. Non-verbal communication in depression[M]. Cambridge: Cambridge University Press, 2007.
5	ROTTENBERG J, GROSS J J, GOTLIB I H. Emotion context insensitivity in major depressive disorder[J]. J Abnorm Psychol, 2005, 114(4): 627-639.
6	BYLSMA L M, MORRIS B H, ROTTENBERG J. A meta-analysis of emotional reactivity in major depressive disorder[J]. Clin Psychol Rev, 2008, 28(4): 676-691.
7	PAMPOUCHIDOU A, SIMANTIRAKI O, VAZAKOPOULOU C M, et al. Facial geometry and speech analysis for depression detection[C]//2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Jeju, Korea (South): IEEE, 2017: 1433-1436.
8	CHOLLET F. Xception: deep learning with depthwise separable convolutions[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017: 1800-1807.
9	MOLLAHOSSEINI A, CHAN D, MAHOOR M H. Going deeper in facial expression recognition using deep neural networks[C]//2016 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Placid, NY, USA: IEEE, 2016: 1-10.
10	YADAV Y, KUMAR V, RANGA V, et al. Analysis of facial sentiments: a deep-learning way[C]//2020 International Conference on Electronics and Sustainable Communication Systems (ICESC). Coimbatore, India: IEEE, 2020: 541-545.
11	JIA C, LI C L, YING Z. Facial expression recognition based on the ensemble learning of CNNs[C]//2020 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC). Macao, China: IEEE, 2020: 1-5.
12	TANG Y, ZHANG X M, WANG H. Geometric-convolutional feature fusion based on learning propagation for facial expression recognition[J]. IEEE Access, 2018, 6: 42532-42540.
13	WANG X, WANG Y, ZHOU M, et al. Identifying psychological symptoms based on facial movements[J]. Front Psychiatry, 2020, 11: 607890.
14	JONITTA MERYL C, DHARSHINI K, SUJITHA JULIET D, et al. Deep learning based facial expression recognition for psychological health analysis[C]//2020 International Conference on Communication and Signal Processing (ICCSP). Chennai, India: IEEE, 2020: 1155-1158.
15	JIANG Z, HARATI S, CROWELL A, et al. Classifying major depressive disorder and response to deep brain stimulation over time by analyzing facial expressions[J]. IEEE Trans Biomed Eng, 2021, 68(2): 664-672.
16	MULAY A, DHEKNE A, WANI R, et al. Automatic depression level detection through visual input[C]//2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability WorldS4. London, UK: IEEE, 2020: 19-22.
17	LI J, LIU Z, DING Z, et al. A novel study for MDD detection through task-elicited facial cues[C]//2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Madrid, Spain: IEEE, 2018: 1003-1008.
18	GOGATE U, PARATE A, SAH S, et al. Real time emotion recognition and gender classification[C]//2020 International Conference on Smart Innovations in Design, Environment, Management, Planning and Computing (ICSIDEMPC). Aurangabad, India: IEEE, 2020: 138-143.
19	JAN A, MENG H, GAUS Y F B A, et al. Artificial intelligent system for automatic depression level analysis through visual and vocal expressions[J]. IEEE Trans Cogn Dev Syst, 2018, 10(3): 668-680.
20	CARNEIRO DE MELO W, GRANGER E, LOPEZ M B. Encoding temporal information for automatic depression recognition from facial analysis[C]//ICASSP 2020‒2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Barcelona, Spain: IEEE, 2020: 1080-1084.
21	CARNEIRO DE MELO W, GRANGER E, HADID A. A deep multiscale spatiotemporal network for assessing depression from facial dynamics[J]. IEEE Trans Affect Comput, 2020. DOI: 10.1109/TAFFC.2020.3021755.
22	ZHOU X, JIN K, SHANG Y, et al. Visually interpretable representation learning for depression recognition from facial images[J]. IEEE Trans Affect Comput, 2020, 11(3): 542-552.
23	SWAMY P M, JANARDHAN KURAPOTHULA P, MURTHY S V, et al. Voice assistant and facial analysis based approach to screen test clinical depression[C]//2019 1st International Conference on Advances in Information Technology (ICAIT). Chikmagalur, India: IEEE, 2019: 39-44.
24	CARNEIRO DE MELO W, GRANGER E, HADID A. Combining global and local convolutional 3D networks for detecting depression from facial expressions[C]//2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). Lille, France: IEEE, 2019: 1-8.
25	YANG L. Multi-modal depression detection and estimation[C]//2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). Cambridge, UK: IEEE, 2019: 26-30.
26	LI X, GUO W, YANG H. Depression severity prediction from facial expression based on the DRR_DepressionNet network[C]//2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Seoul, Korea (South): IEEE, 2020: 2757-2764.
27	KUMAR G A R, KUMAR R K, SANYAL G. Facial emotion analysis using deep convolution neural network[C]//2017 International Conference on Signal Processing and Communication (ICSPC). Coimbatore, India: IEEE, 2017: 369-374.
28	CARNEIRO DE MELO W, GRANGER E, HADID A. Depression detection based on deep distribution learning[C]//2019 IEEE International Conference on Image Processing (ICIP). Taipei, Taiwan, China: IEEE, 2019: 4544-4548.
29	JAZAERY MAL, GUO G. Video-based depression level analysis by encoding deep spatiotemporal features[J]. IEEE Trans Affect Comput, 2021, 12(1): 262-268.
30	WANG Y, MA J, HAO B, et al. Automatic depression detection via facial expressions using multiple instance learning[C]//2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). Iowa City, IA, USA: IEEE, 2020: 1933-1936.
31	SU M, WU C, HUANG K, et al. Exploring microscopic fluctuation of facial expression for mood disorder classification[C]//2017 International Conference on Orange Technologies (ICOT). Singapore: IEEE, 2017: 65-69.
32	GUO W, YANG H, LIU Z. Deep neural networks for depression recognition based on facial expressions caused by stimulus tasks[C]//2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). Cambridge, UK: IEEE, 2019: 133-139.
33	DIBEKLIOĞLU H, HAMMAL Z, COHN J F. Dynamic multimodal measurement of depression severity using deep autoencoding[J]. IEEE J Biomed Health Inform, 2018, 22(2): 525-536.
34	WEN L, LI X, GUO G, et al. Automated depression diagnosis based on facial dynamic analysis and sparse coding[J]. IEEE Trans Inf Forensics Secur, 2015, 10(7): 1432-1441.
35	安昳, 曲珍, 许宁, 等. 面部动态特征描述的抑郁症识别[J]. 中国图象图形学报, 2020, 25(11): 2415-2427.
36	HARATI S, CROWELL A, MAYBERG H, et al. Discriminating clinical phases of recovery from major depressive disorder using the dynamics of facial expression[J]. Annu Int Conf IEEE Eng Med Biol Soc, 2016, 2016: 2254-2257.
37	HARATI S, CROWELL A, HUANG Y, et al. Classifying depression severity in recovery from major depressive disorder via dynamic facial features[J]. IEEE J Biomed Health Informatics, 2020, 24(3): 815-824.
38	TRÉMEAU F, MALASPINA D, DUVAL F, et al. Facial expressiveness in patients with schizophrenia compared to depressed patients and nonpatient comparison subjects[J]. Am J Psychiatry, 2005, 162(1): 92-101.
39	GIRARD J M, COHN J F, MAHOOR M H, et al. Social risk and depression: evidence from manual and automatic facial expression analysis[J]. Proc Int Conf Autom Face Gesture Recognit, 2013: 1-8.
40	ALGHOWINEM S, GOECKE R, WAGNER M, et al. Multimodal depression detection: fusion analysis of paralinguistic, head pose and eye gaze behaviors[J]. IEEE Trans Affect Comput, 2018, 9(4): 478-490.
41	TOTO E, TLACHAC M, STEVENS F L, et al. Audio-based depression screening using sliding window Sub-clip pooling[C]//2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA). Miami, FL, USA: IEEE, 2020: 791-796.
42	YALAMANCHILI B, KOTA N S, ABBARAJU M S, et al. Real-time acoustic based depression detection using machine learning techniques[C]//2020 International Conference on Emerging Trends in Information Technology and Engineering (ic-ETITE). Vellore, India: IEEE, 2020: 1-6.
43	WANG Z, CHEN L, WANG L, et al. Recognition of audio depression based on convolutional neural network and generative antagonism network model[J]. IEEE Access, 2020, 8: 101181-101191.
44	FANG J, WANG T, LI C, et al. Depression prevalence in postgraduate students and its association with gait abnormality[J]. IEEE Access, 2019, 7: 174425-174437.
45	GAO S, CALHOUN V D, SUI J. Machine learning in major depression: from classification to treatment outcome prediction[J]. CNS Neurosci Ther, 2018, 24(11): 1037-1052.
46	FAN Y, YU R, LI J, et al. EEG-based mild depression recognition using multi-kernel convolutional and spatial-temporal Feature[C]//2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Seoul, Korea (South): IEEE, 2020: 1777-1784.
47	YANG L, JIANG D, SAHLI H. Integrating deep and shallow models for multi-modal depression analysis: hybrid architectures[J]. IEEE Trans Affect Comput, 2021, 12(1): 239-253.
48	STEPANOV E A, LATHUILIÈRE S, CHOWDHURY S A, et al. Depression severity estimation from multiple modalities[C]//2018 IEEE 20th International Conference on e-Health Networking, Applications and Services (Healthcom). Ostrava, Czech Republic: IEEE, 2018: 1-6.
49	HAQUE A, GUO M, MINER A S, et al. Measuring depression symptom severity from spoken language and 3D facial expressions[Z/OL]. [2021-03-22]. https://arxiv.org/pdf/1811.08592.pdf.

[1]	沈梦婷, 张选红, 钱禛颖, 李惠, 盛建华, 王继军, 唐莺莹. 精神分裂症与抑郁症失匹配负波异常的对照研究[J]. 上海交通大学学报(医学版), 2021, 41(8): 1041-1045.
[2]	耿瑞杰, 姚琳, 黄欣欣, 禹顺英, 苑成梅, 洪武, 吕钦谕, 王庆中, 易正辉, 方贻儒. 基于加权基因共表达网络分析识别抑郁症的差异表达基因模块[J]. 上海交通大学学报(医学版), 2021, 41(6): 724-731.
[3]	吴静, 李学义, 陈京红, 王泽剑. 抑郁模型小鼠海马中胆汁酸受体变化的研究[J]. 上海交通大学学报(医学版), 2021, 41(12): 1628-1634.
[4]	施波, 陈建民, 赵俊雄, 唐伟, 范卫星, 张程赪, 张晨. CREB1基因与抑郁症和双相Ⅱ型障碍的关联研究[J]. 上海交通大学学报(医学版), 2021, 41(10): 1303-1307.
[5]	袁瑞雪, 傅迎美, 禹顺英. 辅助性T细胞17和调节性T细胞在抑郁症中的作用机制研究进展[J]. 上海交通大学学报(医学版), 2021, 41(10): 1384-1388.
[6]	文万军1，陈晔1，覃宗厚1，吕望强2，唐伟3，卢卫红4，范卫星2，张晨4. 血清白介素-6水平与精神分裂症伴抑郁症状之间的关系[J]. 上海交通大学学报(医学版), 2020, 40(12): 1618-1620.
[7]	赵俊雄1，华深1，范卫星1，王卫平1，唐伟2，张晨1, 3. SIRT1基因与抑郁症及其临床表型的关联研究[J]. 上海交通大学学报（医学版）, 2020, 40(1): 18-.
[8]	林青青，仇剑崟. 抑郁症患者的社会和人际功能障碍[J]. 上海交通大学学报(医学版), 2020, 40(07): 985-989.
[9]	程佳月，李璞玉，顾秋梦，王佩，陈珏，刘强#，王振#. 强迫症患者的强迫症状在归因方式与抑郁症状间的中介作用[J]. 上海交通大学学报(医学版), 2020, 40(06): 785-790.
[10]	郭垒1，胡嫣然1, 2，亢清1，王钰萍1，王振1，陈涵1#，陈珏1#. 神经性厌食患者应激负性感受和体象关注的关系：抑郁症状的中介作用[J]. 上海交通大学学报(医学版), 2020, 40(06): 798-803.
[11]	钱时兴，房圆#，孙琳，仇琦，林治光，肖世富，李霞#. 西酞普兰对抑郁症患者外周血miRNA-16/5- 羟色胺转运体通路的影响[J]. 上海交通大学学报(医学版), 2020, 40(06): 814-819.
[12]	叶惠玲，吕娜，范青. 抑郁症语义加工异常脑机制研究进展[J]. 上海交通大学学报(医学版), 2020, 40(05): 679-682.
[13]	贾芷莹 1, 2，董旻晔 1, 2，施贞夙 2, 3，金春林 4，李国红 1, 2. 基于机器学习的轻度认知功能障碍筛查研究[J]. 上海交通大学学报（医学版）, 2019, 39(8): 908-.
[14]	余玲芳，张晨. 抑郁症的遗传学研究进展[J]. 上海交通大学学报（医学版）, 2019, 39(8): 914-.
[15]	赵冬梅 1, 2，丁蕾 1，刘虹晔 1, 3，彭代辉 1. 抑郁症伴自杀的犬尿氨酸通路机制研究进展[J]. 上海交通大学学报（医学版）, 2019, 39(7): 805-.