Application progress of machine learning in the study of facial features of patients with depression

doi:10.3969/j.issn.1674-8115.2022.01.019

Abstract

Abstract:

Major depressive disorder (MDD) is a mental illness that severely affects the quality of life, accompanied by changes in facial expressions and other behaviors. The current diagnosis for MDD mainly relies on self-reports and observations from doctors, which has subjective errors. There is a lack of objective and effective automated MDD detection methods. Facial expressions are important nonverbal behaviors, and the researchers have begun to use facial features to assist in identifying and diagnosing depression. As the core of artificial intelligence, machine learning has outstanding advantages in image feature extraction and classification. Taking IEEE Xplore database as the data source, this article sorts out the researches on the facial features of MDD patients based on machine learning from 2016 to 2021, and prospects the future research directions, to provide reference for clinical intelligent diagnosis and tracking of MDD in the future.

Key words: major depressive disorder (MDD), machine learning, facial feature, facial expression

CLC Number:

R749.4

Xin LI, Qing FAN. Application progress of machine learning in the study of facial features of patients with depression[J]. JOURNAL OF SHANGHAI JIAOTONG UNIVERSITY (MEDICAL SCIENCE), 2022, 42(1): 124-129.

TrendMD

References 49

1	AGARWAL S, MUKHERJEE D P. Facial expression recognition through adaptive learning of local motion descriptor[J]. Multimed Tools Appl, 2017, 76(1): 1073-1099.
2	WHO. The world health report 2001. Mental health: new understanding, new hope[R/OL]. [2021-03-22]. https://www.who.int/whr/2001/en/whr01_en.pdf.
3	BHATIA S, HAYAT M, BREAKSPEAR M, et al. A video-based facial behaviour analysis approach to melancholia[C]//2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017). Washington, DC, USA: IEEE, 2017: 754-761.
4	ELLGRING H. Non-verbal communication in depression[M]. Cambridge: Cambridge University Press, 2007.
5	ROTTENBERG J, GROSS J J, GOTLIB I H. Emotion context insensitivity in major depressive disorder[J]. J Abnorm Psychol, 2005, 114(4): 627-639.
6	BYLSMA L M, MORRIS B H, ROTTENBERG J. A meta-analysis of emotional reactivity in major depressive disorder[J]. Clin Psychol Rev, 2008, 28(4): 676-691.
7	PAMPOUCHIDOU A, SIMANTIRAKI O, VAZAKOPOULOU C M, et al. Facial geometry and speech analysis for depression detection[C]//2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Jeju, Korea (South): IEEE, 2017: 1433-1436.
8	CHOLLET F. Xception: deep learning with depthwise separable convolutions[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017: 1800-1807.
9	MOLLAHOSSEINI A, CHAN D, MAHOOR M H. Going deeper in facial expression recognition using deep neural networks[C]//2016 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Placid, NY, USA: IEEE, 2016: 1-10.
10	YADAV Y, KUMAR V, RANGA V, et al. Analysis of facial sentiments: a deep-learning way[C]//2020 International Conference on Electronics and Sustainable Communication Systems (ICESC). Coimbatore, India: IEEE, 2020: 541-545.
11	JIA C, LI C L, YING Z. Facial expression recognition based on the ensemble learning of CNNs[C]//2020 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC). Macao, China: IEEE, 2020: 1-5.
12	TANG Y, ZHANG X M, WANG H. Geometric-convolutional feature fusion based on learning propagation for facial expression recognition[J]. IEEE Access, 2018, 6: 42532-42540.
13	WANG X, WANG Y, ZHOU M, et al. Identifying psychological symptoms based on facial movements[J]. Front Psychiatry, 2020, 11: 607890.
14	JONITTA MERYL C, DHARSHINI K, SUJITHA JULIET D, et al. Deep learning based facial expression recognition for psychological health analysis[C]//2020 International Conference on Communication and Signal Processing (ICCSP). Chennai, India: IEEE, 2020: 1155-1158.
15	JIANG Z, HARATI S, CROWELL A, et al. Classifying major depressive disorder and response to deep brain stimulation over time by analyzing facial expressions[J]. IEEE Trans Biomed Eng, 2021, 68(2): 664-672.
16	MULAY A, DHEKNE A, WANI R, et al. Automatic depression level detection through visual input[C]//2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability WorldS4. London, UK: IEEE, 2020: 19-22.
17	LI J, LIU Z, DING Z, et al. A novel study for MDD detection through task-elicited facial cues[C]//2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Madrid, Spain: IEEE, 2018: 1003-1008.
18	GOGATE U, PARATE A, SAH S, et al. Real time emotion recognition and gender classification[C]//2020 International Conference on Smart Innovations in Design, Environment, Management, Planning and Computing (ICSIDEMPC). Aurangabad, India: IEEE, 2020: 138-143.
19	JAN A, MENG H, GAUS Y F B A, et al. Artificial intelligent system for automatic depression level analysis through visual and vocal expressions[J]. IEEE Trans Cogn Dev Syst, 2018, 10(3): 668-680.
20	CARNEIRO DE MELO W, GRANGER E, LOPEZ M B. Encoding temporal information for automatic depression recognition from facial analysis[C]//ICASSP 2020‒2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Barcelona, Spain: IEEE, 2020: 1080-1084.
21	CARNEIRO DE MELO W, GRANGER E, HADID A. A deep multiscale spatiotemporal network for assessing depression from facial dynamics[J]. IEEE Trans Affect Comput, 2020. DOI: 10.1109/TAFFC.2020.3021755.
22	ZHOU X, JIN K, SHANG Y, et al. Visually interpretable representation learning for depression recognition from facial images[J]. IEEE Trans Affect Comput, 2020, 11(3): 542-552.
23	SWAMY P M, JANARDHAN KURAPOTHULA P, MURTHY S V, et al. Voice assistant and facial analysis based approach to screen test clinical depression[C]//2019 1st International Conference on Advances in Information Technology (ICAIT). Chikmagalur, India: IEEE, 2019: 39-44.
24	CARNEIRO DE MELO W, GRANGER E, HADID A. Combining global and local convolutional 3D networks for detecting depression from facial expressions[C]//2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). Lille, France: IEEE, 2019: 1-8.
25	YANG L. Multi-modal depression detection and estimation[C]//2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). Cambridge, UK: IEEE, 2019: 26-30.
26	LI X, GUO W, YANG H. Depression severity prediction from facial expression based on the DRR_DepressionNet network[C]//2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Seoul, Korea (South): IEEE, 2020: 2757-2764.
27	KUMAR G A R, KUMAR R K, SANYAL G. Facial emotion analysis using deep convolution neural network[C]//2017 International Conference on Signal Processing and Communication (ICSPC). Coimbatore, India: IEEE, 2017: 369-374.
28	CARNEIRO DE MELO W, GRANGER E, HADID A. Depression detection based on deep distribution learning[C]//2019 IEEE International Conference on Image Processing (ICIP). Taipei, Taiwan, China: IEEE, 2019: 4544-4548.
29	JAZAERY MAL, GUO G. Video-based depression level analysis by encoding deep spatiotemporal features[J]. IEEE Trans Affect Comput, 2021, 12(1): 262-268.
30	WANG Y, MA J, HAO B, et al. Automatic depression detection via facial expressions using multiple instance learning[C]//2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). Iowa City, IA, USA: IEEE, 2020: 1933-1936.
31	SU M, WU C, HUANG K, et al. Exploring microscopic fluctuation of facial expression for mood disorder classification[C]//2017 International Conference on Orange Technologies (ICOT). Singapore: IEEE, 2017: 65-69.
32	GUO W, YANG H, LIU Z. Deep neural networks for depression recognition based on facial expressions caused by stimulus tasks[C]//2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). Cambridge, UK: IEEE, 2019: 133-139.
33	DIBEKLIOĞLU H, HAMMAL Z, COHN J F. Dynamic multimodal measurement of depression severity using deep autoencoding[J]. IEEE J Biomed Health Inform, 2018, 22(2): 525-536.
34	WEN L, LI X, GUO G, et al. Automated depression diagnosis based on facial dynamic analysis and sparse coding[J]. IEEE Trans Inf Forensics Secur, 2015, 10(7): 1432-1441.
35	安昳, 曲珍, 许宁, 等. 面部动态特征描述的抑郁症识别[J]. 中国图象图形学报, 2020, 25(11): 2415-2427.
36	HARATI S, CROWELL A, MAYBERG H, et al. Discriminating clinical phases of recovery from major depressive disorder using the dynamics of facial expression[J]. Annu Int Conf IEEE Eng Med Biol Soc, 2016, 2016: 2254-2257.
37	HARATI S, CROWELL A, HUANG Y, et al. Classifying depression severity in recovery from major depressive disorder via dynamic facial features[J]. IEEE J Biomed Health Informatics, 2020, 24(3): 815-824.
38	TRÉMEAU F, MALASPINA D, DUVAL F, et al. Facial expressiveness in patients with schizophrenia compared to depressed patients and nonpatient comparison subjects[J]. Am J Psychiatry, 2005, 162(1): 92-101.
39	GIRARD J M, COHN J F, MAHOOR M H, et al. Social risk and depression: evidence from manual and automatic facial expression analysis[J]. Proc Int Conf Autom Face Gesture Recognit, 2013: 1-8.
40	ALGHOWINEM S, GOECKE R, WAGNER M, et al. Multimodal depression detection: fusion analysis of paralinguistic, head pose and eye gaze behaviors[J]. IEEE Trans Affect Comput, 2018, 9(4): 478-490.
41	TOTO E, TLACHAC M, STEVENS F L, et al. Audio-based depression screening using sliding window Sub-clip pooling[C]//2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA). Miami, FL, USA: IEEE, 2020: 791-796.
42	YALAMANCHILI B, KOTA N S, ABBARAJU M S, et al. Real-time acoustic based depression detection using machine learning techniques[C]//2020 International Conference on Emerging Trends in Information Technology and Engineering (ic-ETITE). Vellore, India: IEEE, 2020: 1-6.
43	WANG Z, CHEN L, WANG L, et al. Recognition of audio depression based on convolutional neural network and generative antagonism network model[J]. IEEE Access, 2020, 8: 101181-101191.
44	FANG J, WANG T, LI C, et al. Depression prevalence in postgraduate students and its association with gait abnormality[J]. IEEE Access, 2019, 7: 174425-174437.
45	GAO S, CALHOUN V D, SUI J. Machine learning in major depression: from classification to treatment outcome prediction[J]. CNS Neurosci Ther, 2018, 24(11): 1037-1052.
46	FAN Y, YU R, LI J, et al. EEG-based mild depression recognition using multi-kernel convolutional and spatial-temporal Feature[C]//2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Seoul, Korea (South): IEEE, 2020: 1777-1784.
47	YANG L, JIANG D, SAHLI H. Integrating deep and shallow models for multi-modal depression analysis: hybrid architectures[J]. IEEE Trans Affect Comput, 2021, 12(1): 239-253.
48	STEPANOV E A, LATHUILIÈRE S, CHOWDHURY S A, et al. Depression severity estimation from multiple modalities[C]//2018 IEEE 20th International Conference on e-Health Networking, Applications and Services (Healthcom). Ostrava, Czech Republic: IEEE, 2018: 1-6.
49	HAQUE A, GUO M, MINER A S, et al. Measuring depression symptom severity from spoken language and 3D facial expressions[Z/OL]. [2021-03-22]. https://arxiv.org/pdf/1811.08592.pdf.

[1]	Zhu Menglin, Liu Xiao, Xu Xiaodan, Wang Ganhong, Xia Kaijian, Chen Jian. Prediction of delayed post-polypectomy bleeding using a multimodal model [J]. Journal of Shanghai Jiao Tong University (Medical Science), 2026, 46(4): 509-520.
[2]	Xie Yixia, Wang Fan, Liu Xirui, Chen Ningning, Zhang Wenxuan, Wu Jialin, Ma Ting, Hong Wu. Overview of mechanisms of adult major depressive disorder comorbid attention deficit hyperactivity disorder and the prospects of transcranial direct current stimulation [J]. Journal of Shanghai Jiao Tong University (Medical Science), 2026, 46(1): 107-114.
[3]	HUANG Xin, LIU Jiahui, YE Jingwen, QIAN Wenli, XU Wanxing, WANG Lin. Development and clinical application of a machine learning-driven model for metabolite-based diagnosis of small cell lung cancer [J]. Journal of Shanghai Jiao Tong University (Medical Science), 2025, 45(8): 1009-1016.
[4]	RUAN Qingqing, SU Shuzhi, LI Yanting, REN Yuan, DAI Yong, QIAO Zengyong. Intraoperative complications in percutaneous coronary intervention for acute myocardial infarction: development of a risk prediction model [J]. Journal of Shanghai Jiao Tong University (Medical Science), 2025, 45(12): 1589-1597.
[5]	YU Zhiyuan, DONG Haiping, GAO Nan, MA Ke. Identification and mechanistic analysis of core genes associated with morphine tolerance in dorsal root ganglion: an integrative transcriptomics approach using WGCNA and machine learning algorithms [J]. Journal of Shanghai Jiao Tong University (Medical Science), 2025, 45(10): 1308-1319.
[6]	WU Qizhen, LIU Qiming, CHAI Yezi, TAO Zhengyu, WANG Yinan, GUO Xinning, JIANG Meng, PU Jun. Evaluation of machine learning prediction of altered inflammatory metabolic state after neoadjuvant therapy for breast cancer [J]. Journal of Shanghai Jiao Tong University (Medical Science), 2024, 44(9): 1169-1181.
[7]	XU Wanxing, WANG Lin, GUO Qiaomei, WANG Xueqing, LOU Jiatao. Clinical validation and application value exploration of multi-modal pulmonary nodule diagnosis model [J]. Journal of Shanghai Jiao Tong University (Medical Science), 2024, 44(8): 1030-1036.
[8]	MA Ben, ZHAO Cheng, SHU Yijun, DONG Ping. Application progress of CT radiomics in gastrointestinal stromal tumor [J]. Journal of Shanghai Jiao Tong University (Medical Science), 2023, 43(7): 923-930.
[9]	LI Siyuan, HE Shen, LI Huafang. Recent advance in autophagy-related pathways and key biomarkers in major depressive disorder [J]. Journal of Shanghai Jiao Tong University (Medical Science), 2023, 43(10): 1324-1331.
[10]	Rui-jie GENG, Lin YAO, Xin-xin HUANG, Shun-ying YU, Cheng-mei YUAN, Wu HONG, Qin-yu LÜ, Qing-zhong WANG, Zheng-hui YI, Yi-ru FANG. Identification of differentially expressed gene modules in major depressive disorder based on weighted gene co-expression network analysis [J]. JOURNAL OF SHANGHAI JIAOTONG UNIVERSITY (MEDICAL SCIENCE), 2021, 41(6): 724-731.
[11]	LIN Qing-qing, QIU Jian-yin. Social and interpersonal dysfunction of patients with major depressive disorder [J]. JOURNAL OF SHANGHAI JIAOTONG UNIVERSITY (MEDICAL SCIENCE), 2020, 40(7): 985-989.
[12]	JIA Zhi-ying1, 2, DONG Min-ye1, 2, SHI Zhen-su2, 3, JIN Chun-lin4, LI Guo-hong1, 2. Study of a screening system for mild cognitive impairment based on machine learning model [J]. , 2019, 39(8): 908-.
[13]	ZHAO Dong-mei1, 2, DING Lei1, LIU Hong-ye1, 3, PENG Dai-hui1. Advanced research of kynurenine pathway mechanism in suicide of major depressive disorder [J]. , 2019, 39(7): 805-.
[14]	XU Yu-peng1, DU Yu-chen1, 2, CHEN Feng-e1. Automatic layer segmentation of optical coherence tomography images in retinal vascular diseases [J]. , 2019, 39(6): 613-.
[15]	SANG Chao1, XIE Guo-xiang2, LIANG Dan-dan1, ZHAO Ai-hua1, JIA Wei1, 2, CHEN Tian-lu1. Improvement of liver fibrosis diagnostic models based on Youden index [J]. , 2019, 39(10): 1156-.