Research Article
BibTex RIS Cite

Year 2024, , 77 - 91, 30.04.2024

Ergün Batuhan Kaynak , Hamdi Dibeklioğlu

https://doi.org/10.35377/saucis...1381522

Abstract

References

  • [1] Hamilton M. A rating scale for depression. Journal of Neurology, Neurosurgery & Psychiatry, 23(1):56–62, 1960.
  • [2] Kurt Kroenke, Tara Strine, Robert Spitzer, Janet Williams, Joyce Berry, and Ali Mokdad. The phq-8 as a measure of current depression in the general population. Journal of affective disorders, 114:163–73, 09 2008.
  • [3] Amit Gupta, Priya Mathur, Shruti Bijawat, and Abhishek Dadheech. A novel work on analyzing stress and depression level of indian population during covid-19. Recent Advances in Computer Science and Communications, 13, 11 2020.
  • [4] World Health Organization. Depression and other common mental disorders: global health estimates. Technical report, 2017. License: CC BY-NC-SA 3.0 IGO.
  • [5] Jörg Zinken, Katarzyna Zinken, J. Clare Wilson, Lisa Butler, and Timothy Skinner. Analysis of syntax and word use to predict successful participation in guided self-help for anxiety and depression. Psychiatry Research, 179(2):181–186, 2010.
  • [6] Stephanie Rude, Eva-Maria Gortner, and James Pennebaker. Language use of depressed and depression-vulnerable college students. Cognition Emotion - COGNITION EMOTION, 18:1121–1133, 12 2004.
  • [7] Michael P. Caligiuri and Joel Ellwanger. Motor and cognitive aspects of motor retardation in depression. Journal of Affective Disorders, 57(1):83–93, 2000.
  • [8] Psychomotor symptoms of depression. American Journal of Psychiatry, 154(1):4–17, 1997. PMID: 8988952.
  • [9] Heysem Kaya et al. Predicting depression and emotions in the cross-roads of cultures, para-linguistics, and non-linguistics. In Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, AVEC ’19, page 27–35, New York, NY, USA, 2019. Association for Computing Machinery.
  • [10] Jiayu Ye, Yanhong Yu, Qingxiang Wang, Wentao Li, Hu Liang, Yunshao Zheng, and Gang Fu. Multi-modal depression detection based on emotional audio and evaluation text. Journal of Affective Disorders, 295:904–913, 2021.
  • [11] Nujud Aloshban, Anna Esposito, and Alessandro Vinciarelli. What you say or how you say it? depression detection through joint modeling of linguistic and acoustic aspects of speech. Cognitive Computation, 02 2021.
  • [12] Chujun Yang, Xiangwei Lai, Zhe Hu, Yanni Liu, and Peng Shen. Depression tendency screening use text based emotional analysis technique. Journal of Physics: Conference Series, 1237:032035, 06 2019.
  • [13] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS’13, page 3111–3119, Red Hook, NY, USA, 2013. Curran Associates Inc.
  • [14] Daniel Cer et al. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169–174, Brussels, Belgium, November 2018. Association for Computational Linguistics.
  • [15] Anupama Ray, Siddharth Kumar, Rutvik Reddy, Prerana Mukherjee, and Ritu Garg. Multi-level attention network using text, audio and video for depression prediction. In Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, AVEC ’19, page 81–88, New York, NY, USA, 2019. Association for Computing Machinery.
  • [16] Guramritpal Singh Saggu, Keshav Gupta, and K. V. Arya. Depressnet: A multimodal hierarchical attention mechanism approach for depression detection. International Journal of Engineering Sciences, 2022.
  • [17] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  • [18] Mariana Rodrigues Makiuchi, Tifani Warnita, Kuniaki Uto, and Koichi Shinoda. Multimodal fusion of bert-cnn and gated cnn representations for depression detection. Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, 2019.
  • [19] Hao Sun, Hongyi Wang, Jiaqing Liu, Yen-Wei Chen, and Lanfen Lin. Cubemlp: An mlp-based model for multimodal sentiment analysis and depression estimation. arXiv:2207.14087, 2022.
  • [20] Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar, October 2014. Association for Computational Linguistics.
  • [21] Shweta Yadav, Jainish Chauhan, Joy Prakash Sain, Krishnaprasad Thirunarayan, Amit Sheth, and Jeremiah Schumm. Identifying depressive symptoms from tweets: Figurative language enabled multitask learning framework. In Proceedings of the 28th International Conference on Computational Linguistics, pages 696–709, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.
  • [22] Jaskaran Singh, Narpinder Singh, Mostafa M. Fouda, Luca Saba, and Jasjit S. Suri. Attention-enabled ensemble deep learning models and their validation for depression detection: A domain adoption paradigm. Diagnostics, 13(12), 2023.
  • [23] Lei Tong et al. Cost-sensitive boosting pruning trees for depression detection on twitter. IEEE Transactions on Affective Computing, pages 1–1, 2022.
  • [24] Jingfang Liu and Mengshi Shi. A hybrid feature selection and ensemble approach to identify depressed users in online social media. Frontiers in Psychology, 12, 01 2022.
  • [25] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019.
  • [26] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. arXiv preprint arXiv:2004.09297, 2020.
  • [27] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  • [28] Fabien Ringeval et al. Avec 2019 workshop and challenge: State-of-mind, detecting depression with ai, and cross-cultural affect recognition. In Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, AVEC ’19, page 3–12, New York, NY, USA, 2019. Association for Computing Machinery.
  • [29] L I Lin. A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45 1:255–68, 1989.
  • [30] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, page 448–456. JMLR.org, 2015.
  • [31] Hamdi Dibeklioglu, Zakia Hammal, and Jeffrey Cohn. Dynamic multimodal measurement of depression severity using deep autoencoding. IEEE Journal of Biomedical and Health Informatics, PP:1–1, 03 2017.
  • [32] Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir D. Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. arXiv: Computer Vision and Pattern Recognition, 2014.
  • [33] Mo Zhou, Tianyi Liu, Yan Li, Dachao Lin, Enlu Zhou, and Tuo Zhao. Towards understanding the importance of noise in training neural networks. ArXiv, abs/1909.03172, 2019.
  • [34] Qi Dong, Shaogang Gong, and Xiatian Zhu. Class rectification hard mining for imbalanced deep learning. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 1869–1878, 2017.
  • [35] Max Bucher, Stéphane Herbin, and Frédéric Jurie. Hard negative mining for metric learning based zero-shot classification. In ECCV Workshops, 2016.
  • [36] Ming Fang, Siyu Peng, Yujia Liang, Chih-Cheng Hung, and Shuhua Liu. A multimodal fusion model with multi-level attention mechanism for depression detection. Biomedical Signal Processing and Control, 82:104561, 2023.
  • [37] Congcong Wang, Decheng Liu, Kemeng Tao, Xiaoxiao Cui, Gongtang Wang, Yuefeng Zhao, Zhi Liu. A multi-modal feature layer fusion model for assessment of depression based on attention mechanisms. In 2022 15th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pages 1–6, 2022.
  • [38] Zhuojin Han, Yuanyuan Shang, Zhuhong Shao, Jingyi Liu, Guodong Guo, Tie Liu, Hui Ding, Qiang Hu. Spatial-temporal feature networkfor speech-based depression recognition. IEEE Transactions on Cognitive and Developmental Systems, pages 1–1, 2023.
  • [39] Hao Sun, Jiaqing Liu, Shurong Chai, Zhaolin Qiu, Lanfen Lin, Xinyin Huang, and Yen-Wei Chen. Multi-modal adaptive fusion transformer network for the estimation of depression level. Sensors (Basel, Switzerland), 21, 2021.
  • [40] Shi Yin, Cong Liang, Heyan Ding, and Shangfei Wang. A multi-modal hierarchical recurrent neural network for depression detection. Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, 2019.

Systematic analysis of speech transcription modeling for reliable assessment of depression severity

Year 2024, , 77 - 91, 30.04.2024

Ergün Batuhan Kaynak , Hamdi Dibeklioğlu

https://doi.org/10.35377/saucis...1381522

Abstract

For depression severity assessment, we systematically analyze a modular deep learning pipeline that uses speech transcriptions as input for depression severity prediction. Through our pipeline, we investigate the role of popular deep learning architectures in creating representations for depression assessment. Evaluation of the proposed architectures is performed on the publicly available Extended Distress Analysis Interview Corpus dataset (E-DAIC). Through the results and discussions, we show that informative representations for depression assessment can be obtained without exploiting the temporal dynamics between descriptive text representations. More specifically, temporal pooling of latent representations outperforms the state of the art, which employs recurrent architectures, by 8.8% in terms of Concordance Correlation Coefficient (CCC).

Keywords

Depression severity assessment speech transcription analysis text analysis deep learning

Ethical Statement

It is declared that during the preparation process of this study, scientific and ethical principles were followed, and all the studies benefited from are stated in the bibliography.Since the data used in this research is publicly available, an ethics committee approval is not required / not applicable.

References

  • [1] Hamilton M. A rating scale for depression. Journal of Neurology, Neurosurgery & Psychiatry, 23(1):56–62, 1960.
  • [2] Kurt Kroenke, Tara Strine, Robert Spitzer, Janet Williams, Joyce Berry, and Ali Mokdad. The phq-8 as a measure of current depression in the general population. Journal of affective disorders, 114:163–73, 09 2008.
  • [3] Amit Gupta, Priya Mathur, Shruti Bijawat, and Abhishek Dadheech. A novel work on analyzing stress and depression level of indian population during covid-19. Recent Advances in Computer Science and Communications, 13, 11 2020.
  • [4] World Health Organization. Depression and other common mental disorders: global health estimates. Technical report, 2017. License: CC BY-NC-SA 3.0 IGO.
  • [5] Jörg Zinken, Katarzyna Zinken, J. Clare Wilson, Lisa Butler, and Timothy Skinner. Analysis of syntax and word use to predict successful participation in guided self-help for anxiety and depression. Psychiatry Research, 179(2):181–186, 2010.
  • [6] Stephanie Rude, Eva-Maria Gortner, and James Pennebaker. Language use of depressed and depression-vulnerable college students. Cognition Emotion - COGNITION EMOTION, 18:1121–1133, 12 2004.
  • [7] Michael P. Caligiuri and Joel Ellwanger. Motor and cognitive aspects of motor retardation in depression. Journal of Affective Disorders, 57(1):83–93, 2000.
  • [8] Psychomotor symptoms of depression. American Journal of Psychiatry, 154(1):4–17, 1997. PMID: 8988952.
  • [9] Heysem Kaya et al. Predicting depression and emotions in the cross-roads of cultures, para-linguistics, and non-linguistics. In Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, AVEC ’19, page 27–35, New York, NY, USA, 2019. Association for Computing Machinery.
  • [10] Jiayu Ye, Yanhong Yu, Qingxiang Wang, Wentao Li, Hu Liang, Yunshao Zheng, and Gang Fu. Multi-modal depression detection based on emotional audio and evaluation text. Journal of Affective Disorders, 295:904–913, 2021.
  • [11] Nujud Aloshban, Anna Esposito, and Alessandro Vinciarelli. What you say or how you say it? depression detection through joint modeling of linguistic and acoustic aspects of speech. Cognitive Computation, 02 2021.
  • [12] Chujun Yang, Xiangwei Lai, Zhe Hu, Yanni Liu, and Peng Shen. Depression tendency screening use text based emotional analysis technique. Journal of Physics: Conference Series, 1237:032035, 06 2019.
  • [13] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS’13, page 3111–3119, Red Hook, NY, USA, 2013. Curran Associates Inc.
  • [14] Daniel Cer et al. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169–174, Brussels, Belgium, November 2018. Association for Computational Linguistics.
  • [15] Anupama Ray, Siddharth Kumar, Rutvik Reddy, Prerana Mukherjee, and Ritu Garg. Multi-level attention network using text, audio and video for depression prediction. In Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, AVEC ’19, page 81–88, New York, NY, USA, 2019. Association for Computing Machinery.
  • [16] Guramritpal Singh Saggu, Keshav Gupta, and K. V. Arya. Depressnet: A multimodal hierarchical attention mechanism approach for depression detection. International Journal of Engineering Sciences, 2022.
  • [17] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  • [18] Mariana Rodrigues Makiuchi, Tifani Warnita, Kuniaki Uto, and Koichi Shinoda. Multimodal fusion of bert-cnn and gated cnn representations for depression detection. Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, 2019.
  • [19] Hao Sun, Hongyi Wang, Jiaqing Liu, Yen-Wei Chen, and Lanfen Lin. Cubemlp: An mlp-based model for multimodal sentiment analysis and depression estimation. arXiv:2207.14087, 2022.
  • [20] Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar, October 2014. Association for Computational Linguistics.
  • [21] Shweta Yadav, Jainish Chauhan, Joy Prakash Sain, Krishnaprasad Thirunarayan, Amit Sheth, and Jeremiah Schumm. Identifying depressive symptoms from tweets: Figurative language enabled multitask learning framework. In Proceedings of the 28th International Conference on Computational Linguistics, pages 696–709, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.
  • [22] Jaskaran Singh, Narpinder Singh, Mostafa M. Fouda, Luca Saba, and Jasjit S. Suri. Attention-enabled ensemble deep learning models and their validation for depression detection: A domain adoption paradigm. Diagnostics, 13(12), 2023.
  • [23] Lei Tong et al. Cost-sensitive boosting pruning trees for depression detection on twitter. IEEE Transactions on Affective Computing, pages 1–1, 2022.
  • [24] Jingfang Liu and Mengshi Shi. A hybrid feature selection and ensemble approach to identify depressed users in online social media. Frontiers in Psychology, 12, 01 2022.
  • [25] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019.
  • [26] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. arXiv preprint arXiv:2004.09297, 2020.
  • [27] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  • [28] Fabien Ringeval et al. Avec 2019 workshop and challenge: State-of-mind, detecting depression with ai, and cross-cultural affect recognition. In Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, AVEC ’19, page 3–12, New York, NY, USA, 2019. Association for Computing Machinery.
  • [29] L I Lin. A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45 1:255–68, 1989.
  • [30] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, page 448–456. JMLR.org, 2015.
  • [31] Hamdi Dibeklioglu, Zakia Hammal, and Jeffrey Cohn. Dynamic multimodal measurement of depression severity using deep autoencoding. IEEE Journal of Biomedical and Health Informatics, PP:1–1, 03 2017.
  • [32] Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir D. Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. arXiv: Computer Vision and Pattern Recognition, 2014.
  • [33] Mo Zhou, Tianyi Liu, Yan Li, Dachao Lin, Enlu Zhou, and Tuo Zhao. Towards understanding the importance of noise in training neural networks. ArXiv, abs/1909.03172, 2019.
  • [34] Qi Dong, Shaogang Gong, and Xiatian Zhu. Class rectification hard mining for imbalanced deep learning. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 1869–1878, 2017.
  • [35] Max Bucher, Stéphane Herbin, and Frédéric Jurie. Hard negative mining for metric learning based zero-shot classification. In ECCV Workshops, 2016.
  • [36] Ming Fang, Siyu Peng, Yujia Liang, Chih-Cheng Hung, and Shuhua Liu. A multimodal fusion model with multi-level attention mechanism for depression detection. Biomedical Signal Processing and Control, 82:104561, 2023.
  • [37] Congcong Wang, Decheng Liu, Kemeng Tao, Xiaoxiao Cui, Gongtang Wang, Yuefeng Zhao, Zhi Liu. A multi-modal feature layer fusion model for assessment of depression based on attention mechanisms. In 2022 15th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pages 1–6, 2022.
  • [38] Zhuojin Han, Yuanyuan Shang, Zhuhong Shao, Jingyi Liu, Guodong Guo, Tie Liu, Hui Ding, Qiang Hu. Spatial-temporal feature networkfor speech-based depression recognition. IEEE Transactions on Cognitive and Developmental Systems, pages 1–1, 2023.
  • [39] Hao Sun, Jiaqing Liu, Shurong Chai, Zhaolin Qiu, Lanfen Lin, Xinyin Huang, and Yen-Wei Chen. Multi-modal adaptive fusion transformer network for the estimation of depression level. Sensors (Basel, Switzerland), 21, 2021.
  • [40] Shi Yin, Cong Liang, Heyan Ding, and Shangfei Wang. A multi-modal hierarchical recurrent neural network for depression detection. Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, 2019.
There are 40 citations in total.

Details

Primary Language English
Subjects Computer Software
Journal Section Articles
Authors

Ergün Batuhan Kaynak 0000-0002-3249-3343

Hamdi Dibeklioğlu 0000-0003-0851-7808

Early Pub Date April 27, 2024
Publication Date April 30, 2024
Submission Date October 26, 2023
Acceptance Date March 22, 2024
Published in Issue Year 2024

Cite

IEEE E. B. Kaynak and H. Dibeklioğlu, “Systematic analysis of speech transcription modeling for reliable assessment of depression severity”, SAUCIS, vol. 7, no. 1, pp. 77–91, 2024, doi: 10.35377/saucis...1381522.

29070    The papers in this journal are licensed under a Creative Commons Attribution-NonCommercial 4.0 International License