Research Article
BibTex RIS Cite

Enhancing Facial Expression Recognition in the Wild with Deep Learning Methods Using a New Dataset: RidNet

Year 2019, Volume: 6 Issue: 2, 384 - 396, 26.12.2019
https://doi.org/10.35193/bseufbd.645138

Abstract

In this study, emotion
recognition process was performed by using deep learning methods for seven
different facial expressions with the data set (RidNet) which was created by
using images that are publicly accessible from internet. Afterwards, transfer
learning over RidNet was done with well-known convolutional neural network
architectures such as AlexNet, GoogLeNet and ResNet101. Compound Facial
Expressions of Emotion (CE) and Static Facial Expressions in the Wild (SFEW)
datasets were determined as test datasets. In the first experimental studies,
convolutional neural network architecture with the best classification
performance was determined. This convolutional neural network was trained with
AffectNet, The Karolinska Directed Emotional Faces and RidNet. Similar
classification performances were achieved when the AffectNet, KDEF, and
RidNet-trained networks were tested with the data set (CE) generated in a controlled
environment. In the test data set (SFEW) in an uncontrolled environment,
RidNet-trained network gave a significant advantage over other networks.

Project Number

2019-01.BŞEÜ.03-05

References

  • [1] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014). Going deeper with convolutions. arXiv preprint arXiv:1409.4842.
  • [2] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., & Li, F.-F. (2014). Imagenet large scale visual recognition challenge. arXiv preprint arXiv:1409.0575.
  • [3] Lin, M., Chen, Q., & Yan, S. (2013). Network in network. arXiv preprint arXiv:1312.4400.
  • [4] Krizhevsky, A., Sutskever, I., & Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. 25th International Conference on Neural Information Processing Systems - Volume 1, 1097–1105.
  • [5] Liu, M., Li, S., Shan, S., & Chen, X. (2013) Au-aware deep networks for facial expression recognition. 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 1–6.
  • [6] Kahou, S.E., Pal, C., Bouthillier, X., Froumenty, P., Gulcehre, C., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., Ferrari, R.C., et al. (2013). Combining modality specific deep neural networks for emotion recognition in video. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction, 543–550.
  • [7] Susskind, J.M., Anderson, A.K., & Hinton, G.E. (2010). The toronto face database. Technical report, UTML TR 2010-001, University of Toronto.
  • [8] Dhall, A., Goecke, R., Joshi, J., Wagner, M., & Gedeon, T. (2013). Emotion recognition in the wild challenge 2013. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction, 509–516.
  • [9] Liu, M., Wang, R., Li, S., Shan, S., Huang, Z., & Chen, X. (2014) Combining multiple kernel methods on riemannian manifold for emotion recognition in the wild. In Proceedings of the 16th International Conference on Multimodal Interaction, 494–501.
  • [10] Liu, M., Li, S., Shan, S., Wang, R., & Chen, X. (2014). Deeply learning deformable facial action parts model for dynamic expression analysis. In Computer Vision–ACCV 2014, 143–157.
  • [11] Jung, H., Lee, S., Yim, J., Park, S., Kim, J. (2015). Joint fine-tuning in deep neural networks for facial expression recognition. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 2983–2991.
  • [12] Zhao, K., Chu, W.S. & Zhang, H. (2016). Deep region and multi-label learning for facial action unit detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 3391–3399.
  • [13] Hasani, B., Mahoor, M.H. (2017). Facial expression recognition using enhanced deep 3D convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Hawaii, HI, USA, 1–11.
  • [14] Li, S. & Deng, W. (2018) Deep facial expression recognition: A survey. arXiv preprint arXiv:1804.08348.
  • [15] Zeng, N., Zhang, H., Song, B., Liu, W., Li, Y. & Dobaie, A.M. (2018) Facial expression recognition via learning deep sparse autoencoders. Neurocomputing, 273, 643–649.
  • [16] Otberdout, N., Kacem, A., Daoudi, M., Ballihi, L. & Berretti, S. (2018) Deep covariance descriptors for facial expression recognition. 29th British Machine Vision Conferernce.
  • [17] Li, D., Li, Z., Luo, R., Deng, J. & Sun, S. (2019) Multi-Pose Facial Expression Recognition Based on Generative Adversarial Network. IEEE Access, 7, 143980-143989.
  • [18] Viola, P., & Jones, M.J. (2004). Robust Real-Time Face Detection. International Journal of Computer Vision, 57(2), 137-154.
  • [19] Du, S., Tao, Y., & Martinez, A.M. (2014). Compound facial expressions of emotion. Proceedings of the National Academy of Sciences, 111(15), E1454–E1462.
  • [20] Dhall, A., Murthy, O.R., Goecke, R., Joshi, J., & Gedeon, T. (2015). Video and image based emotion recognition challenges in the wild: Emotiw 2015. International Conference on Multimodal Interaction, 423–426.
  • [21] Adobe Stock, https://stock.adobe.com/, (Erişim tarihi: 08.05.2019).
  • [22] Aydilek, İ.B. (2017). Approximate estimation of the nutritions of consumed food by deep learning. International Conference on Computer Science and Engineering (UBMK), 160-164.
  • [23] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. IEEE Conference on Computer Vision and Pattern Recognition, 1–9.
  • [24] He, H., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 770–778.
  • [25] Savoiu, A., & Wong, J. (2017). Recognizing Facial Expressions Using Deep Learning.

Yeni Bir Veri Kümesi (RidNet) Kullanarak Kontrolsüz Ortamda Yüz İfadesi Tanımanın Derin Öğrenme Yöntemleri ile İyileştirilmesi

Year 2019, Volume: 6 Issue: 2, 384 - 396, 26.12.2019
https://doi.org/10.35193/bseufbd.645138

Abstract

Bu çalışmada, internetten genel erişime açık görüntüler kullanılarak oluşturulan veri kümesi (RidNet) ile yedi farklı yüz ifadesi için derin öğrenme yöntemleri kullanılarak duygu tanıma işlemi yapılmıştır. Daha sonra AlexNet, GoogLeNet ve ResNet101 gibi literatürdeki tanınmış evrişimli sinir ağları mimarileri ile RidNet üzerinden transfer öğrenimi yapılmıştır. Compound Facial Expressions of Emotion (CE) ve Static Facial Expressions in the Wild (SFEW) veri kümeleri test veri kümeleri olarak belirlenmiştir. İlk olarak yapılan deneysel çalışmalar ile en iyi sınıflandırma performansını gösteren evrişimli sinir ağı mimarisi belirlenmiştir. Bu evrişimli sinir ağı AffectNet, The Karolinska Directed Emotional Faces (KDEF) ve RidNet ile eğitilmiştir. AffectNet, KDEF ve RidNet ile eğitilmiş ağlar kontrollü ortamda oluşturulan veri kümesi (CE) ile test edildiğinde benzer sınıflandırma başarımları elde edilmiştir. Kontrolsüz ortamdaki test veri kümesinde (SFEW) ise RidNet ile eğitilen ağ diğer ağlara belirgin bir üstünlük sağlamıştır.

Supporting Institution

Bilecik Şeyh Edebali Üniversitei

Project Number

2019-01.BŞEÜ.03-05

References

  • [1] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014). Going deeper with convolutions. arXiv preprint arXiv:1409.4842.
  • [2] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., & Li, F.-F. (2014). Imagenet large scale visual recognition challenge. arXiv preprint arXiv:1409.0575.
  • [3] Lin, M., Chen, Q., & Yan, S. (2013). Network in network. arXiv preprint arXiv:1312.4400.
  • [4] Krizhevsky, A., Sutskever, I., & Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. 25th International Conference on Neural Information Processing Systems - Volume 1, 1097–1105.
  • [5] Liu, M., Li, S., Shan, S., & Chen, X. (2013) Au-aware deep networks for facial expression recognition. 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 1–6.
  • [6] Kahou, S.E., Pal, C., Bouthillier, X., Froumenty, P., Gulcehre, C., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., Ferrari, R.C., et al. (2013). Combining modality specific deep neural networks for emotion recognition in video. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction, 543–550.
  • [7] Susskind, J.M., Anderson, A.K., & Hinton, G.E. (2010). The toronto face database. Technical report, UTML TR 2010-001, University of Toronto.
  • [8] Dhall, A., Goecke, R., Joshi, J., Wagner, M., & Gedeon, T. (2013). Emotion recognition in the wild challenge 2013. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction, 509–516.
  • [9] Liu, M., Wang, R., Li, S., Shan, S., Huang, Z., & Chen, X. (2014) Combining multiple kernel methods on riemannian manifold for emotion recognition in the wild. In Proceedings of the 16th International Conference on Multimodal Interaction, 494–501.
  • [10] Liu, M., Li, S., Shan, S., Wang, R., & Chen, X. (2014). Deeply learning deformable facial action parts model for dynamic expression analysis. In Computer Vision–ACCV 2014, 143–157.
  • [11] Jung, H., Lee, S., Yim, J., Park, S., Kim, J. (2015). Joint fine-tuning in deep neural networks for facial expression recognition. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 2983–2991.
  • [12] Zhao, K., Chu, W.S. & Zhang, H. (2016). Deep region and multi-label learning for facial action unit detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 3391–3399.
  • [13] Hasani, B., Mahoor, M.H. (2017). Facial expression recognition using enhanced deep 3D convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Hawaii, HI, USA, 1–11.
  • [14] Li, S. & Deng, W. (2018) Deep facial expression recognition: A survey. arXiv preprint arXiv:1804.08348.
  • [15] Zeng, N., Zhang, H., Song, B., Liu, W., Li, Y. & Dobaie, A.M. (2018) Facial expression recognition via learning deep sparse autoencoders. Neurocomputing, 273, 643–649.
  • [16] Otberdout, N., Kacem, A., Daoudi, M., Ballihi, L. & Berretti, S. (2018) Deep covariance descriptors for facial expression recognition. 29th British Machine Vision Conferernce.
  • [17] Li, D., Li, Z., Luo, R., Deng, J. & Sun, S. (2019) Multi-Pose Facial Expression Recognition Based on Generative Adversarial Network. IEEE Access, 7, 143980-143989.
  • [18] Viola, P., & Jones, M.J. (2004). Robust Real-Time Face Detection. International Journal of Computer Vision, 57(2), 137-154.
  • [19] Du, S., Tao, Y., & Martinez, A.M. (2014). Compound facial expressions of emotion. Proceedings of the National Academy of Sciences, 111(15), E1454–E1462.
  • [20] Dhall, A., Murthy, O.R., Goecke, R., Joshi, J., & Gedeon, T. (2015). Video and image based emotion recognition challenges in the wild: Emotiw 2015. International Conference on Multimodal Interaction, 423–426.
  • [21] Adobe Stock, https://stock.adobe.com/, (Erişim tarihi: 08.05.2019).
  • [22] Aydilek, İ.B. (2017). Approximate estimation of the nutritions of consumed food by deep learning. International Conference on Computer Science and Engineering (UBMK), 160-164.
  • [23] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. IEEE Conference on Computer Vision and Pattern Recognition, 1–9.
  • [24] He, H., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 770–778.
  • [25] Savoiu, A., & Wong, J. (2017). Recognizing Facial Expressions Using Deep Learning.
There are 25 citations in total.

Details

Primary Language Turkish
Subjects Engineering
Journal Section Articles
Authors

Rıdvan Özdemir This is me 0000-0002-8599-1709

Mehmet Koç 0000-0003-2919-6011

Project Number 2019-01.BŞEÜ.03-05
Publication Date December 26, 2019
Submission Date November 11, 2019
Acceptance Date December 4, 2019
Published in Issue Year 2019 Volume: 6 Issue: 2

Cite

APA Özdemir, R., & Koç, M. (2019). Yeni Bir Veri Kümesi (RidNet) Kullanarak Kontrolsüz Ortamda Yüz İfadesi Tanımanın Derin Öğrenme Yöntemleri ile İyileştirilmesi. Bilecik Şeyh Edebali Üniversitesi Fen Bilimleri Dergisi, 6(2), 384-396. https://doi.org/10.35193/bseufbd.645138