Skip to main content

22.07.2019 | Special Issue Paper

Person-independent facial expression recognition method based on improved Wasserstein generative adversarial networks in combination with identity aware

verfasst von: Caie Xu, Yang Cui, Yunhui Zhang, Peng Gao, Jiayi Xu

Erschienen in: Multimedia Systems

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Since the distinction between two expressions is fairly vague, usually a subtle change in one part of the human face is enough to change a facial expression. Most of the existing facial expression recognition algorithms are not robust enough because they rely on general facial features or algorithms without considering differences between facial expression and facial identity. In this paper, we propose a person-independent recognition method based on Wasserstein generative adversarial networks for micro-facial expressions, where a facial expression recognition network and a facial identity recognition network are established to improve the accuracy and robustness of facial expression recognition via inhibition of intra-class variation. Extensive experimental results demonstrate that 90% average recognition accuracy of facial expression has been reached on a mixed dataset composed of CK+, Multi-PIE, and JAFFE. Moreover, our method achieves 96% accuracy of person-independent recognition on CK+. A 4.5% performance gain is achieved with the novel identity-inhibited expression feature. The proposed algorithm in this paper has been successfully applied to Haikang Visual Integrated Management Platform (iVMS-8700). At present, it runs well and can effectively recognize facial expressions.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Xia, K., Yin, H., Zhang, Y.: Deep semantic segmentation of kidney and space-occupying lesion area based on SCNN and ResNet models combined with SIFT-flow algorithm. J. Med. Syst. 87(43), 1253–1268 (2019) Xia, K., Yin, H., Zhang, Y.: Deep semantic segmentation of kidney and space-occupying lesion area based on SCNN and ResNet models combined with SIFT-flow algorithm. J. Med. Syst. 87(43), 1253–1268 (2019)
2.
Zurück zum Zitat Shan, C., Gong, S., Mcowan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)CrossRef Shan, C., Gong, S., Mcowan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)CrossRef
3.
Zurück zum Zitat Cohen, I., Sebe, N., Garg, A., et al.: Facial expression recognition from video sequences. Comput. Vis. Image Underst. 91(1–2), 160–187 (2003)CrossRef Cohen, I., Sebe, N., Garg, A., et al.: Facial expression recognition from video sequences. Comput. Vis. Image Underst. 91(1–2), 160–187 (2003)CrossRef
4.
Zurück zum Zitat Cohen, I., Sebe, N., Garg, A., et al.: Facial expression recognition from video sequences. In: Proceedings. IEEE International Conference on Multimedia and Expo, vol. 24, no. 2, pp. 121–124 (2002) Cohen, I., Sebe, N., Garg, A., et al.: Facial expression recognition from video sequences. In: Proceedings. IEEE International Conference on Multimedia and Expo, vol. 24, no. 2, pp. 121–124 (2002)
5.
Zurück zum Zitat Lopes, A.T., Aguiar, E.D., Souza, A.F.D., et al.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn. 61, 610–628 (2016)CrossRef Lopes, A.T., Aguiar, E.D., Souza, A.F.D., et al.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn. 61, 610–628 (2016)CrossRef
6.
Zurück zum Zitat Lee, S.H., Plataniotis, K.N.K., Ro, Y.M.: Intra-class variation reduction using training expression images for sparse representation based facial expression recognition. IEEE Trans. Affect. Comput. 5(3), 340–351 (2014)CrossRef Lee, S.H., Plataniotis, K.N.K., Ro, Y.M.: Intra-class variation reduction using training expression images for sparse representation based facial expression recognition. IEEE Trans. Affect. Comput. 5(3), 340–351 (2014)CrossRef
7.
Zurück zum Zitat Liu, M., Shan, S., Wang, R., et al.: Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. IEEE Trans. Image Process. 25(12), 5920–5932 (2016)MathSciNetCrossRef Liu, M., Shan, S., Wang, R., et al.: Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. IEEE Trans. Image Process. 25(12), 5920–5932 (2016)MathSciNetCrossRef
8.
Zurück zum Zitat Yanpeng, L., Yibin, L., Xin, M., et al.: Facial expression recognition with fusion features extracted from salient facial areas. Sensors 17(4), 712–729 (2017)CrossRef Yanpeng, L., Yibin, L., Xin, M., et al.: Facial expression recognition with fusion features extracted from salient facial areas. Sensors 17(4), 712–729 (2017)CrossRef
9.
Zurück zum Zitat Jiang, Y., Chung, F.-L., Ishibuchi, H., et al.: Multitask TSK fuzzy system modeling by mining intertask common hidden structure. IEEE Trans. Cybern. 45(3), 548–561 (2015)CrossRef Jiang, Y., Chung, F.-L., Ishibuchi, H., et al.: Multitask TSK fuzzy system modeling by mining intertask common hidden structure. IEEE Trans. Cybern. 45(3), 548–561 (2015)CrossRef
10.
Zurück zum Zitat Li, H., Sun, J., Xu, Z., et al.: Multimodal 2D + 3D facial expression recognition with deep fusion convolutional neural network. IEEE Trans. Multimed. 2017, 1 (2017) Li, H., Sun, J., Xu, Z., et al.: Multimodal 2D + 3D facial expression recognition with deep fusion convolutional neural network. IEEE Trans. Multimed. 2017, 1 (2017)
11.
Zurück zum Zitat Ghimire, D., Jeong, S., Joonwhoan, L.: Facial expression recognition based on local region specific features and support vector machines. Multimed. Tools Appl. 76(6), 7803–7821 (2017)CrossRef Ghimire, D., Jeong, S., Joonwhoan, L.: Facial expression recognition based on local region specific features and support vector machines. Multimed. Tools Appl. 76(6), 7803–7821 (2017)CrossRef
12.
Zurück zum Zitat Jiang, Y., Chung, F.-L., Wang, S., Deng, Z., Wang, J., Qian, P.: Collaborative fuzzy clustering from multiple weighted views. IEEE Trans. Cybern. 45(4), 688–701 (2015)CrossRef Jiang, Y., Chung, F.-L., Wang, S., Deng, Z., Wang, J., Qian, P.: Collaborative fuzzy clustering from multiple weighted views. IEEE Trans. Cybern. 45(4), 688–701 (2015)CrossRef
13.
Zurück zum Zitat Valstar, M.F., Sánchez-Lozano, E., Cohn, J.F., et al.: Fera 2017-addressing head pose in the third facial expression recognition and analysis challenge. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA, 30 May–3 June 2017, pp. 839–847 (2017) Valstar, M.F., Sánchez-Lozano, E., Cohn, J.F., et al.: Fera 2017-addressing head pose in the third facial expression recognition and analysis challenge. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA, 30 May–3 June 2017, pp. 839–847 (2017)
14.
Zurück zum Zitat Bazrafkan, S., Nedelcu, T., Filipczuk, P., et al.: Deep learning for facial expression recognition: a step closer to a smartphone that knows your moods. In: 2017 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA, 8–10 January 2017, pp. 1208–1223 (2017) Bazrafkan, S., Nedelcu, T., Filipczuk, P., et al.: Deep learning for facial expression recognition: a step closer to a smartphone that knows your moods. In: 2017 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA, 8–10 January 2017, pp. 1208–1223 (2017)
15.
Zurück zum Zitat Hasani, B., Arzani, M.M., Fathy, M., et al.: Facial expression recognition with discriminatory graphical models. In: 2016 2nd International Conference of Signal Processing and Intelligent Systems (ICSPIS), Tehran, Iran, 14–15 December 2016, pp. 1–7 (2016) Hasani, B., Arzani, M.M., Fathy, M., et al.: Facial expression recognition with discriminatory graphical models. In: 2016 2nd International Conference of Signal Processing and Intelligent Systems (ICSPIS), Tehran, Iran, 14–15 December 2016, pp. 1–7 (2016)
16.
Zurück zum Zitat Meng, Z., Liu, P., Cai, J., et al.: Identity-aware convolutional neural network for facial expression recognition. IEEE Comput. Soc. 24(23), 562–573 (2017) Meng, Z., Liu, P., Cai, J., et al.: Identity-aware convolutional neural network for facial expression recognition. IEEE Comput. Soc. 24(23), 562–573 (2017)
17.
Zurück zum Zitat Hsu, S.C., Huang, H.H., Huang, C.L.: Facial expression recognition for human-robot interaction. In: 2017 First IEEE International Conference on Robotic Computing (IRC), Taichung, Taiwan, 10–12 April 2017. IEEE (2017) Hsu, S.C., Huang, H.H., Huang, C.L.: Facial expression recognition for human-robot interaction. In: 2017 First IEEE International Conference on Robotic Computing (IRC), Taichung, Taiwan, 10–12 April 2017. IEEE (2017)
18.
Zurück zum Zitat Zhang, C., Wang, P., Chen, K., et al.: Identity-aware convolutional neural networks for facial expression recognition. J. Syst. Eng. Electron. 28(4), 784–792 (2017)CrossRef Zhang, C., Wang, P., Chen, K., et al.: Identity-aware convolutional neural networks for facial expression recognition. J. Syst. Eng. Electron. 28(4), 784–792 (2017)CrossRef
19.
Zurück zum Zitat Uddin, M.Z., Hassan, M.M., Almogren, A., et al.: A facial expression recognition system using robust face features from depth videos and deep learning. Comput. Electr. Eng. 63, 114–125 (2017)CrossRef Uddin, M.Z., Hassan, M.M., Almogren, A., et al.: A facial expression recognition system using robust face features from depth videos and deep learning. Comput. Electr. Eng. 63, 114–125 (2017)CrossRef
20.
Zurück zum Zitat Qian, P., Jiang, Y., Deng, Z., Hu, L., Sun, S., Wang, S., Muzic R.F. Jr.: Cluster prototypes and fuzzy memberships jointly leveraged cross-domain maximum entropy clustering. IEEE Trans. Cybern. 46(1), 181–193 (2016)CrossRef Qian, P., Jiang, Y., Deng, Z., Hu, L., Sun, S., Wang, S., Muzic R.F. Jr.: Cluster prototypes and fuzzy memberships jointly leveraged cross-domain maximum entropy clustering. IEEE Trans. Cybern. 46(1), 181–193 (2016)CrossRef
21.
Zurück zum Zitat Liu, X., Kumar, B.V.K.V, You, J., et al.: Adaptive deep metric learning for identity-aware facial expression recognition. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA, 21–26 July 2017, pp. 20–29 (2017) Liu, X., Kumar, B.V.K.V, You, J., et al.: Adaptive deep metric learning for identity-aware facial expression recognition. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA, 21–26 July 2017, pp. 20–29 (2017)
22.
Zurück zum Zitat Baddar, W.J., Kim, D.H., Ro, Y.M.: Learning features robust to image variations with siamese networks for facial expression recognition. In: Amsaleg L., Guðmundsson G., Gurrin C., Jónsson B., Satoh S. (eds) International Conference on Multimedia Modeling, pp.189–200. Springer, Cham (2017) Baddar, W.J., Kim, D.H., Ro, Y.M.: Learning features robust to image variations with siamese networks for facial expression recognition. In: Amsaleg L., Guðmundsson G., Gurrin C., Jónsson B., Satoh S. (eds) International Conference on Multimedia Modeling, pp.189–200. Springer, Cham (2017)
23.
Zurück zum Zitat Li, J., Wang, S., Huo, J., Guo, H., Zhao, H., Mei, Y.: Bayesian framework with non-local and low-rank constraint for image reconstruction. J. Phys. Conf. Ser. 787, 012008 (2017)CrossRef Li, J., Wang, S., Huo, J., Guo, H., Zhao, H., Mei, Y.: Bayesian framework with non-local and low-rank constraint for image reconstruction. J. Phys. Conf. Ser. 787, 012008 (2017)CrossRef
24.
Zurück zum Zitat Sariyanidi, E., Gunes, H., Cavallaro, A.: Learning bases of activity for facial expression recognition. IEEE Trans. Image Process. 26(4), 1965–1978 (2017)MathSciNetCrossRef Sariyanidi, E., Gunes, H., Cavallaro, A.: Learning bases of activity for facial expression recognition. IEEE Trans. Image Process. 26(4), 1965–1978 (2017)MathSciNetCrossRef
25.
Zurück zum Zitat Jang, J., Kim, D.H., Kim, H.I., et al.: Color channel-wise recurrent learning for facial expression recognition. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA, 5–9 March 2017, pp. 1233–1237 (2017) Jang, J., Kim, D.H., Kim, H.I., et al.: Color channel-wise recurrent learning for facial expression recognition. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA, 5–9 March 2017, pp. 1233–1237 (2017)
26.
Zurück zum Zitat Xia, K.J., Yin, H.S., Wang, J.Q.: A novel improved deep convolutional neural network model for medical image fusion. Cluster Comput. 24(3), 1–13 (2018) Xia, K.J., Yin, H.S., Wang, J.Q.: A novel improved deep convolutional neural network model for medical image fusion. Cluster Comput. 24(3), 1–13 (2018)
27.
Zurück zum Zitat Fathallah, A., Abdi, L., Douik, A.: Facial expression recognition via deep learning. In: 2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA), Hammamet, Tunisia, 30 October–3 November 2017, vol. 127, no. 99, pp. 107–114. IEEE Computer Society (2017) Fathallah, A., Abdi, L., Douik, A.: Facial expression recognition via deep learning. In: 2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA), Hammamet, Tunisia, 30 October–3 November 2017, vol. 127, no. 99, pp. 107–114. IEEE Computer Society (2017)
28.
Zurück zum Zitat Mavani, V., Raman, S., Miyapuram, K.P.: Facial expression recognition using visual saliency and deep learning. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy, 22–29 October 2017, pp. 2783–2788 (2017) Mavani, V., Raman, S., Miyapuram, K.P.: Facial expression recognition using visual saliency and deep learning. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy, 22–29 October 2017, pp. 2783–2788 (2017)
Metadaten
Titel
Person-independent facial expression recognition method based on improved Wasserstein generative adversarial networks in combination with identity aware
verfasst von
Caie Xu
Yang Cui
Yunhui Zhang
Peng Gao
Jiayi Xu
Publikationsdatum
22.07.2019
Verlag
Springer Berlin Heidelberg
Erschienen in
Multimedia Systems
Print ISSN: 0942-4962
Elektronische ISSN: 1432-1882
DOI
https://doi.org/10.1007/s00530-019-00628-6