基于残差整流增强卷积神经网络的表情识别

陈斌; 朱晋宁; 东一舟

doi:10.37188/YJYXS20203512.1299

您当前的位置：

首页 >

文章列表页 >

基于残差整流增强卷积神经网络的表情识别

图像处理 | 更新时间：2020-12-04

- 基于残差整流增强卷积神经网络的表情识别
- Expression recognition based on residual rectifier enhanced convolution neural network
- 液晶与显示 2020年35卷第12期页码：1299-1308
- 作者机构：
  
  南京师范大学信息化建设管理处, 江苏南京 210046
- 作者简介：
  
  [ "陈斌(1978-), 男, 江苏南京人, 博士, 高级工程师, 2015年于河海大学获得博士学位, 主要从事云计算、大数据分析、模式识别方面的研究。E-mail:njnuchenbin@njnu.edu.cn" ]
- 基金信息：
  
  国家自然科学基金(61603193);江苏省高校自然科学研究项目(18KJB520027);CERNET Innovation Project(NGII20170524)
- DOI：10.37188/YJYXS20203512.1299
  中图分类号： TP39;TH691.9
- 收稿日期：2020-08-02，
  
  修回日期：2020-08-31，
  
  录用日期：2020-8-31，
  
  纸质出版日期：2020-12
- 稿件说明：
移动端阅览
陈斌, 朱晋宁, 东一舟. 基于残差整流增强卷积神经网络的表情识别[J]. 液晶与显示, 2020,35(12):1299-1308.

Bin CHEN, Jin-ning ZHU, Yi-zhou DONG. Expression recognition based on residual rectifier enhanced convolution neural network[J]. Chinese journal of liquid crystals and displays, 2020, 35(12): 1299-1308.
陈斌, 朱晋宁, 东一舟. 基于残差整流增强卷积神经网络的表情识别[J]. 液晶与显示, 2020,35(12):1299-1308. DOI： 10.37188/YJYXS20203512.1299.

Bin CHEN, Jin-ning ZHU, Yi-zhou DONG. Expression recognition based on residual rectifier enhanced convolution neural network[J]. Chinese journal of liquid crystals and displays, 2020, 35(12): 1299-1308. DOI： 10.37188/YJYXS20203512.1299.

摘要

针对受控场景下人脸表情识别率低的问题，提出了一种受控场景下基于残差整流增强卷积神经网络的表情识别算法。该方法以卷积神经网络为原型，在训练模型的过程中，引入残差网络的思想，修正测试集效果与训练集效果之间的差。通过内嵌于卷积层中的激励函数对残差块的线性整流操作，协助表达复杂特征。同时，通过数据增强方法抑制深度神经网络模型在训练过程中过快的拟合，提升其在给定识别任务上的泛化性能，进而提高模型学习效果的鲁棒性。实验中通过将该方法运用于模拟在线授课环境中，最终达到受控场景下有效人脸表情识别的效果。实验结果表明，该方法可以有效对受控情况下的人脸图像输入进行表情分类，最高准确率达到了91.7%。该研究对人脸表情识别领域及人机交互领域的发展具有积极意义。

Abstract

In order to solve the problem of low face recognition rate in controlled scene

an expression recognition algorithm based on residual rectification enhanced convolutional neural network is proposed. This method takes convolutional neural network as the prototype. In the process of training model

the idea of residual network is introduced to correct the difference between the effect of test set and the effect of training set. The linear rectification operation of the residual block by the excitation function embedded in the convolution layer helps to express complex features. At the same time

the data enhancement method is used to suppress the fast fitting of the deep neural network model during the training process

and also to improve its generalization performance on a given recognition task

and then to improve the robustness of the model learning effect. In the experiment

the method is applied to simulate the online teaching environment

and the effect of effective facial expression recognition in controlled scene is achieved. According to the experimental data

this method can effectively classify the facial image input under controlled conditions

and the highest accuracy is up to 91.7%. This research is helpful to the development of facial expression recognition and human-computer interaction.

关键词

Keywords

references

Y GUROVICH , Y HANANI , O BAR , 等 . Identifying facial phenotypes of genetic disorders using deep learning . Nature Medicine , 2019 . 25 ( 1 ): 60 - 69 . DOI: 10.1038/s41591-018-0279-0 http://doi.org/10.1038/s41591-018-0279-0 .

P EKMAN , W V FRIESEN . Pictures of Facial Affect , : Palo Alto, USA Consulting Psychologists' Press , 1976 . 1 - 7 .

张飞飞 , 张天柱 , 毛启容 , 等 . 基于生成对抗网络的多姿态人脸表情识别 . 计算机学报 , 2019 . 42 ( 120 ): 1 - 16 .

F F ZHANG , T Z ZHANG , Q R MAO , 等 . Multi-pose facial expression recognition via generative adversarial network . Chinese Journal of Computers , 2019 . 42 ( 120 ): 1 - 16 .

谭小慧 , 李昭伟 , 樊亚春 . 基于多尺度细节增强的面部表情识别方法 . 电子与信息学报 , 2019 . 41 ( 11 ): 2752 - 2759 . http://www.cnki.com.cn/Article/CJFDTotal-DZYX201911029.htm http://www.cnki.com.cn/Article/CJFDTotal-DZYX201911029.htm .

X H TAN , Z W LI , Y C FAN . Facial expression recognition method based on multi-scale detail enhancement . Journal of Electronics & Information Technology , 2019 . 41 ( 11 ): 2752 - 2759 . http://www.cnki.com.cn/Article/CJFDTotal-DZYX201911029.htm http://www.cnki.com.cn/Article/CJFDTotal-DZYX201911029.htm .

李宏菲 , 李庆 , 周莉 . 基于多视觉描述子及音频特征的动态序列人脸表情识别 . 电子学报 , 2019 . 47 ( 8 ): 1643 - 1653 . http://www.cnki.com.cn/Article/CJFDTotal-DZXU201908006.htm http://www.cnki.com.cn/Article/CJFDTotal-DZXU201908006.htm .

H F LI , Q LI , L ZHOU . Dynamic facial expression recognition based on multi-visual and audio descriptors . Acta Electronica Sinica , 2019 . 47 ( 8 ): 1643 - 1653 . http://www.cnki.com.cn/Article/CJFDTotal-DZXU201908006.htm http://www.cnki.com.cn/Article/CJFDTotal-DZXU201908006.htm .

K LEKDIOUI , R MESSOUSSI , Y RUICHEK , 等 . Facial decomposition for expression recognition using texture/shape descriptors and SVM classifier . Signal Processing:Image Communication , 2017 . 58 300 - 312 . DOI: 10.1016/j.image.2017.08.001 http://doi.org/10.1016/j.image.2017.08.001 .

夏添 , 张毅锋 , 刘袁 . 基于特征点与多网络联合训练的表情识别 . 计算机辅助设计与图形学学报 , 2019 . 31 ( 4 ): 552 - 559 . http://www.cnki.com.cn/Article/CJFDTotal-JSJF201904005.htm http://www.cnki.com.cn/Article/CJFDTotal-JSJF201904005.htm .

T XIA , Y F ZHANG , Y LIU . Landmark-based facial expression recognition by joint training of multiple networks . Journal of Computer-Aided Design & Computer Graphics , 2019 . 31 ( 4 ): 552 - 559 . http://www.cnki.com.cn/Article/CJFDTotal-JSJF201904005.htm http://www.cnki.com.cn/Article/CJFDTotal-JSJF201904005.htm .

孙晓 , 潘汀 . 基于兴趣区域深度神经网络的静态面部表情识别 . 电子学报 , 2017 . 45 ( 5 ): 1189 - 1197 . http://www.cnki.com.cn/Article/CJFDTOTAL-DZXU201705023.htm http://www.cnki.com.cn/Article/CJFDTOTAL-DZXU201705023.htm .

X SUN , T PAN . Static facial expression recognition system using ROI deep neural networks . Acta Electronica Sinica , 2017 . 45 ( 5 ): 1189 - 1197 . http://www.cnki.com.cn/Article/CJFDTOTAL-DZXU201705023.htm http://www.cnki.com.cn/Article/CJFDTOTAL-DZXU201705023.htm .

Y P LIU , Y B LI , X MA , 等 . Facial expression recognition with fusion features extracted from salient facial areas . Sensors , 2017 . 17 ( 4 ): 712 DOI: 10.3390/s17040712 http://doi.org/10.3390/s17040712 .

贾明兴 , 杜俊强 , 宋鹏飞 , 等 . 基于不同分块多特征优化融合的人脸识别研究 . 东北大学学报(自然科学版) , 2017 . 38 ( 3 ): 310 - 314 .

M X JIA , J Q DU , P F SONG , 等 . Face recognition based on multi-feature optimization fusion of LBP, LPQ and Gabor with multi-scale blocks . Journal of Northeastern University (Natural Science) , 2017 . 38 ( 3 ): 310 - 314 .

LUCEY P, COHN J F, KANADE T, et al . The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression[C]// Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops . San Francisco, USA: IEEE, 2010: 94-101.

GOODFELLOW I J, ERHAN D, CARRIER P L, et al . Challenges in representation learning: A report on three machine learning contests[C]// Proceedings of the 20 th International Conference on Neural Information Processing . Daegu, Korea: Springer, 2013: 117-124.

W M ZHENG , X Y ZHOU , C R ZOU , 等 . Facial expression recognition using kernel canonical correlation analysis (KCCA) . IEEE Transactions on Neural Networks , 2006 . 17 ( 1 ): 233 - 238 . DOI: 10.1109/TNN.2005.860849 http://doi.org/10.1109/TNN.2005.860849 .

H L ZHOU , K M LAM , X J HE . Shape-appearance-correlated active appearance model . Pattern Recognition , 2016 . 56 88 - 99 . DOI: 10.1016/j.patcog.2016.03.002 http://doi.org/10.1016/j.patcog.2016.03.002 .

SU H, MAJI S, KALOGERAKIS E, et al . Multi-view convolutional neural networks for 3D shape recognition[C]// Proceedings of the IEEE International Conference on Computer Vision . Santiago, Chile: IEEE, 2015: 945-953.

H KAYA , F GRPINAR , A A SALAH . Video-based emotion recognition in the wild using deep transfer learning and score fusion . Image and Vision Computing , 2017 . 65 66 - 75 . DOI: 10.1016/j.imavis.2017.01.012 http://doi.org/10.1016/j.imavis.2017.01.012 .

W F GU , C XIANG , Y V VENKATESH , 等 . Facial expression recognition using radial encoding of local Gabor features and classifier synthesis . Pattern Recognition , 2012 . 45 ( 1 ): 80 - 91 .

H SOYEL , H DEMIREL . Localized discriminative scale invariant feature transform based facial expression recognition . Computers & Electrical Engineering , 2012 . 38 ( 5 ): 1299 - 1309 . http://www.sciencedirect.com/science/article/pii/S0045790611001662 http://www.sciencedirect.com/science/article/pii/S0045790611001662 .

刘娟 , 胡敏 , 黄忠 . 基于最优支持度的证据融合表情识别方法 . 电子测量与仪器学报 , 2016 . 30 ( 5 ): 714 - 721 . http://www.cnki.com.cn/Article/CJFDTotal-DZIY201605010.htm http://www.cnki.com.cn/Article/CJFDTotal-DZIY201605010.htm .

J LIU , M HU , Z HUANG . Expression recognition method based on evidence fusion of optimal supportability . Journal of Electronic Measurement and Instrumentation , 2016 . 30 ( 5 ): 714 - 721 . http://www.cnki.com.cn/Article/CJFDTotal-DZIY201605010.htm http://www.cnki.com.cn/Article/CJFDTotal-DZIY201605010.htm .

付傲威 , 赵敏 , 罗令 , 等 . 自由立体显示中基于深度卷积神经网络的虚拟视点生成方法 . 液晶与显示 , 2019 . 34 ( 11 ): 1031 - 1036 . http://cjlcd.lightpublishing.cn/thesisDetails?columnId=1384230&Fpath=&index=-1&l=zh http://cjlcd.lightpublishing.cn/thesisDetails?columnId=1384230&Fpath=&index=-1&l=zh .

A W FU , M ZHAO , L LUO , 等 . Virtual viewpoint image generation method based on deep convolutional neural network in autostereoscopic display . Chinese Journal of Liquid Crystals and Displays , 2019 . 34 ( 11 ): 1031 - 1036 . http://cjlcd.lightpublishing.cn/thesisDetails?columnId=1384230&Fpath=&index=-1&l=zh http://cjlcd.lightpublishing.cn/thesisDetails?columnId=1384230&Fpath=&index=-1&l=zh .

张万征 , 胡志坤 , 李小龙 . 基于LeNet-5的卷积神经图像识别算法 . 液晶与显示 , 2020 . 35 ( 5 ): 486 - 490 . http://cjlcd.lightpublishing.cn/thesisDetails?columnId=1399065&Fpath=&index=-1&l=zh http://cjlcd.lightpublishing.cn/thesisDetails?columnId=1399065&Fpath=&index=-1&l=zh .

W Z ZHANG , Z K HU , X L LI . Convolutional neural image recognition algorithm based on LeNet-5 . Chinese Journal of Liquid Crystals and Displays , 2020 . 35 ( 5 ): 486 - 490 . http://cjlcd.lightpublishing.cn/thesisDetails?columnId=1399065&Fpath=&index=-1&l=zh http://cjlcd.lightpublishing.cn/thesisDetails?columnId=1399065&Fpath=&index=-1&l=zh .

王春哲 , 安军社 , 姜秀杰 , 等 . 基于卷积神经网络的候选区域优化算法 . 中国光学 , 2019 . 12 ( 6 ): 1348 - 1361 .

C Z WANG , J S AN , X J JIANG , 等 . Region proposal optimization algorithm based on convolutional neural networks . Chinese Optics , 2019 . 12 ( 6 ): 1348 - 1361 .

黄倩露 , 王强 . 基于跨连特征融合网络的面部表情识别 . 计算机工程与设计 , 2019 . 40 ( 10 ): 2969 - 2973 .

Q L HUANG , Q WANG . Facial expression recognition based on cross-connect feature fusion network . Computer Engineering and Design , 2019 . 40 ( 10 ): 2969 - 2973 .

GOODFELLOW I, BENGIO Y, COURVILLE A. Deep Learning ( Vol . 1)[M]. Cambridge: MIT Press, 2016: 326-366.

GU J X, WANG Z H, KUEN J, et al . Recent advances in convolutional neural networks[J]. arXiv preprint arXiv : 1512.07108.

王秀友 , 范建中 , 刘华明 , 等 . 基于概率协作表示的多表情序列融合识别 . 激光与光电子学进展 , 2019 . 56 ( 13 ): 131004 http://www.cnki.com.cn/Article/CJFDTotal-JGDJ201913011.htm http://www.cnki.com.cn/Article/CJFDTotal-JGDJ201913011.htm .

X Y WANG , J Z FAN , H M LIU , 等 . Multi-expression sequence fusion recognition based on probabilistic cooperative representation . Laser & Optoelectronics Progress , 2019 . 56 ( 13 ): 131004 http://www.cnki.com.cn/Article/CJFDTotal-JGDJ201913011.htm http://www.cnki.com.cn/Article/CJFDTotal-JGDJ201913011.htm .

姚丽莎 , 徐国明 , 赵凤 . 基于CNN局部特征融合的人脸表情识别 . 激光与光电子学进展 , 2020 . 57 ( 3 ): 031501 .

L S YAO , G M XU , F ZHAO . Facial expression recognition based on CNN local feature fusion . Laser & Optoelectronics Progress , 2020 . 57 ( 3 ): 031501 .

N H BARNOUTI , S S M AL-DABBAGH , W E MATTI , 等 . Face detection and recognition using viola-jones with PCA-LDA and square Euclidean distance . International Journal of Advanced Computer Science and Applications , 2016 . 7 ( 5 ): 371 - 377 .

李勇 , 林小竹 , 蒋梦莹 . 基于跨连接LeNet-5网络的面部表情识别 . 自动化学报 , 2018 . 44 ( 1 ): 176 - 182 . http://www.cnki.com.cn/Article/CJFDTotal-MOTO201801015.htm http://www.cnki.com.cn/Article/CJFDTotal-MOTO201801015.htm .

Y LI , X Z LIN , M Y JIANG . Facial expression recognition with cross-connect LeNet-5 network . Acta Automatica Sinica , 2018 . 44 ( 1 ): 176 - 182 . http://www.cnki.com.cn/Article/CJFDTotal-MOTO201801015.htm http://www.cnki.com.cn/Article/CJFDTotal-MOTO201801015.htm .

GOYAL S J, UPADHYAY A K, JADON R S, et al . RETRACTED CHAPTER: Real-life facial expression recognition systems: A review[M]//SATAPATHY S S, BHATEJA V, DAS S. Smart Computing and Informatics . Singapore: Springer, 2018: 311-331.

Q R MAO , F F ZHANG , L J WANG , 等 . Cascaded multi-level transformed dirichlet process for multi-pose facial expression recognition . The Computer Journal , 2018 . 61 ( 11 ): 1605 - 1619 . DOI: 10.1093/comjnl/bxy016 http://doi.org/10.1093/comjnl/bxy016 .

浏览量

106

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于CNN的异构FPGA硬件加速器设计

残差学习与层注意力相结合的轻量级图像超分辨

基于CNN-Transformer结构的遥感影像变化检测

基于P-U-net的角锥波前探测器的波前校正方法

基于多级联递进卷积结构的图像去雨算法