基于视频放大和双分支网络的微表情识别

李召峰; 朱明

doi:10.37188/CJLCD.2021-0320

您当前的位置：

首页 >

文章列表页 >

基于视频放大和双分支网络的微表情识别

图像处理 | 更新时间：2022-03-25

- 基于视频放大和双分支网络的微表情识别
- Micro-expression recognition based on video magnification and dual-branch network
- 液晶与显示 2022年37卷第3期页码：386-394
- 作者机构：
  
  1.中国科学院长春光学精密机械与物理研究所，吉林长春 130033
  2.中国科学院大学，北京 100049
- 作者简介：
  
  [ "李召峰 (1991-)，男，甘肃天水人，硕士研究生，2013年于西北工业大学获得学士学位，主要从事图像处理、机器视觉方面的研究。E-mail: lzf0215@163.com" ]
  [ "朱明 (1964-)，男，江西南昌人，硕士，研究员，1991年于中国科学院长春光学精密机械研究所获得硕士学位，主要从事视频图像处理，自动目标识别技术及成像目标跟踪等方面的研究。E-mail: zhu_mingca@163.com" ]
- 基金信息：
  
  国家自然科学基金青年基金(61401425)
- DOI：10.37188/CJLCD.2021-0320
  中图分类号： TP391.4;TH691.9
- 收稿日期：2021-12-19，
  
  修回日期：2021-12-24，
  
  纸质出版日期：2022-03
- 稿件说明：
移动端阅览
李召峰, 朱明. 基于视频放大和双分支网络的微表情识别[J]. 液晶与显示, 2022,37(3):386-394.

Zhao-feng LI, Ming ZHU. Micro-expression recognition based on video magnification and dual-branch network[J]. Chinese journal of liquid crystals and displays, 2022, 37(3): 386-394.
李召峰, 朱明. 基于视频放大和双分支网络的微表情识别[J]. 液晶与显示, 2022,37(3):386-394. DOI： 10.37188/CJLCD.2021-0320.

Zhao-feng LI, Ming ZHU. Micro-expression recognition based on video magnification and dual-branch network[J]. Chinese journal of liquid crystals and displays, 2022, 37(3): 386-394. DOI： 10.37188/CJLCD.2021-0320.

摘要

微表情是指当人们想要隐藏其真实情绪时，所自发产生的表情，持续时间在1/5 s之内，其面部肌肉运动短暂、幅度微小，有限的数据集使得特征提取变得困难，给其识别带来巨大的挑战。针对这些问题，本文提出一种基于图像预处理技术和双分支网络的识别方法，首先利用先进的人脸对齐网络获取有效的面部表情区域，再采取欧拉视频放大技术捕捉面部动作的微弱变化，以及提取光流信息作为视频序列的特征，然后将光流信息不同组合的特征图输入到双分支分类网络中得到表情标签输出。在SMIC和CASME Ⅱ两个公开的数据集上进行实验，采用留一法交叉验证，准确率分别达到0.545和0.584，实验结果的定量分析和定性分析均验证了所提出识别模型的有效性。

Abstract

Micro-expression refers to the expression that people spontaneously generate when they want to hide their true emotions. The duration is within 1/5 s. Their facial muscles are short and small in amplitude. The limited data set makes feature extraction difficult. These factors bring huge challenges to its recognition. In response to these problems

this paper proposes a recognition method based on image preprocessing technology and dual-branch network. Firstly

an advanced face alignment network is used to obtain effective facial regions

and then Euler video magnification technology is used to capture the subtle changes in facial movements. Optical flow information is used as the feature of the video sequence

and the feature maps of different combinations of optical flow information are input into the two-branch classification network to obtain the expression label output. Experiments aere performed on the two public data sets of SMIC and CASME Ⅱ

and the leave-one-subject-out(LOSO) is employed for cross-validation. The accuracy can reach 0.545 and 0.584

respectively. The quantitative and qualitative analysis of the experimental results verify the effectiveness of the proposed method.

关键词

Keywords

references

P EKMAN , W V FRIESEN . Nonverbal leakage and clues to deception . Psychiatry , 1969 . 32 ( 1 ): 88 - 106 . DOI: 10.1080/00332747.1969.11023575 http://doi.org/10.1080/00332747.1969.11023575 .

W J YAN , Q WU , J LIANG , 等 . How fast are the leaked facial expressions: the duration of micro-expressions . Journal of Nonverbal Behavior , 2013 . 37 ( 4 ): 217 - 230 . DOI: 10.1007/s10919-013-0159-8 http://doi.org/10.1007/s10919-013-0159-8 .

WANG S J, YAN W J, LI X B, et al . Micro-expression recognition using dynamic textures on tensor independent color space [C]// Proceedings of the 22nd International Conference on Pattern Recognition . Stockholm: IEEE, 2014: 4678-4683.

J ENDRES , A LAIDLAW . Micro-expression recognition training in medical students: a pilot study . BMC Medical Education , 2009 . 9 ( 1 ): 47 DOI: 10.1186/1472-6920-9-47 http://doi.org/10.1186/1472-6920-9-47 .

FRANK M G, KIM D J, KANG S, et al. Improving the ability to detect micro expressions in law enforcement officers. Manuscript in preparation, 2014.

A K DAVISON , W MERGHANI , M H YAP . Objective classes for micro-facial expression recognition . Journal of Imaging , 2018 . 4 ( 10 ): 119 DOI: 10.3390/jimaging4100119 http://doi.org/10.3390/jimaging4100119 .

EKMAN M P. Micro expression training tool[R]. Oakland: CD-ROM, 2003.

LI X B, PFISTER T, HUANG X H, et al . A spontaneous micro-expression database: inducement, collection and baseline [C]// Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition . Shanghai, China: IEEE, 2013: 1-6.

YAN W J, WU Q, LIU Y J, et al . CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces [C]// Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition . Shanghai, China: IEEE, 2013: 1-7.

W J YAN , X B LI , S J WANG , 等 . CASME Ⅱ: an improved spontaneous micro-expression database and the baseline evaluation . PLoS One , 2014 . 9 ( 1 ): e86041 .

Y J LIU , J K ZHANG , W J YAN , 等 . A main directional mean optical flow feature for spontaneous micro-expression recognition . IEEE Transactions on Affective Computing , 2016 . 7 ( 4 ): 299 - 310 . DOI: 10.1109/TAFFC.2015.2485205 http://doi.org/10.1109/TAFFC.2015.2485205 .

CRISTINACCE D, COOTES T F. Feature detection and tracking with constrained local models [C]// Proceedings of the British Machine Vision Conference 2006 . Edinburgh, 2006: 929-938.

LIONG S T, SEE J, WONG K, et al . Automatic micro-expression recognition from long video using a single spotted apex [C] // Proceedings of the Asian Conference on Computer Vision . Taipei, China: Springer, 2016: 345-360.

LIONG S T, SEE J, WONG K, et al . Automatic apex frame spotting in micro-expression database [C]// Proceedings of the 3rd IAPR Asian Conference on Pattern Recognition ( ACPR ). Kuala Lumpur: IEEE, 2015: 665-669.

KIM D H, BADDAR W J, RO Y M. Micro-expression recognition with expression-state constrained spatio-temporal feature representations [C]// Proceedings of the 24th ACM International Conference on Multimedia . Amsterdam, Netherlands: ACM, 2016: 382-386.

J LI , Y D WANG , J SEE , 等 . Micro-expression recognition based on 3D flow convolutional neural network . Pattern Analysis and Applications , 2019 . 22 ( 4 ): 1331 - 1339 . DOI: 10.1007/s10044-018-0757-5 http://doi.org/10.1007/s10044-018-0757-5 .

KHOR H Q, SEE J, PHAN R C W, et al . Enriched long-term recurrent convolutional network for facial micro-expression recognition [C]// Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition ( FG 2018 ). Xi'an, China: IEEE, 2018.

LEI L, LI J F, CHEN T, et al . A novel graph-TCN with a graph structured representation for micro-expression recognition [C]// Proceedings of the 28th ACM International Conference on Multimedia . Seattle: ACM, 2020.

LIU J M, LI K, SONG B L, et al . A multi-stream convolutional neural network for micro-expression recognition using optical flow and EVM [J]. arXiv: 2011.03756, 2020.

BULAT A, TZIMIROPOULOS G. How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230, 000 3D facial landmarks) [C]// Proceedings of 2017 IEEE International Conference on Computer Vision . Venice: IEEE, 2017: 1021-1030.

NEWELL A, YANG K Y, DENG J. Stacked hourglass networks for human pose estimation [C]// Proceedings of the 14th European Conference on Computer Vision . Amsterdam: Springer, 2016.

ZHU X X, RAMANAN D. Face detection, pose estimation, and landmark localization in the wild [C]// Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition . Providence: IEEE, 2012.

Y H WU , M RUBINSTEIN , E SHIH , 等 . Eulerian video magnification for revealing subtle changes in the world . ACM Transactions on Graphics , 2012 . 31 ( 4 ): 65 .

S T LIONG , J SEE , K WONG , 等 . Less is more: micro-expression recognition from video using apex frame . Signal Processing: Image Communication , 2018 . 62 82 - 92 . DOI: 10.1016/j.image.2017.11.006 http://doi.org/10.1016/j.image.2017.11.006 .

ILG E, MAYER N, SAIKIA T, et al . FlowNet 2.0: evolution of optical flow estimation with deep networks [C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu: IEEE, 2017.

GODAVARTHY S. Microexpression spotting in video using optical strain [D]. Florida: University of South Florida, 2010.

HE K M, ZHANG X Y, REN S Q, et al . Deep residual learning for image recognition [C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas: IEEE, 2016: 770-778.

HE K M, ZHANG X Y, REN S Q, et al . Identity mappings in deep residual networks [C]// Proceedings of the 14th European Conference on Computer Vision . Amsterdam: Springer, 2016.

WOO S, PARK J, LEE J Y, et al . CBAM: convolutional block attention module [C]// Proceedings of the 15th European Conference on Computer Vision . Munich: Springer, 2018: 3-19.

ZAGORUYKO S, KOMODAKIS N. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer [C]// Proceedings of the 5th International Conference on Learning Representations . Toulon: OpenReview. net, 2017.

PENG M, WU Z, ZHANG Z H, et al . From macro to micro expression recognition: deep learning on small datasets using transfer learning [C]// Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition . Xi′an, China: IEEE, 2018: 657-661.

S J WANG , B J LI , Y J LIU , 等 . Micro-expression recognition with small sample size by transferring long-term convolutional neural network . Neurocomputing , 2018 . 312 251 - 262 . DOI: 10.1016/j.neucom.2018.05.107 http://doi.org/10.1016/j.neucom.2018.05.107 .

KHOR H Q, SEE J, LIONG S T, et al . Dual-stream shallow networks for facial micro-expression recognition [C]// Proceedings of 2019 IEEE International Conference on Image Processing ( ICIP ). Taipei, China: IEEE, 2019: 36-40.

G ZHAO , M PIETIKALNEN . Dynamic texture recognition using local binary patterns with an application to facial expressions . TEEE Trcms. Pattern Anal. Mach. Intell. , 2007 . 29 ( 6 ): 915 - 928 . DOI: 10.1109/TPAMI.2007.1110 http://doi.org/10.1109/TPAMI.2007.1110 .

F XU , J P ZHANG , J Z WANG . Micro expression identification and categorization using a facial dynamics map . IEEE Transactions on Affective Computing , 2017 . 8 ( 2 ): 254 - 267 . DOI: 10.1109/TAFFC.2016.2518162 http://doi.org/10.1109/TAFFC.2016.2518162 .

WANG Y, SEE J, PHAN R C, et al. LBP with six intersection points: Reducing redundant information in LBP-TOP for micro-expression recog nition [C]// Proc. Asian Conf. Comput. Vis. (Beijing: IEEE), 2014: 525-537.

TAKALKAR M A, XU M. Image based facial micro-expression recognition using deep learning on small datasets [C]// Proceedings of 2017 International Conference on Digital Image Computing : Techniques and Applications ( DICTA ). Sydney: IEEE, 2017: 1-7.

SELVARAJU R R, COGSWELL M, DAS A, et al . Grad-CAM: visual explanations from deep networks via gradient-based localization [C]// Proceedings of 2017 IEEE International Conference on Computer Vision . Venice: IEEE, 2017: 618-626.

P EKMAN , W V FRIESEN . Facial Action Coding System: Investigator's Guide , : Palo Alto Consulting Psychologists Press , 1978 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

融合序列影像相关区域信息的光流估计网络

一种裸眼3D显示中的多视点校正方案