Micro-expression recognition based on video magnification and dual-branch network

Zhao-feng LI; Ming ZHU

doi:10.37188/CJLCD.2021-0320

您当前的位置：

首页 >

文章列表页 >

Micro-expression recognition based on video magnification and dual-branch network

Image Processing | 更新时间：2022-03-25

- Micro-expression recognition based on video magnification and dual-branch network
- Chinese Journal of Liquid Crystals and Displays Vol. 37, Issue 3, Pages: 386-394(2022)
- 作者机构：
  
  1.中国科学院长春光学精密机械与物理研究所，吉林长春 130033
  2.中国科学院大学，北京 100049
- 作者简介：
- 基金信息：
  
  Youth Program of National Natural Science Foundation of China(61401425)
- DOI：10.37188/CJLCD.2021-0320
  CLC： TP391.4;TH691.9
- Received：19 December 2021，
  
  Revised：24 December 2021，
  
  Published：2022-03
- 稿件说明：
移动端阅览
Zhao-feng LI, Ming ZHU. Micro-expression recognition based on video magnification and dual-branch network[J]. Chinese journal of liquid crystals and displays, 2022, 37(3): 386-394.
DOI：

Zhao-feng LI, Ming ZHU. Micro-expression recognition based on video magnification and dual-branch network[J]. Chinese journal of liquid crystals and displays, 2022, 37(3): 386-394. DOI： 10.37188/CJLCD.2021-0320.

摘要

微表情是指当人们想要隐藏其真实情绪时，所自发产生的表情，持续时间在1/5 s之内，其面部肌肉运动短暂、幅度微小，有限的数据集使得特征提取变得困难，给其识别带来巨大的挑战。针对这些问题，本文提出一种基于图像预处理技术和双分支网络的识别方法，首先利用先进的人脸对齐网络获取有效的面部表情区域，再采取欧拉视频放大技术捕捉面部动作的微弱变化，以及提取光流信息作为视频序列的特征，然后将光流信息不同组合的特征图输入到双分支分类网络中得到表情标签输出。在SMIC和CASME Ⅱ两个公开的数据集上进行实验，采用留一法交叉验证，准确率分别达到0.545和0.584，实验结果的定量分析和定性分析均验证了所提出识别模型的有效性。

Abstract

Micro-expression refers to the expression that people spontaneously generate when they want to hide their true emotions. The duration is within 1/5 s. Their facial muscles are short and small in amplitude. The limited data set makes feature extraction difficult. These factors bring huge challenges to its recognition. In response to these problems

this paper proposes a recognition method based on image preprocessing technology and dual-branch network. Firstly

an advanced face alignment network is used to obtain effective facial regions

and then Euler video magnification technology is used to capture the subtle changes in facial movements. Optical flow information is used as the feature of the video sequence

and the feature maps of different combinations of optical flow information are input into the two-branch classification network to obtain the expression label output. Experiments aere performed on the two public data sets of SMIC and CASME Ⅱ

and the leave-one-subject-out(LOSO) is employed for cross-validation. The accuracy can reach 0.545 and 0.584

respectively. The quantitative and qualitative analysis of the experimental results verify the effectiveness of the proposed method.

关键词

Keywords

references

P EKMAN , W V FRIESEN . Nonverbal leakage and clues to deception . Psychiatry , 1969 . 32 ( 1 ): 88 - 106 . DOI: 10.1080/00332747.1969.11023575 http://doi.org/10.1080/00332747.1969.11023575 .

W J YAN , Q WU , J LIANG , 等 . How fast are the leaked facial expressions: the duration of micro-expressions . Journal of Nonverbal Behavior , 2013 . 37 ( 4 ): 217 - 230 . DOI: 10.1007/s10919-013-0159-8 http://doi.org/10.1007/s10919-013-0159-8 .

WANG S J, YAN W J, LI X B, et al . Micro-expression recognition using dynamic textures on tensor independent color space [C]// Proceedings of the 22nd International Conference on Pattern Recognition . Stockholm: IEEE, 2014: 4678-4683.

J ENDRES , A LAIDLAW . Micro-expression recognition training in medical students: a pilot study . BMC Medical Education , 2009 . 9 ( 1 ): 47 DOI: 10.1186/1472-6920-9-47 http://doi.org/10.1186/1472-6920-9-47 .

FRANK M G, KIM D J, KANG S, et al. Improving the ability to detect micro expressions in law enforcement officers. Manuscript in preparation, 2014.

A K DAVISON , W MERGHANI , M H YAP . Objective classes for micro-facial expression recognition . Journal of Imaging , 2018 . 4 ( 10 ): 119 DOI: 10.3390/jimaging4100119 http://doi.org/10.3390/jimaging4100119 .

EKMAN M P. Micro expression training tool[R]. Oakland: CD-ROM, 2003.

LI X B, PFISTER T, HUANG X H, et al . A spontaneous micro-expression database: inducement, collection and baseline [C]// Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition . Shanghai, China: IEEE, 2013: 1-6.

YAN W J, WU Q, LIU Y J, et al . CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces [C]// Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition . Shanghai, China: IEEE, 2013: 1-7.

W J YAN , X B LI , S J WANG , 等 . CASME Ⅱ: an improved spontaneous micro-expression database and the baseline evaluation . PLoS One , 2014 . 9 ( 1 ): e86041 .

Y J LIU , J K ZHANG , W J YAN , 等 . A main directional mean optical flow feature for spontaneous micro-expression recognition . IEEE Transactions on Affective Computing , 2016 . 7 ( 4 ): 299 - 310 . DOI: 10.1109/TAFFC.2015.2485205 http://doi.org/10.1109/TAFFC.2015.2485205 .

CRISTINACCE D, COOTES T F. Feature detection and tracking with constrained local models [C]// Proceedings of the British Machine Vision Conference 2006 . Edinburgh, 2006: 929-938.

LIONG S T, SEE J, WONG K, et al . Automatic micro-expression recognition from long video using a single spotted apex [C] // Proceedings of the Asian Conference on Computer Vision . Taipei, China: Springer, 2016: 345-360.

LIONG S T, SEE J, WONG K, et al . Automatic apex frame spotting in micro-expression database [C]// Proceedings of the 3rd IAPR Asian Conference on Pattern Recognition ( ACPR ). Kuala Lumpur: IEEE, 2015: 665-669.

KIM D H, BADDAR W J, RO Y M. Micro-expression recognition with expression-state constrained spatio-temporal feature representations [C]// Proceedings of the 24th ACM International Conference on Multimedia . Amsterdam, Netherlands: ACM, 2016: 382-386.

J LI , Y D WANG , J SEE , 等 . Micro-expression recognition based on 3D flow convolutional neural network . Pattern Analysis and Applications , 2019 . 22 ( 4 ): 1331 - 1339 . DOI: 10.1007/s10044-018-0757-5 http://doi.org/10.1007/s10044-018-0757-5 .

KHOR H Q, SEE J, PHAN R C W, et al . Enriched long-term recurrent convolutional network for facial micro-expression recognition [C]// Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition ( FG 2018 ). Xi'an, China: IEEE, 2018.

LEI L, LI J F, CHEN T, et al . A novel graph-TCN with a graph structured representation for micro-expression recognition [C]// Proceedings of the 28th ACM International Conference on Multimedia . Seattle: ACM, 2020.

LIU J M, LI K, SONG B L, et al . A multi-stream convolutional neural network for micro-expression recognition using optical flow and EVM [J]. arXiv: 2011.03756, 2020.

BULAT A, TZIMIROPOULOS G. How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230, 000 3D facial landmarks) [C]// Proceedings of 2017 IEEE International Conference on Computer Vision . Venice: IEEE, 2017: 1021-1030.

NEWELL A, YANG K Y, DENG J. Stacked hourglass networks for human pose estimation [C]// Proceedings of the 14th European Conference on Computer Vision . Amsterdam: Springer, 2016.

ZHU X X, RAMANAN D. Face detection, pose estimation, and landmark localization in the wild [C]// Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition . Providence: IEEE, 2012.

Y H WU , M RUBINSTEIN , E SHIH , 等 . Eulerian video magnification for revealing subtle changes in the world . ACM Transactions on Graphics , 2012 . 31 ( 4 ): 65 .

S T LIONG , J SEE , K WONG , 等 . Less is more: micro-expression recognition from video using apex frame . Signal Processing: Image Communication , 2018 . 62 82 - 92 . DOI: 10.1016/j.image.2017.11.006 http://doi.org/10.1016/j.image.2017.11.006 .

ILG E, MAYER N, SAIKIA T, et al . FlowNet 2.0: evolution of optical flow estimation with deep networks [C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu: IEEE, 2017.

GODAVARTHY S. Microexpression spotting in video using optical strain [D]. Florida: University of South Florida, 2010.

HE K M, ZHANG X Y, REN S Q, et al . Deep residual learning for image recognition [C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas: IEEE, 2016: 770-778.

HE K M, ZHANG X Y, REN S Q, et al . Identity mappings in deep residual networks [C]// Proceedings of the 14th European Conference on Computer Vision . Amsterdam: Springer, 2016.

WOO S, PARK J, LEE J Y, et al . CBAM: convolutional block attention module [C]// Proceedings of the 15th European Conference on Computer Vision . Munich: Springer, 2018: 3-19.

ZAGORUYKO S, KOMODAKIS N. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer [C]// Proceedings of the 5th International Conference on Learning Representations . Toulon: OpenReview. net, 2017.

PENG M, WU Z, ZHANG Z H, et al . From macro to micro expression recognition: deep learning on small datasets using transfer learning [C]// Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition . Xi′an, China: IEEE, 2018: 657-661.

S J WANG , B J LI , Y J LIU , 等 . Micro-expression recognition with small sample size by transferring long-term convolutional neural network . Neurocomputing , 2018 . 312 251 - 262 . DOI: 10.1016/j.neucom.2018.05.107 http://doi.org/10.1016/j.neucom.2018.05.107 .

KHOR H Q, SEE J, LIONG S T, et al . Dual-stream shallow networks for facial micro-expression recognition [C]// Proceedings of 2019 IEEE International Conference on Image Processing ( ICIP ). Taipei, China: IEEE, 2019: 36-40.

G ZHAO , M PIETIKALNEN . Dynamic texture recognition using local binary patterns with an application to facial expressions . TEEE Trcms. Pattern Anal. Mach. Intell. , 2007 . 29 ( 6 ): 915 - 928 . DOI: 10.1109/TPAMI.2007.1110 http://doi.org/10.1109/TPAMI.2007.1110 .

F XU , J P ZHANG , J Z WANG . Micro expression identification and categorization using a facial dynamics map . IEEE Transactions on Affective Computing , 2017 . 8 ( 2 ): 254 - 267 . DOI: 10.1109/TAFFC.2016.2518162 http://doi.org/10.1109/TAFFC.2016.2518162 .

WANG Y, SEE J, PHAN R C, et al. LBP with six intersection points: Reducing redundant information in LBP-TOP for micro-expression recog nition [C]// Proc. Asian Conf. Comput. Vis. (Beijing: IEEE), 2014: 525-537.

TAKALKAR M A, XU M. Image based facial micro-expression recognition using deep learning on small datasets [C]// Proceedings of 2017 International Conference on Digital Image Computing : Techniques and Applications ( DICTA ). Sydney: IEEE, 2017: 1-7.

SELVARAJU R R, COGSWELL M, DAS A, et al . Grad-CAM: visual explanations from deep networks via gradient-based localization [C]// Proceedings of 2017 IEEE International Conference on Computer Vision . Venice: IEEE, 2017: 618-626.

P EKMAN , W V FRIESEN . Facial Action Coding System: Investigator's Guide , : Palo Alto Consulting Psychologists Press , 1978 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Optical flow estimation via fusing sequence image intensity correlation information

Multi-view correction scheme for naked eye 3D display

Related Author

ZHU Ming

LI Zhaofeng

CAI Peng

ZHANG Jia-bao

JIA Di

AN Tong

Xun-bo YU

Xin GAO

Related Institution

Changchun Institute of Optics， Fine Mechanics and Physics， Chinese Academy of Sciences

College of Electronic and Information Engineering， Liaoning Technical University

College of Electrical and Control Engineering， Liaoning Technical University

School of Electronic Engineering， Beijing University of Posts and Telecommunications

AI问答

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China 130033 Postal code：130033
Tel：0431-86176059 Email：yjxs@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰