3D hand pose estimation algorithm based on cascaded features and graph convolution

Yi-lin LIN; Shan-ling LIN; Zhi-xian LIN

doi:10.37188/CJLCD.2021-0307

您当前的位置：

首页 >

文章列表页 >

3D hand pose estimation algorithm based on cascaded features and graph convolution

Image Processing | 更新时间：2022-06-06

- 3D hand pose estimation algorithm based on cascaded features and graph convolution
- Chinese Journal of Liquid Crystals and Displays Vol. 37, Issue 6, Pages: 736-745(2022)
- 作者机构：
  
  1.福州大学物理与信息工程学院，福建福州 350116
  2.中国福建光电信息科学与技术创新实验室，福建福州 350116
  3.福州大学先进制造学院，福建泉州 362200
- 作者简介：
- 基金信息：
  
  National Key R&D Program of China(2021YFB3600603);Projects supported by Natural Science Foundation of Fujian Province(2020J01468)
- DOI：10.37188/CJLCD.2021-0307
  CLC： TP391.4
- Received：29 November 2021，
  
  Revised：27 December 2021，
  
  Published：05 June 2022
- 稿件说明：
移动端阅览
Yi-lin LIN, Shan-ling LIN, Zhi-xian LIN. 3D hand pose estimation algorithm based on cascaded features and graph convolution[J]. Chinese journal of liquid crystals and displays, 2022, 37(6): 736-745.
DOI：

Yi-lin LIN, Shan-ling LIN, Zhi-xian LIN. 3D hand pose estimation algorithm based on cascaded features and graph convolution[J]. Chinese journal of liquid crystals and displays, 2022, 37(6): 736-745. DOI： 10.37188/CJLCD.2021-0307.

摘要

针对手部的高自由度问题和结构相似问题引起的三维关键点姿态估计误差，本文提出了一套联合识别、检测以及姿态估计的三维手部骨架姿态回归网络。采用基于YOLOv3的预处理网络，提出基于级联多特征热度图的二维和三维关键点检测网络，并在特征提取网络架构中引入人体骨架手部约束，利用渐进的图卷积神经网络特征增强模块对骨架关键点结果进行进一步精细化修正，完成姿态由粗到细的调整。本文与现有多种算法在不同公共数据集下进行PCK指标和AUC指标比较，本文算法在不同测试集上的AUC指标均达到最高，平均AUC精度达到92.9%。实验表明本文方法可以通过单张二维数据准确、细致地估计三维手部姿态，并且在测试集与自然场景下均有较好表现。

Abstract

For the 3D key point pose estimation error caused by the high degree of freedom problem and structural similarity problem of the hand， this paper proposes a novel 3D hand skeleton pose regression framework for joint identification， detection， and pose estimation. The framework firstly adopts a YOLOv3-based detector to obtain the position of hands， then a cascade pose estimation network is designed to get initial hand poses with 2D and 3D pose supervisions. Finally， considering the natural constrains in hand graph connection， we present progressive GCN module to further refine the initial hand pose from coarse to fine. This paper compares PCK metrics and AUC metrics with the state-of-the-art approaches under different public benchmarks， and the proposed method achieves the highest AUC metrics on different test sets， with an average AUC accuracy of 92.9%. The experiments illustrate that the proposed method is able to effectively and robustly predict 3D hand pose from monocular image， performing well in both test sets and in the wild.

关键词

Keywords

references

CAO Z ， SIMON T ， WEI S E ， et al . Realtime multi-person 2D pose estimation using part affinity fields ［C］// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu ： IEEE ， 2017 ： 1302 - 1310 . doi: 10.1109/cvpr.2017.143 http://dx.doi.org/10.1109/cvpr.2017.143

FANG H S ， XIE S Q ， TAI Y W ， et al . RMPE： regional multi-person pose estimation ［C］// Proceedings of 2017 IEEE International Conference on Computer Vision . Venice ： IEEE ， 2017 ： 2353 - 2362 . doi: 10.1109/iccv.2017.256 http://dx.doi.org/10.1109/iccv.2017.256

CAI Y J ， GE L H ， CAI J F ， et al . Weakly-supervised 3D hand pose estimation from monocular RGB images ［C］// Proceedings of the 15th European Conference on Computer Vision . Munich ： Springer ， 2018 ： 678 - 694 . doi: 10.1007/978-3-030-01231-1_41 http://dx.doi.org/10.1007/978-3-030-01231-1_41

GE L H ， REN Z ， LI Y C ， et al . 3D hand shape and pose estimation from a single RGB image ［C］// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach ： IEEE ， 2019 ： 10825 - 10834 . doi: 10.1109/cvpr.2019.01109 http://dx.doi.org/10.1109/cvpr.2019.01109

ZHOU Y X ， HABERMANN M ， XU W P ， et al . Monocular real-time hand shape and motion capture using multi-modal data ［C］// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle ： IEEE ， 2020 ： 5345 - 5354 . doi: 10.1109/cvpr42600.2020.00539 http://dx.doi.org/10.1109/cvpr42600.2020.00539

KIPF T N ， WELLING M . Semi-supervised classification with graph convolutional networks ［C］// Proceedings of the 5th International Conference on Learning Representations . Toulon ： OpenReview.net ， 2017 .

贠卫国，史其琦，王民 . 基于深度卷积神经网络的多特征融合的手势识别［J］. 液晶与显示， 2019 ， 34 （ 4 ）： 417 - 422 ． doi: 10.3788/YJYXS20193404.0417 http://dx.doi.org/10.3788/YJYXS20193404.0417

YUN W G ， SHI Q Q ， WANG M . Multi-feature fusion gesture recognition based on deep convolutional neural network ［J］. Chinese Journal of Liquid Crystals and Displays ， 2019 ， 34 （ 4 ）： 417 - 422 . （in Chinese） . doi: 10.3788/YJYXS20193404.0417 http://dx.doi.org/10.3788/YJYXS20193404.0417

REDMON J ， FARHADI A . YOLOV3： an incremental improvement ［J］. arXiv ： 1804.02767 ， 2018 . doi: 10.1109/cvpr.2017.690 http://dx.doi.org/10.1109/cvpr.2017.690

HE K M ， ZHANG X Y ， REN S Q ， et al . Deep residual learning for image recognition ［C］// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas ： IEEE ， 2016 ： 770 - 778 . doi: 10.1109/cvpr.2016.90 http://dx.doi.org/10.1109/cvpr.2016.90

PFISTER T ， CHARLES J ， ZISSERMAN A . Flowing convnets for human pose estimation in videos ［C］// Proceedings of 2015 IEEE International Conference on Computer Vision . Santiago ： IEEE ， 2015 ： 1913 - 1921 . doi: 10.1109/iccv.2015.222 http://dx.doi.org/10.1109/iccv.2015.222

MEHTA D ， SRIDHAR S ， SOTNYCHENKO O ， et al . VNect： real-time 3D human pose estimation with a single RGB camera ［J］. ACM Transactions on Graphics ， 2017 ， 36 （ 4 ）： 44 . doi: 10.1145/3072959.3073596 http://dx.doi.org/10.1145/3072959.3073596

ZHAO L ， PENG X ， TIAN Y ， et al . Semantic graph convolutional networks for 3D human pose regression ［C］// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach ： IEEE ， 2019 ： 3420 - 3430 . doi: 10.1109/cvpr.2019.00354 http://dx.doi.org/10.1109/cvpr.2019.00354

SUN X ， XIAO B ， WEI F Y ， et al . Integral human pose regression ［C］// Proceedings of the 15th European Conference on Computer Vision . Munich ： Springer ， 2018 ： 536 - 553 . doi: 10.1007/978-3-030-01231-1_33 http://dx.doi.org/10.1007/978-3-030-01231-1_33

SIMON T ， JOO H ， MATTHEWS I ， et al . Hand keypoint detection in single images using multiview bootstrapping ［C］// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu ： IEEE ， 2017 ： 4645 - 4653 . doi: 10.1109/cvpr.2017.494 http://dx.doi.org/10.1109/cvpr.2017.494

ZIMMERMANN C ， BROX T . Learning to estimate 3D hand pose from single RGB images ［C］// Proceedings of 2017 IEEE International Conference on Computer Vision . Venice ： IEEE ， 2017 ： 4913 - 4921 . doi: 10.1109/iccv.2017.525 http://dx.doi.org/10.1109/iccv.2017.525

SRIDHAR S ， MUELLER F ， ZOLLHÖFER M ， et al . Real-time joint tracking of a hand manipulating an object from RGB-D input ［C］// Proceedings of the 14th European Conference on Computer Vision . Amsterdam ： Springer ， 2016 ： 294 - 310 . doi: 10.1007/978-3-319-46475-6_19 http://dx.doi.org/10.1007/978-3-319-46475-6_19

MUELLER F ， MEHTA D ， SOTNYCHENKO O ， et al . Real-time hand tracking under occlusion from an egocentric RGB-D sensor ［C］// Proceedings of 2017 IEEE International Conference on Computer Vision . Venice ： IEEE ， 2017 ： 1163 - 1172 . doi: 10.1109/iccvw.2017.82 http://dx.doi.org/10.1109/iccvw.2017.82

MUELLER F ， BERNARD F ， SOTNYCHENKO O ， et al . Ganerated hands for real-time 3D hand tracking from monocular RGB ［C］// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City ： IEEE ， 2018 ： 49 - 59 . doi: 10.1109/cvpr.2018.00013 http://dx.doi.org/10.1109/cvpr.2018.00013

ZHANG X ， LI Q ， MO H ， et al . End-to-end hand mesh recovery from a monocular RGB image ［C］// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision . Seoul ： IEEE ， 2019 ： 2354 - 2364 . doi: 10.1109/iccv.2019.00244 http://dx.doi.org/10.1109/iccv.2019.00244

XIANG D L ， JOO H ， SHEIKH Y . Monocular total capture： posing face， body， and hands in the wild ［C］// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach ： IEEE ， 2019 ： 10957 - 10966 . doi: 10.1109/cvpr.2019.01122 http://dx.doi.org/10.1109/cvpr.2019.01122

BOUKHAYMA A ， DE BEM R ， TORR P H S . 3D hand shape and pose from images in the wild ［C］// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach ： IEEE ， 2019 ： 10835 - 10844 . doi: 10.1109/cvpr.2019.01110 http://dx.doi.org/10.1109/cvpr.2019.01110

Views

468

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

CJLCD | 3D hand pose estimation algorithm

Improved algorithm of GDT-YOLOV 3 image target detection

Design of heterogeneous FPGA hardware accelerator based on CNN

Lightweight image super-resolution combining residual learning and layer attention

Remote sensing image change detection based on CNN-Transformer structure

Related Author

林依林

林珊玲

林志贤

Yue TANG

Ge WU

Yan PIAO

JI Haolin

XU Wei

Related Institution

School of Electronics and Information Engineering, Changchun University of Science and Technology

Changchun Institute of Optics， Fine Mechanics and Physics， Chinese Academy of Sciences

University of Chinese Academy of Sciences

Key Laboratory of Space-based Dynamic & Rapid Optical Imaging Technology， Chinese Academy of Sciences

School of Electronic Information and Artificial Intelligence， Shaanxi University of Science & Technology， Xi′an

AI问答

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China 130033 Postal code：130033
Tel：0431-86176059 Email：yjxs@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰