Face image repair network based on face structure guidance

SHI Hao-de; CHEN Ming-ju; HOU Jin; LI Lan

doi:10.37188/CJLCD.2022-0181

您当前的位置：

首页 >

文章列表页 >

Face image repair network based on face structure guidance

Image Processing | 更新时间：2023-02-10

- Face image repair network based on face structure guidance
- Chinese Journal of Liquid Crystals and Displays Vol. 38, Issue 2, Pages: 245-255(2023)
- 作者机构：
  
  1.四川轻化工大学自动化与信息工程学院，四川宜宾 644000
  2.四川轻化工大学人工智能四川省重点实验室，四川宜宾 644000
- 作者简介：
- 基金信息：
  
  the Sichuan Science and Technology Project(2021YFG0055);Artificial Intelligence Key Laboratory of Sichuan Province Project(2021RYY04)
- DOI：10.37188/CJLCD.2022-0181
  CLC： TP391.4
- Received：28 May 2022，
  
  Revised：20 June 2022，
  
  Published：05 February 2023
- 稿件说明：
移动端阅览
SHI Hao-de, CHEN Ming-ju, HOU Jin, et al. Face image repair network based on face structure guidance[J]. Chinese journal of liquid crystals and displays, 2023, 38(2): 245-255.
DOI：

SHI Hao-de, CHEN Ming-ju, HOU Jin, et al. Face image repair network based on face structure guidance[J]. Chinese journal of liquid crystals and displays, 2023, 38(2): 245-255. DOI： 10.37188/CJLCD.2022-0181.

摘要

针对人脸图像修复的深度学习网络存在修复后的人脸图像面部语义信息不合理和面部轮廓不协调的问题，提出了一种基于人脸结构信息引导的人脸图像修复网络。首先，采用编码器-解码器网络技术构建人脸结构草图生成网络，并在结构草图生成网络的生成器中加入跳跃连接和引入带膨胀卷积的残差块以生成待修复区域的结构草图。其次，在构建人脸修复网络时，在修复网络生成器中引入注意力机制，让修复网络在修复过程中更多关注待修复区域，并以生成的人脸结构草图为引导从而实现人脸图像面部语义结构和纹理信息的生动修复。最后，在结构草图生成网络的损失函数中引入特征匹配损失进行模型训练，从而约束生成器生成与真实结构草图更相似的结果；在修复网络的损失函数中联合感知损失和风格损失进行模型训练，从而更好地重建待修复区域的人脸图像面部轮廓结构和颜色纹理，使修复后的图像更接近真实图像。对比实验结果表明，在人脸图像数据集中，本文所设计的网络模型的修复性能有较高的提升。

Abstract

Aiming at the problems of unreasonable facial semantic information and inconsistency of facial contours in the restored face image in the deep learning network for face image inpainting， a face image inpainting network guided by face structure information is proposed. Firstly， the encoder-decoder network technology is used to build a face structure sketch generation network， and skip connections and residual blocks with dilated convolution are added to the generator of the structure sketch generation network to generate the structure sketch of the region to be repaired. Secondly， when a face inpainting network is builted， an attention mechanism is introduced into the inpainting network generator， so that the inpainting network pays more attention to the area to be repaired during the inpainting process， and uses the generated face structure sketch as a guide to realize the face image vivid inpainting of facial semantic structure and texture information. Finally， the feature matching loss is introduced into the loss function of the structure sketch generation network for the model training， so as to constrain the generator to generate results more similar to the real structure sketch. In the loss function of the repair network， the perceptual loss and style loss are combined for the model training， therefore， the facial contour structure and color texture of the face image in the area to be repaired can be better reconstructed， so that the repaired image is closer to the real image. The comparative experimental results show that in the face image dataset， the repair performance of the network model designed in this paper has a high improvement.

关键词

Keywords

references

赵露露，沈玲，洪日昌 . 图像修复研究进展综述［J］. 计算机科学， 2021 ， 48 （ 3 ）： 14 - 26 . doi: 10.11896/jsjkx.210100048 http://dx.doi.org/10.11896/jsjkx.210100048

ZHAO L L ， SHEN L ， HONG R C . Survey on image inpainting research progress ［J］. Computer Science ， 2021 ， 48 （ 3 ）： 14 - 26 . （in Chinese） . doi: 10.11896/jsjkx.210100048 http://dx.doi.org/10.11896/jsjkx.210100048

YAMAUCHI H ， HABER J ， SEIDEL H P . Image restoration using multiresolution texture synthesis and image inpainting ［C］// Proceedings Computer Graphics International 2003 . Tokyo ： IEEE ， 2003 ： 120 - 125 .

BARNES C ， SHECHTMAN E ， FINKELSTEIN A ， et al . PatchMatch： a randomized correspondence algorithm for structural image editing ［J］. ACM Transactions on Graphics ， 2009 ， 28 （ 3 ）： 24 . doi: 10.1145/1531326.1531330 http://dx.doi.org/10.1145/1531326.1531330

GOODFELLOW I J ， POUGET-ABADIE J ， MIRZA M ， et al . Generative adversarial nets ［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems . Montreal ： MIT Press ， 2014 ： 2672 - 2680 .

PATHAK D ， KRÄHENBÜHL P ， DONAHUE J ， et al . Context encoders： feature learning by inpainting ［C］// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas ： IEEE ， 2016 ： 2536 - 2544 . doi: 10.1109/cvpr.2016.278 http://dx.doi.org/10.1109/cvpr.2016.278

IIZUKA S ， SIMO-SERRA E ， ISHIKAWA H . Globally and locally consistent image completion ［J］. ACM Transactions on Graphics ， 2017 ， 36 （ 4 ）： 107 . doi: 10.1145/3072959.3073659 http://dx.doi.org/10.1145/3072959.3073659

YU J H ， LIN Z ， YANG J M ， et al . Generative image inpainting with contextual attention ［C］// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City ： IEEE ， 2018 ： 5505 - 5514 . doi: 10.1109/cvpr.2018.00577 http://dx.doi.org/10.1109/cvpr.2018.00577

NAZERI K ， NG E ， JOSEPH T ， et al . EdgeConnect： generative image inpainting with adversarial edge learning ［J/OL］. arXiv ， 2019 ： 1901 . 00212 . doi: 10.1109/iccvw.2019.00408 http://dx.doi.org/10.1109/iccvw.2019.00408

YANG Y ， GUO X J ， MA J Y ， et al . LaF In： generative landmark guided face inpainting ［J/OL］. arXiv ， 2019 ： 1911 . 11394 . doi: 10.1007/978-3-030-60633-6_2 http://dx.doi.org/10.1007/978-3-030-60633-6_2

XIONG W ， YU J H ， LIN Z ， et al . Foreground-aware image inpainting ［C］// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach ： IEEE ， 2019 ： 5840 - 5848 . doi: 10.1109/cvpr.2019.00599 http://dx.doi.org/10.1109/cvpr.2019.00599

YANG J ， QI Z Q ， SHI Y . Learning to incorporate structure knowledge for image inpainting ［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence . New York ： AAAI Press ， 2020 ： 12605 - 12612 . doi: 10.1609/aaai.v34i07.6951 http://dx.doi.org/10.1609/aaai.v34i07.6951

LIAO L ， XIAO J ， WANG Z ， et al . Guidance and evaluation： semantic-aware image inpainting for mixed scenes ［C］// Proceedings of the 16th European Conference on Computer Vision . Glasgow ： Springer ， 2020 ： 683 - 700 . doi: 10.1007/978-3-030-58583-9_41 http://dx.doi.org/10.1007/978-3-030-58583-9_41

GUO X F ， YANG H Y ， HUANG D . Image inpainting via conditional texture and structure dual generation ［C］// Proceedings of 2021 IEEE/CVF International Conference on Computer Vision . Montreal ： IEEE ， 2021 ： 14134 - 14143 . doi: 10.1109/iccv48922.2021.01387 http://dx.doi.org/10.1109/iccv48922.2021.01387

RONNEBERGER O ， FISCHER P ， BROX T . U-Net： convolutional networks for biomedical image segmentation ［C］// Proceedings of the 18th International Conference on Medical Image Computing and Computer-assisted Intervention . Munich ： Springer ， 2015 ： 234 - 241 . doi: 10.1007/978-3-319-24574-4_28 http://dx.doi.org/10.1007/978-3-319-24574-4_28

YU F ， KOLTUN V . Multi-scale context aggregation by dilated convolutions ［C］// Proceedings of the 4th International Conference on Learning Representations . San Juan ： IEEE ， 2016 . doi: 10.1109/cvpr.2017.75 http://dx.doi.org/10.1109/cvpr.2017.75

ZHENG C X ， CHAM T J ， CAI J F . Pluralistic image completion ［C］// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach ： IEEE ， 2019 ： 1438 - 1447 . doi: 10.1109/cvpr.2019.00153 http://dx.doi.org/10.1109/cvpr.2019.00153

ISOLA P ， ZHU J Y ， ZHOU T H ， et al . Image-to-image translation with conditional adversarial networks ［C］// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu ： IEEE ， 2017 ： 1125 - 1134 . doi: 10.1109/cvpr.2017.632 http://dx.doi.org/10.1109/cvpr.2017.632

MIYATO T ， KATAOKA T ， KOYAMA M ， et al . Spectral normalization for generative adversarial networks ［C］// Proceedings of the 6th International Conference on Learning Representations . Vancouver ： OpenReview. net ， 2018 .

JOHNSON J ， ALAHI A ， FEI-FEI L . Perceptual losses for real-time style transfer and super-resolution ［C］// Proceedings of the 14th European Conference on Computer Vision . Amsterdam ： Springer ， 2016 ： 694 - 711 . doi: 10.1007/978-3-319-46475-6_43 http://dx.doi.org/10.1007/978-3-319-46475-6_43

GATYS L A ， ECKER A S ， BETHGE M . A neural algorithm of artistic style ［J/OL］. arXiv ， 2015 ： 1508 . 06576 . doi: 10.1167/16.12.326 http://dx.doi.org/10.1167/16.12.326

SIMONYAN K ， ZISSERMAN A . Very deep convolutional networks for large-scale image recognition ［C］// Proceedings of the 3rd International Conference on Learning Representations . San Diego ， 2015 .

KARRAS T ， AILA T ， LAINE S ， et al . Progressive growing of GANs for improved quality， stability， and variation ［C］// Proceedings of the 6th International Conference on Learning Representations . Vancouver ： OpenReview.net ， 2018 .

LIU G L ， REDA F A ， SHIH K J ， et al . Image inpainting for irregular holes using partial convolutions ［C］// Proceedings of the 15th European Conference on Computer Vision . Munich ： Springer ， 2018 ： 85 - 100 . doi: 10.1007/978-3-030-01252-6_6 http://dx.doi.org/10.1007/978-3-030-01252-6_6

GUO J T ， LIU Y . Image completion using structure and texture GAN network ［J］. Neurocomputing ， 2019 ， 360 ： 75 - 84 . doi: 10.1016/j.neucom.2019.06.010 http://dx.doi.org/10.1016/j.neucom.2019.06.010

WANG Z ， BOVIK A C ， SHEIKH H R ， et al . Image quality assessment： from error visibility to structural similarity ［J］. IEEE Transactions on Image Processing ， 2004 ， 13 （ 4 ）： 600 - 612 . doi: 10.1109/tip.2003.819861 http://dx.doi.org/10.1109/tip.2003.819861

DOWSON D C ， LANDAU B V . The Fréchet distance between multivariate normal distributions ［J］. Journal of Multivariate Analysis ， 1982 ， 12 （ 3 ）： 450 - 455 . doi: 10.1016/0047-259x(82)90077-x http://dx.doi.org/10.1016/0047-259x(82)90077-x

罗仕胜，陈明举，陈柳，等 . 基于面部特征点的人脸图像修复网络［J］. 中国科技论文， 2021 ， 16 （ 7 ）： 729 - 734，742 . doi: 10.3969/j.issn.2095-2783.2021.07.007 http://dx.doi.org/10.3969/j.issn.2095-2783.2021.07.007

LUO S S ， CHEN M J ， CHEN L ， et al . Face image inpainting network based on generative facial landmark ［J］. China Sciencepaper ， 2021 ， 16 （ 7 ）： 729 - 734， 742 . （in Chinese） . doi: 10.3969/j.issn.2095-2783.2021.07.007 http://dx.doi.org/10.3969/j.issn.2095-2783.2021.07.007

Views

144

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Multi-scale pedestrian detection algorithm based on joint head and overall information

3D hand reconstruction method based on adaptive occlusion recovery and topology-pose bidirectional perception

Fatigue driving detection based on improved YOLOv8n-Pose

Neural architecture search combined with efficient attention for hyperspectral image classification

Related Author

SHI Hao-de

CHENG Ming-ju

HOU Jin

LI Lan

MA Ximing

LI Ning

WU Di

WANG Jianfei

Related Institution

Changchun Institute of Optics， Fine Mechanics and Physics， Chinese Academy of Sciences

Military Representative Office Stationed in a Region by Air Force Equipment Department

School of Automation， Nanjing University of Information Science & Technology

School of Computer and Computing Science， Hangzhou City University

School of Advanced Manufacturing， Fuzhou University

AI问答

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China 130033 Postal code：130033
Tel：0431-86176059 Email：yjxs@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰