Face image repair network based on face structure guidance

SHI Hao-de; CHEN Ming-ju; HOU Jin; LI Lan

doi:10.37188/CJLCD.2022-0181

您当前的位置：

首页 >

文章列表页 >

Face image repair network based on face structure guidance

Image Processing | 更新时间：2023-02-10

- Face image repair network based on face structure guidance
- Chinese Journal of Liquid Crystals and Displays Vol. 38, Issue 2, Pages: 245-255(2023)
- 作者机构：
  
  1.四川轻化工大学自动化与信息工程学院，四川宜宾 644000
  2.四川轻化工大学人工智能四川省重点实验室，四川宜宾 644000
- 作者简介：
- 基金信息：
  
  the Sichuan Science and Technology Project(2021YFG0055);Artificial Intelligence Key Laboratory of Sichuan Province Project(2021RYY04)
- DOI：10.37188/CJLCD.2022-0181
  CLC：
扫描看全文
SHI Hao-de, CHEN Ming-ju, HOU Jin, et al. Face image repair network based on face structure guidance. [J]. Chinese Journal of Liquid Crystals and Displays 38(2):245-255(2023)
DOI：

SHI Hao-de, CHEN Ming-ju, HOU Jin, et al. Face image repair network based on face structure guidance. [J]. Chinese Journal of Liquid Crystals and Displays 38(2):245-255(2023) DOI： 10.37188/CJLCD.2022-0181.

摘要

针对人脸图像修复的深度学习网络存在修复后的人脸图像面部语义信息不合理和面部轮廓不协调的问题，提出了一种基于人脸结构信息引导的人脸图像修复网络。首先，采用编码器-解码器网络技术构建人脸结构草图生成网络，并在结构草图生成网络的生成器中加入跳跃连接和引入带膨胀卷积的残差块以生成待修复区域的结构草图。其次，在构建人脸修复网络时，在修复网络生成器中引入注意力机制，让修复网络在修复过程中更多关注待修复区域，并以生成的人脸结构草图为引导从而实现人脸图像面部语义结构和纹理信息的生动修复。最后，在结构草图生成网络的损失函数中引入特征匹配损失进行模型训练，从而约束生成器生成与真实结构草图更相似的结果；在修复网络的损失函数中联合感知损失和风格损失进行模型训练，从而更好地重建待修复区域的人脸图像面部轮廓结构和颜色纹理，使修复后的图像更接近真实图像。对比实验结果表明，在人脸图像数据集中，本文所设计的网络模型的修复性能有较高的提升。

Abstract

Aiming at the problems of unreasonable facial semantic information and inconsistency of facial contours in the restored face image in the deep learning network for face image inpainting， a face image inpainting network guided by face structure information is proposed. Firstly， the encoder-decoder network technology is used to build a face structure sketch generation network， and skip connections and residual blocks with dilated convolution are added to the generator of the structure sketch generation network to generate the structure sketch of the region to be repaired. Secondly， when a face inpainting network is builted， an attention mechanism is introduced into the inpainting network generator， so that the inpainting network pays more attention to the area to be repaired during the inpainting process， and uses the generated face structure sketch as a guide to realize the face image vivid inpainting of facial semantic structure and texture information. Finally， the feature matching loss is introduced into the loss function of the structure sketch generation network for the model training， so as to constrain the generator to generate results more similar to the real structure sketch. In the loss function of the repair network， the perceptual loss and style loss are combined for the model training， therefore， the facial contour structure and color texture of the face image in the area to be repaired can be better reconstructed， so that the repaired image is closer to the real image. The comparative experimental results show that in the face image dataset， the repair performance of the network model designed in this paper has a high improvement.

关键词

人脸修复解码器-编码器膨胀卷积跳跃连接注意力机制

Keywords

face inpatingencoder-decoderdilated convolutionskip connectionsattention mechanism

references

赵露露，沈玲，洪日昌. 图像修复研究进展综述［J］. 计算机科学，2021，48（3）：14-26. doi: 10.11896/jsjkx.210100048http://dx.doi.org/10.11896/jsjkx.210100048

ZHAO L L， SHEN L， HONG R C. Survey on image inpainting research progress ［J］. Computer Science， 2021， 48（3）： 14-26. （in Chinese）. doi: 10.11896/jsjkx.210100048http://dx.doi.org/10.11896/jsjkx.210100048

YAMAUCHI H， HABER J， SEIDEL H P. Image restoration using multiresolution texture synthesis and image inpainting ［C］//Proceedings Computer Graphics International 2003. Tokyo： IEEE， 2003： 120-125.

BARNES C， SHECHTMAN E， FINKELSTEIN A， et al. PatchMatch： a randomized correspondence algorithm for structural image editing ［J］. ACM Transactions on Graphics， 2009， 28（3）： 24. doi: 10.1145/1531326.1531330http://dx.doi.org/10.1145/1531326.1531330

GOODFELLOW I J， POUGET-ABADIE J， MIRZA M， et al. Generative adversarial nets ［C］//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal： MIT Press， 2014： 2672-2680.

PATHAK D， KRÄHENBÜHL P， DONAHUE J， et al. Context encoders： feature learning by inpainting ［C］//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas： IEEE， 2016： 2536-2544. doi: 10.1109/cvpr.2016.278http://dx.doi.org/10.1109/cvpr.2016.278

IIZUKA S， SIMO-SERRA E， ISHIKAWA H. Globally and locally consistent image completion ［J］. ACM Transactions on Graphics， 2017， 36（4）： 107. doi: 10.1145/3072959.3073659http://dx.doi.org/10.1145/3072959.3073659

YU J H， LIN Z， YANG J M， et al. Generative image inpainting with contextual attention ［C］//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City： IEEE， 2018： 5505-5514. doi: 10.1109/cvpr.2018.00577http://dx.doi.org/10.1109/cvpr.2018.00577

NAZERI K， NG E， JOSEPH T， et al. EdgeConnect： generative image inpainting with adversarial edge learning ［J/OL］. arXiv， 2019： 1901.00212. doi: 10.1109/iccvw.2019.00408http://dx.doi.org/10.1109/iccvw.2019.00408

YANG Y， GUO X J， MA J Y， et al. LaFIn： generative landmark guided face inpainting ［J/OL］. arXiv， 2019： 1911.11394. doi: 10.1007/978-3-030-60633-6_2http://dx.doi.org/10.1007/978-3-030-60633-6_2

XIONG W， YU J H， LIN Z， et al. Foreground-aware image inpainting ［C］//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach： IEEE， 2019： 5840-5848. doi: 10.1109/cvpr.2019.00599http://dx.doi.org/10.1109/cvpr.2019.00599

YANG J， QI Z Q， SHI Y. Learning to incorporate structure knowledge for image inpainting ［C］//Proceedings of the 34th AAAI Conference on Artificial Intelligence. New York： AAAI Press， 2020： 12605-12612. doi: 10.1609/aaai.v34i07.6951http://dx.doi.org/10.1609/aaai.v34i07.6951

LIAO L， XIAO J， WANG Z， et al. Guidance and evaluation： semantic-aware image inpainting for mixed scenes ［C］//Proceedings of the 16th European Conference on Computer Vision. Glasgow： Springer， 2020： 683-700. doi: 10.1007/978-3-030-58583-9_41http://dx.doi.org/10.1007/978-3-030-58583-9_41

GUO X F， YANG H Y， HUANG D. Image inpainting via conditional texture and structure dual generation ［C］//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal： IEEE， 2021： 14134-14143. doi: 10.1109/iccv48922.2021.01387http://dx.doi.org/10.1109/iccv48922.2021.01387

RONNEBERGER O， FISCHER P， BROX T. U-Net： convolutional networks for biomedical image segmentation ［C］//Proceedings of the 18th International Conference on Medical Image Computing and Computer-assisted Intervention. Munich： Springer， 2015： 234-241. doi: 10.1007/978-3-319-24574-4_28http://dx.doi.org/10.1007/978-3-319-24574-4_28

YU F， KOLTUN V. Multi-scale context aggregation by dilated convolutions ［C］//Proceedings of the 4th International Conference on Learning Representations. San Juan： IEEE， 2016. doi: 10.1109/cvpr.2017.75http://dx.doi.org/10.1109/cvpr.2017.75

ZHENG C X， CHAM T J， CAI J F. Pluralistic image completion ［C］//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach： IEEE， 2019： 1438-1447. doi: 10.1109/cvpr.2019.00153http://dx.doi.org/10.1109/cvpr.2019.00153

ISOLA P， ZHU J Y， ZHOU T H， et al. Image-to-image translation with conditional adversarial networks ［C］//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu： IEEE， 2017： 1125-1134. doi: 10.1109/cvpr.2017.632http://dx.doi.org/10.1109/cvpr.2017.632

MIYATO T， KATAOKA T， KOYAMA M， et al. Spectral normalization for generative adversarial networks ［C］//Proceedings of the 6th International Conference on Learning Representations. Vancouver： OpenReview. net， 2018.

JOHNSON J， ALAHI A， FEI-FEI L. Perceptual losses for real-time style transfer and super-resolution ［C］//Proceedings of the 14th European Conference on Computer Vision. Amsterdam： Springer， 2016： 694-711. doi: 10.1007/978-3-319-46475-6_43http://dx.doi.org/10.1007/978-3-319-46475-6_43

GATYS L A， ECKER A S， BETHGE M. A neural algorithm of artistic style ［J/OL］. arXiv， 2015： 1508.06576. doi: 10.1167/16.12.326http://dx.doi.org/10.1167/16.12.326

SIMONYAN K， ZISSERMAN A. Very deep convolutional networks for large-scale image recognition ［C］//Proceedings of the 3rd International Conference on Learning Representations. San Diego， 2015.

KARRAS T， AILA T， LAINE S， et al. Progressive growing of GANs for improved quality， stability， and variation ［C］//Proceedings of the 6th International Conference on Learning Representations. Vancouver： OpenReview.net， 2018.

LIU G L， REDA F A， SHIH K J， et al. Image inpainting for irregular holes using partial convolutions ［C］//Proceedings of the 15th European Conference on Computer Vision. Munich： Springer， 2018： 85-100. doi: 10.1007/978-3-030-01252-6_6http://dx.doi.org/10.1007/978-3-030-01252-6_6

GUO J T， LIU Y. Image completion using structure and texture GAN network ［J］. Neurocomputing， 2019， 360： 75-84. doi: 10.1016/j.neucom.2019.06.010http://dx.doi.org/10.1016/j.neucom.2019.06.010

WANG Z， BOVIK A C， SHEIKH H R， et al. Image quality assessment： from error visibility to structural similarity ［J］. IEEE Transactions on Image Processing， 2004， 13（4）： 600-612. doi: 10.1109/tip.2003.819861http://dx.doi.org/10.1109/tip.2003.819861

DOWSON D C， LANDAU B V. The Fréchet distance between multivariate normal distributions ［J］. Journal of Multivariate Analysis， 1982， 12（3）： 450-455. doi: 10.1016/0047-259x(82)90077-xhttp://dx.doi.org/10.1016/0047-259x(82)90077-x

罗仕胜，陈明举，陈柳，等. 基于面部特征点的人脸图像修复网络［J］. 中国科技论文，2021，16（7）：729-734，742. doi: 10.3969/j.issn.2095-2783.2021.07.007http://dx.doi.org/10.3969/j.issn.2095-2783.2021.07.007

LUO S S， CHEN M J， CHEN L， et al. Face image inpainting network based on generative facial landmark ［J］. China Sciencepaper， 2021， 16（7）： 729-734， 742. （in Chinese）. doi: 10.3969/j.issn.2095-2783.2021.07.007http://dx.doi.org/10.3969/j.issn.2095-2783.2021.07.007

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Facial expression recognition based on attention mechanism ResNet lightweight network

Infrared dim-small target detection under complex background based on attention mechanism

Underwater image enhancement method combining feature fusion and physical correction

Reservoir computing based network for few-shot image classification

Related Author

No data

Related Institution

School of Electronic Information and Artificial Intelligence， Shaanxi University of Science & Technology

University of Chinese Academy of Sciences

Changchun Institute of Optics， Fine Mechanics and Physics， Chinese Academy of Sciences

School of Information， Shanghai Ocean University

Fujian Science & Technology Innovation Laboratory for Optoelectronic Information of China （Mindu Innovation Laboratory）

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China 130033 Postal code：130033
Tel：0431-86176059 Email：yjxs@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰