Adversarial grayscale image colorization combined with global semantic optimization

Yuan-yuan WAN; Yu-qing WANG; Xiao-ning ZHANG; Da-qun LI; Xiao-lin CHEN

doi:10.37188/CJLCD.2021-0012

您当前的位置：

首页 >

文章列表页 >

Adversarial grayscale image colorization combined with global semantic optimization

Image Processing | 更新时间：2021-09-10

- Adversarial grayscale image colorization combined with global semantic optimization
- Chinese Journal of Liquid Crystals and Displays Vol. 36, Issue 9, Pages: 1305-1313(2021)
- 作者机构：
  
  1.中国科学院长春光学精密机械与物理研究所, 吉林长春 130033
  2.中国科学院大学, 北京 100049
- 作者简介：
- 基金信息：
  
  Supported by National Key Research and Development Project(2018YFC0308100)
- DOI：10.37188/CJLCD.2021-0012
  CLC： TP391.4
- Received：12 January 2021，
  
  Revised：28 January 2021，
  
  Published：2021-09
- 稿件说明：
移动端阅览
Yuan-yuan WAN, Yu-qing WANG, Xiao-ning ZHANG, et al. Adversarial grayscale image colorization combined with global semantic optimization[J]. Chinese journal of liquid crystals and displays, 2021, 36(9): 1305-1313.
DOI：

Yuan-yuan WAN, Yu-qing WANG, Xiao-ning ZHANG, et al. Adversarial grayscale image colorization combined with global semantic optimization[J]. Chinese journal of liquid crystals and displays, 2021, 36(9): 1305-1313. DOI： 10.37188/CJLCD.2021-0012.

摘要

针对当前灰度图像彩色化算法容易出现色彩枯燥、颜色溢出和图像细节损失等问题，本文提出一种结合全局语义优化的对抗性灰度图像彩色化算法。其中，生成网络采用自主改进的U-Net网络。一方面，改进的U-Net网络利用多层卷积对输入图像进行逐步下采样，在获取多尺度层级特征和全局特征的同时，在跳跃连接中将全局特征和多尺度层级特征进行自适应融合，从而有效增强算法对全局语义信息的理解能力并缓解颜色溢出的现象；另一方面，改进的U-Net网络在上采样过程中融合通道注意力模块，使得在提取卷积特征时能够有效抑制噪声并降低特征冗余性。判别网络主要采用全卷积结构，通过反向传播误差以达到优化生成网络的目的。此外，本文算法的损失函数将WGAN-GP网络的优化思想和颜色损失相结合，从而解决传统生成对抗网络训练时出现的梯度消失和模式崩溃等问题。本文算法在Place365测试集上所获取的峰值信噪比、结构相似性和信息熵指标分别为24.455 dB、0.943和7.489。实验结果表明，本文算法能够缓解颜色溢出，且细节保持和颜色饱和度方面都具有一定优势。

Abstract

The current colorization algorithms generally suffer from color boring

color bleeding and detail loss. In order to address these problems

this paper proposes a adversarial colorization algorithm combined with global semantic optimization. The generator adopts the improved U-Net network. On the one hand

the improved U-Net gradually obtains multi-scale features and global feature by multiple convolution during the downsampling process. At the same time

the global feature and multi-scale features get adaptive fused in the skip connections

thus it can effectively enhance algorithm's ability to understand global semantic and ease the color bleeding. On the other hand

the improved U-Net network integrates the channel attention module in the upsampling process

which can effectively suppress the noise and reduce the feature redundancy when the convolutional features are extracted. The discriminator mainly adopts full convolution structure and achieves the purpose of optimizing the generator by reverse error transfer. In addition

the loss function combines the optimization idea of WGAN-GP with color loss to solve the gradient disappearance and mode collapse problems in the training of traditional GAN. The proposed algorithm is tested on Place365 test set

and the PSNR

SSIM and IE indexes are 24.455 dB

0.943 and 7.489

respectively. The test results show that the proposed algorithm can alleviate color bleeding

and has certain advantages in detail retention and color saturation.

关键词

Keywords

references

CHENG Z Z, YANG Q X, SHENG B. Deep colorization[C]// Proceedings of 2015 IEEE International Conference on Computer Vision . Santiago: IEEE, 2015: 415-423.

S IIZUKA , E SIMO-SERRA , H ISHIKAWA . Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification . ACM Transactions on Graphics , 2016 . 35 ( 4 ): 110 10.1145/2897824.2925974 10.1145/2897824.2925974 .

ZHANG R, ISOLA P, EFROS A A. Colorful image colorization[C]// Proceedings of the 14 th European Conference on Computer Vision . Amsterdam: Springer, 2016: 649-666.

LARSSON G, MAIRE M, SHAKHNAROVICH G. Learning representations for automatic colorization[C]// Proceedings of the 14 th European Conference on Computer Vision . Amsterdam: Springer, 2016: 577-593.

WANG X T, YU K, WU S X, et al . ESRGAN: enhanced super-resolution generative adversarial networks[C]// Proceedings of European Conference on Computer Vision . Munich: Springer, 2018: 63-79.

余思泉 , 韩志 , 唐延东 , 等 . 基于对抗生成网络的纹理合成方法 . 红外与激光工程 , 2018 . 47 ( 2 ): 0203005 10.3788/IRLA201847.0203005 10.3788/IRLA201847.0203005 .

S Q YU , Z HAN , Y D TANG , 等 . Texture synthesis method based on generative adversarial networks . Infrared and Laser Engineering , 2018 . 47 ( 2 ): 0203005 10.3788/IRLA201847.0203005 10.3788/IRLA201847.0203005 .

杜振龙 , 沈海洋 , 宋国美 , 等 . 基于改进CycleGAN的图像风格迁移 . 光学精密工程 , 2019 . 27 ( 8 ): 1836 - 1844 . DOI: 10.3788/OPE.20192708.1836 http://doi.org/10.3788/OPE.20192708.1836 .

Z L DU , H Y SHEN , G M SONG , 等 . Image style transfer based on improved CycleGAN . Optics and Precision Engineering , 2019 . 27 ( 8 ): 1836 - 1844 . DOI: 10.3788/OPE.20192708.1836 http://doi.org/10.3788/OPE.20192708.1836 .

ISOLA P, ZHU J Y, ZHOU T H, et al . Image-to-image translation with conditional adversarial networks[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu: IEEE, 2017: 5967-5976.

NAZERI K, NG E, EBRAHIMI M. Image colorization with generative adversarial networks[J]. arXiv: 1803.05400, 2018.

CAO Y, ZHOU Z M, ZHANG W N, et al . Unsupervised diverse colorization via generative adversarial networks[C]// Proceedings of Joint European Conference on Machine Learning and Knowledge Discovery in Databases . Skopje: Springer, 2017: 151-166.

VITORIA P, RAAD L, BALLESTER C. ChromaGAN: adversarial picture colorization with semantic classdistribution[C]// Proceedings of 2020 IEEE Winter Conference on Applications of Computer Vision . Snowmass: IEEE, 2020: 2434-2443.

S H ZHANG , H LI , W H KONG . Object counting method based on dual attention network . IET Image Processing , 2020 . 14 ( 8 ): 1621 - 1627 . DOI: 10.1049/iet-ipr.2019.0465 http://doi.org/10.1049/iet-ipr.2019.0465 .

ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein generative adversarial networks[C]// Proceedings of the 34 th International Conference on Machine Learning . Sydney: PMLR, 2017: 214-223.

VITORIA P, SINTES J, BALLESTER C. Semantic image inpainting through improved wasserstein generative adversarial networks[C]// Proceedings of the 14 th International Joint Conference on Computer Vision , Imaging and Computer Graphics Theory and Applications . Funchal: SciTePress, 2019: 249-260.

RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[C]// Proceedings of the 18 th International Conference on Medical Image Computing and Computer-Assisted Intervention . Munich: Springer, 2015: 234-241.

张良 , 车进 , 杨琦 . 多粒度特征融合的行人再识别研究 . 液晶与显示 , 2020 . 35 ( 6 ): 555 - 563 . DOI: 10.3788/YJYXS20203506.0555 http://doi.org/10.3788/YJYXS20203506.0555 .

L ZHANG , J CHE , Q YANG . Multi-granularity feature fusion for person re-identification . Chinese Journal of Liquid Crystals and Displays , 2020 . 35 ( 6 ): 555 - 563 . DOI: 10.3788/YJYXS20203506.0555 http://doi.org/10.3788/YJYXS20203506.0555 .

N BAKER , H J LU , G ERLIKHMAN , 等 . Local features and global shape information in object classification by deep convolutional neural networks . Vision Research , 2020 . 172 46 - 61 . DOI: 10.1016/j.visres.2020.04.003 http://doi.org/10.1016/j.visres.2020.04.003 .

张海涛 , 张梦 . 引入通道注意力机制的SSD目标检测算法 . 计算机工程 , 2020 . 46 ( 8 ): 264 - 270 . DOI: 10.19678/j.issn.1000-3428.0054946 http://doi.org/10.19678/j.issn.1000-3428.0054946 .

H T ZHANG , M ZHANG . SSD target detection algorithm with channel attention mechanism . Computer Engineering , 2020 . 46 ( 8 ): 264 - 270 . DOI: 10.19678/j.issn.1000-3428.0054946 http://doi.org/10.19678/j.issn.1000-3428.0054946 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Multi-granularity feature fusion for person re-identification

Related Author

Yuan-yuan WAN

Yu-qing WANG

Xiao-ning ZHANG

Da-qun LI

Xiao-lin CHEN

Liang ZHANG

Jin CHE

Qi Yang

Related Institution

Changchun Institute of Optics， Fine Mechanics and Physics， Chinese Academy of Sciences

School of Physics and Electronic-Electrical Engineering, Ningxia University

Key Laboratory of Intelligent Sensing for Desert Information, Ningxia University

AI问答

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China 130033 Postal code：130033
Tel：0431-86176059 Email：yjxs@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰