结合全局语义优化的对抗性灰度图像彩色化

万园园; 王雨青; 张晓宁; 李荅群; 陈小林

doi:10.37188/CJLCD.2021-0012

您当前的位置：

首页 >

文章列表页 >

结合全局语义优化的对抗性灰度图像彩色化

图像处理 | 更新时间：2021-09-10

- 结合全局语义优化的对抗性灰度图像彩色化
- Adversarial grayscale image colorization combined with global semantic optimization
- 液晶与显示 2021年36卷第9期页码：1305-1313
- 作者机构：
  
  1.中国科学院长春光学精密机械与物理研究所, 吉林长春 130033
  2.中国科学院大学, 北京 100049
- 作者简介：
  
  [ "万园园(1996-), 女, 江西南昌人, 硕士研究生, 2016年于南昌大学获得学士学位, 主要从事计算机视觉及机器学习方面的研究。E-mail: 1123362384@qq.com" ]
  [ "陈小林(1980-), 男, 吉林长春人, 博士, 副研究员, 2010年于中国科学院长春光学精密机械与物理研究所获得博士学位, 研究方向为实时视频图像处理, 嵌入式处理器设计, 景象模拟仿真嵌入式实时处理器设计。E-mail: cl.hong@163.com" ]
- 基金信息：
  
  国家重点研发计划(2018YFC0308100)
- DOI：10.37188/CJLCD.2021-0012
  中图分类号： TP391.4
- 收稿日期：2021-01-12，
  
  修回日期：2021-01-28，
  
  纸质出版日期：2021-09
- 稿件说明：
移动端阅览
万园园, 王雨青, 张晓宁, 等. 结合全局语义优化的对抗性灰度图像彩色化[J]. 液晶与显示, 2021,36(9):1305-1313.

Yuan-yuan WAN, Yu-qing WANG, Xiao-ning ZHANG, et al. Adversarial grayscale image colorization combined with global semantic optimization[J]. Chinese journal of liquid crystals and displays, 2021, 36(9): 1305-1313.
万园园, 王雨青, 张晓宁, 等. 结合全局语义优化的对抗性灰度图像彩色化[J]. 液晶与显示, 2021,36(9):1305-1313. DOI： 10.37188/CJLCD.2021-0012.

Yuan-yuan WAN, Yu-qing WANG, Xiao-ning ZHANG, et al. Adversarial grayscale image colorization combined with global semantic optimization[J]. Chinese journal of liquid crystals and displays, 2021, 36(9): 1305-1313. DOI： 10.37188/CJLCD.2021-0012.

摘要

针对当前灰度图像彩色化算法容易出现色彩枯燥、颜色溢出和图像细节损失等问题，本文提出一种结合全局语义优化的对抗性灰度图像彩色化算法。其中，生成网络采用自主改进的U-Net网络。一方面，改进的U-Net网络利用多层卷积对输入图像进行逐步下采样，在获取多尺度层级特征和全局特征的同时，在跳跃连接中将全局特征和多尺度层级特征进行自适应融合，从而有效增强算法对全局语义信息的理解能力并缓解颜色溢出的现象；另一方面，改进的U-Net网络在上采样过程中融合通道注意力模块，使得在提取卷积特征时能够有效抑制噪声并降低特征冗余性。判别网络主要采用全卷积结构，通过反向传播误差以达到优化生成网络的目的。此外，本文算法的损失函数将WGAN-GP网络的优化思想和颜色损失相结合，从而解决传统生成对抗网络训练时出现的梯度消失和模式崩溃等问题。本文算法在Place365测试集上所获取的峰值信噪比、结构相似性和信息熵指标分别为24.455 dB、0.943和7.489。实验结果表明，本文算法能够缓解颜色溢出，且细节保持和颜色饱和度方面都具有一定优势。

Abstract

The current colorization algorithms generally suffer from color boring

color bleeding and detail loss. In order to address these problems

this paper proposes a adversarial colorization algorithm combined with global semantic optimization. The generator adopts the improved U-Net network. On the one hand

the improved U-Net gradually obtains multi-scale features and global feature by multiple convolution during the downsampling process. At the same time

the global feature and multi-scale features get adaptive fused in the skip connections

thus it can effectively enhance algorithm's ability to understand global semantic and ease the color bleeding. On the other hand

the improved U-Net network integrates the channel attention module in the upsampling process

which can effectively suppress the noise and reduce the feature redundancy when the convolutional features are extracted. The discriminator mainly adopts full convolution structure and achieves the purpose of optimizing the generator by reverse error transfer. In addition

the loss function combines the optimization idea of WGAN-GP with color loss to solve the gradient disappearance and mode collapse problems in the training of traditional GAN. The proposed algorithm is tested on Place365 test set

and the PSNR

SSIM and IE indexes are 24.455 dB

0.943 and 7.489

respectively. The test results show that the proposed algorithm can alleviate color bleeding

and has certain advantages in detail retention and color saturation.

关键词

Keywords

references

CHENG Z Z, YANG Q X, SHENG B. Deep colorization[C]// Proceedings of 2015 IEEE International Conference on Computer Vision . Santiago: IEEE, 2015: 415-423.

S IIZUKA , E SIMO-SERRA , H ISHIKAWA . Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification . ACM Transactions on Graphics , 2016 . 35 ( 4 ): 110 10.1145/2897824.2925974 10.1145/2897824.2925974 .

ZHANG R, ISOLA P, EFROS A A. Colorful image colorization[C]// Proceedings of the 14 th European Conference on Computer Vision . Amsterdam: Springer, 2016: 649-666.

LARSSON G, MAIRE M, SHAKHNAROVICH G. Learning representations for automatic colorization[C]// Proceedings of the 14 th European Conference on Computer Vision . Amsterdam: Springer, 2016: 577-593.

WANG X T, YU K, WU S X, et al . ESRGAN: enhanced super-resolution generative adversarial networks[C]// Proceedings of European Conference on Computer Vision . Munich: Springer, 2018: 63-79.

余思泉 , 韩志 , 唐延东 , 等 . 基于对抗生成网络的纹理合成方法 . 红外与激光工程 , 2018 . 47 ( 2 ): 0203005 10.3788/IRLA201847.0203005 10.3788/IRLA201847.0203005 .

S Q YU , Z HAN , Y D TANG , 等 . Texture synthesis method based on generative adversarial networks . Infrared and Laser Engineering , 2018 . 47 ( 2 ): 0203005 10.3788/IRLA201847.0203005 10.3788/IRLA201847.0203005 .

杜振龙 , 沈海洋 , 宋国美 , 等 . 基于改进CycleGAN的图像风格迁移 . 光学精密工程 , 2019 . 27 ( 8 ): 1836 - 1844 . DOI: 10.3788/OPE.20192708.1836 http://doi.org/10.3788/OPE.20192708.1836 .

Z L DU , H Y SHEN , G M SONG , 等 . Image style transfer based on improved CycleGAN . Optics and Precision Engineering , 2019 . 27 ( 8 ): 1836 - 1844 . DOI: 10.3788/OPE.20192708.1836 http://doi.org/10.3788/OPE.20192708.1836 .

ISOLA P, ZHU J Y, ZHOU T H, et al . Image-to-image translation with conditional adversarial networks[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu: IEEE, 2017: 5967-5976.

NAZERI K, NG E, EBRAHIMI M. Image colorization with generative adversarial networks[J]. arXiv: 1803.05400, 2018.

CAO Y, ZHOU Z M, ZHANG W N, et al . Unsupervised diverse colorization via generative adversarial networks[C]// Proceedings of Joint European Conference on Machine Learning and Knowledge Discovery in Databases . Skopje: Springer, 2017: 151-166.

VITORIA P, RAAD L, BALLESTER C. ChromaGAN: adversarial picture colorization with semantic classdistribution[C]// Proceedings of 2020 IEEE Winter Conference on Applications of Computer Vision . Snowmass: IEEE, 2020: 2434-2443.

S H ZHANG , H LI , W H KONG . Object counting method based on dual attention network . IET Image Processing , 2020 . 14 ( 8 ): 1621 - 1627 . DOI: 10.1049/iet-ipr.2019.0465 http://doi.org/10.1049/iet-ipr.2019.0465 .

ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein generative adversarial networks[C]// Proceedings of the 34 th International Conference on Machine Learning . Sydney: PMLR, 2017: 214-223.

VITORIA P, SINTES J, BALLESTER C. Semantic image inpainting through improved wasserstein generative adversarial networks[C]// Proceedings of the 14 th International Joint Conference on Computer Vision , Imaging and Computer Graphics Theory and Applications . Funchal: SciTePress, 2019: 249-260.

RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[C]// Proceedings of the 18 th International Conference on Medical Image Computing and Computer-Assisted Intervention . Munich: Springer, 2015: 234-241.

张良 , 车进 , 杨琦 . 多粒度特征融合的行人再识别研究 . 液晶与显示 , 2020 . 35 ( 6 ): 555 - 563 . DOI: 10.3788/YJYXS20203506.0555 http://doi.org/10.3788/YJYXS20203506.0555 .

L ZHANG , J CHE , Q YANG . Multi-granularity feature fusion for person re-identification . Chinese Journal of Liquid Crystals and Displays , 2020 . 35 ( 6 ): 555 - 563 . DOI: 10.3788/YJYXS20203506.0555 http://doi.org/10.3788/YJYXS20203506.0555 .

N BAKER , H J LU , G ERLIKHMAN , 等 . Local features and global shape information in object classification by deep convolutional neural networks . Vision Research , 2020 . 172 46 - 61 . DOI: 10.1016/j.visres.2020.04.003 http://doi.org/10.1016/j.visres.2020.04.003 .

张海涛 , 张梦 . 引入通道注意力机制的SSD目标检测算法 . 计算机工程 , 2020 . 46 ( 8 ): 264 - 270 . DOI: 10.19678/j.issn.1000-3428.0054946 http://doi.org/10.19678/j.issn.1000-3428.0054946 .

H T ZHANG , M ZHANG . SSD target detection algorithm with channel attention mechanism . Computer Engineering , 2020 . 46 ( 8 ): 264 - 270 . DOI: 10.19678/j.issn.1000-3428.0054946 http://doi.org/10.19678/j.issn.1000-3428.0054946 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

多粒度特征融合的行人再识别研究