基于改进生成对抗网络的单帧图像超分辨率重建

陈宗航; 胡海龙; 姚剑敏; 严群; 林志贤

doi:10.37188/CJLCD.2020-0250

您当前的位置：

首页 >

文章列表页 >

基于改进生成对抗网络的单帧图像超分辨率重建

图像处理 | 更新时间：2021-04-28

- 基于改进生成对抗网络的单帧图像超分辨率重建
- Single frame image super-resolution reconstruction based on improved generative adversarial network
- 液晶与显示 2021年36卷第5期页码：705-712
- 作者机构：
  
  1.福州大学物理与信息工程学院, 福建福州 350108
  2.晋江市博感电子科技有限公司, 福建晋江 362200
- 作者简介：
  
  [ "陈宗航(1996-), 男, 福建福州人, 硕士研究生, 2017年于集美大学获得学士学位, 主要从事图像超分辨率重建及服务器开发方面的研究。E-mail: czh1995czh@gmail.com" ]
  [ "姚剑敏(1978-), 男, 福建莆田人, 2005年于中国科学院长春光学精密机械与物理研究所获得博士学位, 主要从事深度学习、图像处理等方面的研究。E-mail: yaojm@fzu.edu.cn" ]
- 基金信息：
  
  国家重点研发计划课题(2016YFB0401503);广东省科技重大专项(2016B090906001);福建省科技重大专项(2014HZ0003-1);广东省光信息材料与技术重点实验室开放基金(2017B030301007)
- DOI：10.37188/CJLCD.2020-0250
  中图分类号： TP394.1;TH691.9
- 收稿日期：2020-09-24，
  
  修回日期：2020-11-08，
  
  纸质出版日期：2021-05
- 稿件说明：
移动端阅览
陈宗航, 胡海龙, 姚剑敏, 等. 基于改进生成对抗网络的单帧图像超分辨率重建[J]. 液晶与显示, 2021,36(5):705-712.

Zong-hang CHEN, Hai-long HU, Jian-min YAO, et al. Single frame image super-resolution reconstruction based on improved generative adversarial network[J]. Chinese journal of liquid crystals and displays, 2021, 36(5): 705-712.
陈宗航, 胡海龙, 姚剑敏, 等. 基于改进生成对抗网络的单帧图像超分辨率重建[J]. 液晶与显示, 2021,36(5):705-712. DOI： 10.37188/CJLCD.2020-0250.

Zong-hang CHEN, Hai-long HU, Jian-min YAO, et al. Single frame image super-resolution reconstruction based on improved generative adversarial network[J]. Chinese journal of liquid crystals and displays, 2021, 36(5): 705-712. DOI： 10.37188/CJLCD.2020-0250.

摘要

为了获得更好的图像超分辨率重建质量，提高网络训练的稳定性，对生成对抗网络、损失函数进行研究。首先，介绍了SRGAN和DenseNet，并设计了基于DenseNet的生成网络用以生成图像，且将子像素卷积模块加入到DenseNet中。接着，移除了原本DenseNet中冗余的BN层，提高了模型的训练效率。最后，介绍了SRGAN的损失函数并基于Earth-Mover距离来重新设计损失函数，并且用SmoothL1损失取代MSE损失来计算VGG特征图，以防止MSE放大最大误差和最小误差间的差距。实验证明：该模型在网络训练过程中能够达到稳定收敛的状态。重建出的图像质量对比SRGAN，在3个基准测试集SET5，SET14，BSD100上的平均PSNR要高约2.02 dB，SSIM高约0.042(5.6%)。重建出的图像不仅在指标上有所提升，且拥有更好的清晰度，高频细节更为丰富。

Abstract

In order to obtain better image super-resolution reconstruction quality and improve the stability of network training

the generation of confrontation networks and loss functions are studied. Firstly

SRGAN and DenseNet are introduced

a generation network is designed to generate image based on DenseNet

and the sub-pixel convolution module is added to DenseNet. Then

the redundant BN layer in the original DenseNet is removed to improve the training efficiency of the model. Finally

the loss function of SRGAN is introduced and the loss function is redesigned based on the Earth-Mover distance

and the SmoothL1 loss is used to replace the MSE loss to calculate the VGG feature map to prevent MSE from amplifying the gap between the maximum error and the minimum error. Experiments prove that the model can achieve a stable convergence state during the network training process. The quality of the reconstructed image is compared with SRGAN

the average PSNR on the three benchmark test sets SET5

SET14

and BSD100 is about 2.02 dB higher

and SSIM is about 0.042 (5.6%) higher. The reconstructed image not only has improved indicators

but also has better definition and richer high-frequency details.

关键词

Keywords

references

朱福珍 , 刘越 , 黄鑫 , 等 . 改进的稀疏表示遥感图像超分辨重建 . 光学精密工程 , 2019 . 27 ( 3 ): 718 - 725 . https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM201903026.htm https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM201903026.htm .

F Z ZHU , Y LIU , X HUANG , 等 . Remote sensing image super-resolution based on improved sparse representation . Optics and Precision Engineering , 2019 . 27 ( 3 ): 718 - 725 . https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM201903026.htm https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM201903026.htm .

WAN X F, YANG Y. Super-resolution image reconstruction[C] // Proceedings of 2010 International Conference on Computer Application and System Modeling . Taiyuan, China: IEEE, 2010.

C E DUCHON . Lanczos filtering in one and two dimensions . Journal of Applied Meteorology and Climatology , 1979 . 18 ( 8 ): 1016 - 1022 . DOI: 10.1175/1520-0450(1979)018<1016:LFIOAT>2.0.CO;2 http://doi.org/10.1175/1520-0450(1979)018<1016:LFIOAT>2.0.CO;2 .

HASAN M S, HAQUE S T. Single image super-resolution using back-propagation neural networks[C]// Proceedings of 2017 20 th International Conference of Computer and Information Technology . Dhaka: IEEE, 2017.

刘涛 , 钱锋 , 张葆 . 遥感图像的MAP超分辨重建 . 液晶与显示 , 2018 . 33 ( 10 ): 884 - 892 . http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20183310.0884 http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20183310.0884 .

T LIU , F QIAN , B ZHANG . MAP super-resolution reconstruction of remote sensing image . Chinese Journal of Liquid Crystals and Displays , 2018 . 33 ( 10 ): 884 - 892 . http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20183310.0884 http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20183310.0884 .

R R SCHULTZ , R L STEVENSON . Extraction of high-resolution frames from video sequences . IEEE Transactions on Image Processing , 1996 . 5 ( 6 ): 996 - 1011 . DOI: 10.1109/83.503915 http://doi.org/10.1109/83.503915 .

C DONG , C C LOY , K M HE , 等 . Image super-resolution using deep convolutional networks . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2016 . 38 ( 2 ): 295 - 307 . DOI: 10.1109/TPAMI.2015.2439281 http://doi.org/10.1109/TPAMI.2015.2439281 .

KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas: IEEE, 2016.

W S LAI , J B HUANG , N AHUJA , 等 . Fast and accurate image super-resolution with deep laplacian pyramid networks . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2019 . 41 ( 11 ): 2599 - 2613 . DOI: 10.1109/TPAMI.2018.2865304 http://doi.org/10.1109/TPAMI.2018.2865304 .

Y L TANG , W G GONG , X CHEN , 等 . Deep inception-residual laplacian pyramid networks for accurate single-image super-resolution . IEEE Transactions on Neural Networks and Learning Systems , 2020 . 31 ( 5 ): 1514 - 1528 . DOI: 10.1109/TNNLS.2019.2920852 http://doi.org/10.1109/TNNLS.2019.2920852 .

LEDIG C, THEIS L, HUSZÁR F, et al . Photo-realistic single image super-resolution using a generative adversarial network[C]// Proceedings of 2017 Conference on Computer Vision and Pattern Recognition . Honolulu: IEEE, 2017.

GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al . Generative adversarial networks[C]// Advances in Neural Information Processing Systems . 2014: 2672-2680.

H ZHAO , O GALLO , I FROSIO , 等 . Loss functions for image restoration with neural networks . IEEE Transactions on Computational Imaging , 2017 . 3 ( 1 ): 47 - 57 . DOI: 10.1109/TCI.2016.2644865 http://doi.org/10.1109/TCI.2016.2644865 .

ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN[J]. arXiv preprint arXiv , 1701.07875, 2017.

王功明 , 乔俊飞 , 王磊 . 一种能量函数意义下的生成式对抗网络 . 自动化学报 , 2018 . 44 ( 5 ): 793 - 803 . https://www.cnki.com.cn/Article/CJFDTOTAL-MOTO201805003.htm https://www.cnki.com.cn/Article/CJFDTOTAL-MOTO201805003.htm .

G M WANG , J F QIAO , L WANG . A generative adversarial network based on energy function . Acta Automatica Sinica , 2018 . 44 ( 5 ): 793 - 803 . https://www.cnki.com.cn/Article/CJFDTOTAL-MOTO201805003.htm https://www.cnki.com.cn/Article/CJFDTOTAL-MOTO201805003.htm .

HE K M, ZHANG X Y, REN S Q, et al . Deep residual learning for image recognition[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas: IEEE, 2016.

HUANG G, LIU Z, VAN DER MAATEN L, et al . Densely connected convolutional networks[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu: IEEE, 2017.

SHI W Z, CABALLERO J, HUSZÁR F, et al . Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas: IEEE, 2016.

LIM B, SON S, KIM H, et al . Enhanced deep residual networks for single image super-resolution[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops . Honolulu: IEEE, 2017.

IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift[C]// Proceedings of the 32 nd International Conference on International Conference on Machine Learning . Lille: ACM, 2015.

O SHAMIR . Discussion of: Nonparametric regression using deep neural networks with ReLU activation function . The Annals of Statistics , 2020 . 48 ( 4 ): 1911 - 1915 .

ARJOVSKY M, BOTTOU L. Towards principled methods for training generative adversarial networks[J]. arXiv preprint arXiv , 1701.04862, 2017.

吴磊 , 吕国强 , 赵晨 , 等 . 基于多尺度残差网络的CT图像超分辨率重建 . 液晶与显示 , 2019 . 34 ( 10 ): 1006 - 1012 . http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20193410.1006 http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20193410.1006 .

L WU , G Q YU , C ZHAO , 等 . CT image super-resolution reconstruction based on multi-scale residual network . Chinese Journal of Liquid Crystals and Displays , 2019 . 34 ( 10 ): 1006 - 1012 . http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20193410.1006 http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20193410.1006 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于集成生成对抗网络的视频异常事件检测方法

基于梯度感知的单幅图像超分辨

基于残差卷积注意力网络的视频修复