Single frame image super-resolution reconstruction based on improved generative adversarial network

Zong-hang CHEN; Hai-long HU; Jian-min YAO; Qun YAN; Zhi-xian LIN

doi:10.37188/CJLCD.2020-0250

您当前的位置：

首页 >

文章列表页 >

Single frame image super-resolution reconstruction based on improved generative adversarial network

Image Processing | 更新时间：2021-04-28

- Single frame image super-resolution reconstruction based on improved generative adversarial network
- Chinese Journal of Liquid Crystals and Displays Vol. 36, Issue 5, Pages: 705-712(2021)
- 作者机构：
  
  1.福州大学物理与信息工程学院, 福建福州 350108
  2.晋江市博感电子科技有限公司, 福建晋江 362200
- 作者简介：
- 基金信息：
  
  National Key Research and Development Program of China(2016YFB0401503);Open Fund of Guangdong Provincial Key Laboratory of Optical Information Materials and Technology(2017B030301007)
- DOI：10.37188/CJLCD.2020-0250
  CLC： TP394.1;TH691.9
- Received：24 September 2020，
  
  Revised：08 November 2020，
  
  Published：2021-05
- 稿件说明：
移动端阅览
Zong-hang CHEN, Hai-long HU, Jian-min YAO, et al. Single frame image super-resolution reconstruction based on improved generative adversarial network[J]. Chinese journal of liquid crystals and displays, 2021, 36(5): 705-712.
DOI：

Zong-hang CHEN, Hai-long HU, Jian-min YAO, et al. Single frame image super-resolution reconstruction based on improved generative adversarial network[J]. Chinese journal of liquid crystals and displays, 2021, 36(5): 705-712. DOI： 10.37188/CJLCD.2020-0250.

摘要

为了获得更好的图像超分辨率重建质量，提高网络训练的稳定性，对生成对抗网络、损失函数进行研究。首先，介绍了SRGAN和DenseNet，并设计了基于DenseNet的生成网络用以生成图像，且将子像素卷积模块加入到DenseNet中。接着，移除了原本DenseNet中冗余的BN层，提高了模型的训练效率。最后，介绍了SRGAN的损失函数并基于Earth-Mover距离来重新设计损失函数，并且用SmoothL1损失取代MSE损失来计算VGG特征图，以防止MSE放大最大误差和最小误差间的差距。实验证明：该模型在网络训练过程中能够达到稳定收敛的状态。重建出的图像质量对比SRGAN，在3个基准测试集SET5，SET14，BSD100上的平均PSNR要高约2.02 dB，SSIM高约0.042(5.6%)。重建出的图像不仅在指标上有所提升，且拥有更好的清晰度，高频细节更为丰富。

Abstract

In order to obtain better image super-resolution reconstruction quality and improve the stability of network training

the generation of confrontation networks and loss functions are studied. Firstly

SRGAN and DenseNet are introduced

a generation network is designed to generate image based on DenseNet

and the sub-pixel convolution module is added to DenseNet. Then

the redundant BN layer in the original DenseNet is removed to improve the training efficiency of the model. Finally

the loss function of SRGAN is introduced and the loss function is redesigned based on the Earth-Mover distance

and the SmoothL1 loss is used to replace the MSE loss to calculate the VGG feature map to prevent MSE from amplifying the gap between the maximum error and the minimum error. Experiments prove that the model can achieve a stable convergence state during the network training process. The quality of the reconstructed image is compared with SRGAN

the average PSNR on the three benchmark test sets SET5

SET14

and BSD100 is about 2.02 dB higher

and SSIM is about 0.042 (5.6%) higher. The reconstructed image not only has improved indicators

but also has better definition and richer high-frequency details.

关键词

Keywords

references

朱福珍 , 刘越 , 黄鑫 , 等 . 改进的稀疏表示遥感图像超分辨重建 . 光学精密工程 , 2019 . 27 ( 3 ): 718 - 725 . https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM201903026.htm https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM201903026.htm .

F Z ZHU , Y LIU , X HUANG , 等 . Remote sensing image super-resolution based on improved sparse representation . Optics and Precision Engineering , 2019 . 27 ( 3 ): 718 - 725 . https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM201903026.htm https://www.cnki.com.cn/Article/CJFDTOTAL-GXJM201903026.htm .

WAN X F, YANG Y. Super-resolution image reconstruction[C] // Proceedings of 2010 International Conference on Computer Application and System Modeling . Taiyuan, China: IEEE, 2010.

C E DUCHON . Lanczos filtering in one and two dimensions . Journal of Applied Meteorology and Climatology , 1979 . 18 ( 8 ): 1016 - 1022 . DOI: 10.1175/1520-0450(1979)018<1016:LFIOAT>2.0.CO;2 http://doi.org/10.1175/1520-0450(1979)018<1016:LFIOAT>2.0.CO;2 .

HASAN M S, HAQUE S T. Single image super-resolution using back-propagation neural networks[C]// Proceedings of 2017 20 th International Conference of Computer and Information Technology . Dhaka: IEEE, 2017.

刘涛 , 钱锋 , 张葆 . 遥感图像的MAP超分辨重建 . 液晶与显示 , 2018 . 33 ( 10 ): 884 - 892 . http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20183310.0884 http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20183310.0884 .

T LIU , F QIAN , B ZHANG . MAP super-resolution reconstruction of remote sensing image . Chinese Journal of Liquid Crystals and Displays , 2018 . 33 ( 10 ): 884 - 892 . http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20183310.0884 http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20183310.0884 .

R R SCHULTZ , R L STEVENSON . Extraction of high-resolution frames from video sequences . IEEE Transactions on Image Processing , 1996 . 5 ( 6 ): 996 - 1011 . DOI: 10.1109/83.503915 http://doi.org/10.1109/83.503915 .

C DONG , C C LOY , K M HE , 等 . Image super-resolution using deep convolutional networks . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2016 . 38 ( 2 ): 295 - 307 . DOI: 10.1109/TPAMI.2015.2439281 http://doi.org/10.1109/TPAMI.2015.2439281 .

KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas: IEEE, 2016.

W S LAI , J B HUANG , N AHUJA , 等 . Fast and accurate image super-resolution with deep laplacian pyramid networks . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2019 . 41 ( 11 ): 2599 - 2613 . DOI: 10.1109/TPAMI.2018.2865304 http://doi.org/10.1109/TPAMI.2018.2865304 .

Y L TANG , W G GONG , X CHEN , 等 . Deep inception-residual laplacian pyramid networks for accurate single-image super-resolution . IEEE Transactions on Neural Networks and Learning Systems , 2020 . 31 ( 5 ): 1514 - 1528 . DOI: 10.1109/TNNLS.2019.2920852 http://doi.org/10.1109/TNNLS.2019.2920852 .

LEDIG C, THEIS L, HUSZÁR F, et al . Photo-realistic single image super-resolution using a generative adversarial network[C]// Proceedings of 2017 Conference on Computer Vision and Pattern Recognition . Honolulu: IEEE, 2017.

GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al . Generative adversarial networks[C]// Advances in Neural Information Processing Systems . 2014: 2672-2680.

H ZHAO , O GALLO , I FROSIO , 等 . Loss functions for image restoration with neural networks . IEEE Transactions on Computational Imaging , 2017 . 3 ( 1 ): 47 - 57 . DOI: 10.1109/TCI.2016.2644865 http://doi.org/10.1109/TCI.2016.2644865 .

ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN[J]. arXiv preprint arXiv , 1701.07875, 2017.

王功明 , 乔俊飞 , 王磊 . 一种能量函数意义下的生成式对抗网络 . 自动化学报 , 2018 . 44 ( 5 ): 793 - 803 . https://www.cnki.com.cn/Article/CJFDTOTAL-MOTO201805003.htm https://www.cnki.com.cn/Article/CJFDTOTAL-MOTO201805003.htm .

G M WANG , J F QIAO , L WANG . A generative adversarial network based on energy function . Acta Automatica Sinica , 2018 . 44 ( 5 ): 793 - 803 . https://www.cnki.com.cn/Article/CJFDTOTAL-MOTO201805003.htm https://www.cnki.com.cn/Article/CJFDTOTAL-MOTO201805003.htm .

HE K M, ZHANG X Y, REN S Q, et al . Deep residual learning for image recognition[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas: IEEE, 2016.

HUANG G, LIU Z, VAN DER MAATEN L, et al . Densely connected convolutional networks[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu: IEEE, 2017.

SHI W Z, CABALLERO J, HUSZÁR F, et al . Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas: IEEE, 2016.

LIM B, SON S, KIM H, et al . Enhanced deep residual networks for single image super-resolution[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops . Honolulu: IEEE, 2017.

IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift[C]// Proceedings of the 32 nd International Conference on International Conference on Machine Learning . Lille: ACM, 2015.

O SHAMIR . Discussion of: Nonparametric regression using deep neural networks with ReLU activation function . The Annals of Statistics , 2020 . 48 ( 4 ): 1911 - 1915 .

ARJOVSKY M, BOTTOU L. Towards principled methods for training generative adversarial networks[J]. arXiv preprint arXiv , 1701.04862, 2017.

吴磊 , 吕国强 , 赵晨 , 等 . 基于多尺度残差网络的CT图像超分辨率重建 . 液晶与显示 , 2019 . 34 ( 10 ): 1006 - 1012 . http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20193410.1006 http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20193410.1006 .

L WU , G Q YU , C ZHAO , 等 . CT image super-resolution reconstruction based on multi-scale residual network . Chinese Journal of Liquid Crystals and Displays , 2019 . 34 ( 10 ): 1006 - 1012 . http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20193410.1006 http://cjlcd.lightpublishing.cn/thesisDetails#10.3788/YJYXS20193410.1006 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Video anomaly detection based on ensemble generative adversarial networks

Gradient-aware based single image super-resolution

Video inpainting based on residual convolution attention network

Related Author

Zong-hang CHEN

Hai-long HU

Jian-min YAO

Qun YAN

Zhi-xian Lin

GU Jia-cheng

LONG Ying-wen

JI Ming-ming

Related Institution

College of Physics and Information Engineering， Fuzhou University

Jinjiang RichSense Electronic Technology Co.， Ltd.， Jinjiang

School of Electronic and Electrical Engineering， Shanghai University of Engineering Science

School of Electronic Information and Artificial Intelligence，Shaanxi University of Science and Technology

National Astronomical Observatories， Chinese Academy of Sciences

AI问答

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China 130033 Postal code：130033
Tel：0431-86176059 Email：yjxs@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰