Gradient-aware based single image super-resolution

Le ZHOU; Long XU; Xiao-yan LIU; Xin-ze ZHANG; Xuan-de ZHANG

doi:10.37188/CJLCD.2022-0083

您当前的位置：

首页 >

文章列表页 >

Gradient-aware based single image super-resolution

Image Processing | 更新时间：2022-09-26

- Gradient-aware based single image super-resolution
- Chinese Journal of Liquid Crystals and Displays Vol. 37, Issue 10, Pages: 1334-1344(2022)
- 作者机构：
  
  1.陕西科技大学电子信息与人工智能学院，陕西西安 710021
  2.中国科学院国家天文台，北京 100101
  3.西安石油大学理学院，陕西西安 710065
- 作者简介：
- 基金信息：
  
  Supported by National Natural Science Foundation of China(61871260)
- DOI：10.37188/CJLCD.2022-0083
  CLC： TP391;TP751.1
- Received：11 March 2022，
  
  Revised：26 March 2022，
  
  Published：05 October 2022
- 稿件说明：
移动端阅览
Le ZHOU, Long XU, Xiao-yan LIU, et al. Gradient-aware based single image super-resolution[J]. Chinese journal of liquid crystals and displays, 2022, 37(10): 1334-1344.
DOI：

Le ZHOU, Long XU, Xiao-yan LIU, et al. Gradient-aware based single image super-resolution[J]. Chinese journal of liquid crystals and displays, 2022, 37(10): 1334-1344. DOI： 10.37188/CJLCD.2022-0083.

摘要

随着生成对抗网络在图像超分辨（SR）领域的应用，一些感知驱动的SR方法可以恢复出纹理细节更加丰富的SR图像，有效地缓解了由PSNR主导的SR方法导致重建图像趋于平滑的问题。然而梯度信息作为图像纹理的一种重要表现形式，鲜有SR方法能准确、高效地利用。为此，提出一种基于梯度感知的图像超分辨（GASR）算法，可以更准确地利用梯度信息。一方面，使用梯度域的特征图作为作用在图像域特征图上的卷积核，有效地避免了不同域特征图串联所带来的域冲突问题；另一方面，通过对卷积核尺寸等设计细节的调整使两个分支对应位置所输出的图像域与梯度域特征图的感受野一致。此外，由于实际应用对网络轻量化需求的提高，提出的GASR算法还有效降低了参数量和计算量。与同样利用梯度信息的SPSR相比，GASR最终以约1/6的参数量与1/10的计算量取得了与其相近的性能。在Set14数据集上，LPIPS与PSNR分别提升了0.002 2与0.217。实验结果验证了GASR可以在纹理生成与图像平滑度之间取得一个很好的平衡，视觉化效果也验证了GASR不仅可以准确地恢复SR图像，而且在一定程度上缓解了杂乱纹理的生成。

Abstract

With the application of generative adversarial networks in the field of image super-resolution （SR)‍， some perception-driven SR methods can recover SR images with richer texture details， effectively alleviating the over-smoothing problem of the PSNR dominated SR methods. Gradient information is an important representation of image texture. However， few SR methods can make use of this information accurately and efficiently. In this paper， a gradient-aware single image super-resolution（GASR） is proposed， using gradient information better from two aspects. On the one hand， the feature map of gradient domain is used as the convolution kernel imposing on the feature map of image domain， which can effectively avoid the domain conflict caused by the concatenation of feature map of different domains. On the other hand， by elaborating the network details such as convolution kernel size，

etc

.， the image fields output at the corresponding positions of the two branches are consistent with the feel fields of the feature maps in the gradient domain. In addition， the proposed GASR algorithm also effectively reduces the number of parameters and the amount of computation due to the increased demand for network lightweight in practical applications. Compared to SPSR， GASR can achieve the same performance at the cost of about 1/6 of the parameters and 1/10 of the computation of SPSR. On Set14 dataset， LPIPS and PSNR increase by 0.002 2 and 0.217， respectively. The experimental results show that GASR can achieve a good trade-off between texture details and image smoothness. In addition， GASR can not only reconstruct high fidelity SR image， but also alleviate the generation of messy textures.

关键词

Keywords

references

刘涛，钱锋，张葆 . 遥感图像的MAP超分辨重建［J］. 液晶与显示， 2018 ， 33 （ 10 ）： 884 - 892 . doi: 10.3788/yjyxs20183310.0884 http://dx.doi.org/10.3788/yjyxs20183310.0884

LIU T ， QIAN F ， ZHANG B . MAP super-resolution reconstruction of remote sensing image ［J］. Chinese Journal of Liquid Crystals and Displays ， 2018 ， 33 （ 10 ）： 884 - 892 . （in Chinese） . doi: 10.3788/yjyxs20183310.0884 http://dx.doi.org/10.3788/yjyxs20183310.0884

DONG C ， LOY C C ， HE K M ， et al . Learning a deep convolutional network for image super-resolution ［C］// Proceedings of the 13th European Conference on Computer Vision . Zurich ： Springer ， 2014 ： 184 - 199 . doi: 10.1007/978-3-319-10593-2_13 http://dx.doi.org/10.1007/978-3-319-10593-2_13

LEDIG C ， THEIS L ， HUSZÁR F ， et al . Photo-realistic single image super-resolution using a generative adversarial network ［C］// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu ： IEEE ， 2017 ： 105 - 114 . doi: 10.1109/cvpr.2017.19 http://dx.doi.org/10.1109/cvpr.2017.19

MA C ， RAO Y M ， CHENG Y A ， et al . Structure-preserving super resolution with gradient guidance ［C］// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle ： IEEE ， 2020 ： 7766 - 7775 . doi: 10.1109/cvpr42600.2020.00779 http://dx.doi.org/10.1109/cvpr42600.2020.00779

JOHNSON J ， ALAHI A ， LI F F . Perceptual losses for real-time style transfer and super-resolution ［C］// Proceedings of the 14th European Conference on Computer Vision . Amsterdam ： Springer ， 2016 ： 694 - 711 . doi: 10.1007/978-3-319-46475-6_43 http://dx.doi.org/10.1007/978-3-319-46475-6_43

KIM J ， LEE J K ， LEE K M . Accurate image super-resolution using very deep convolutional networks ［C］// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas ： IEEE ， 2016 ： 1646 - 1654 . doi: 10.1109/cvpr.2016.182 http://dx.doi.org/10.1109/cvpr.2016.182

ZHANG K ， ZUO W M ， CHEN Y J ， et al . Beyond a Gaussian denoiser： residual learning of deep CNN for image denoising ［J］. IEEE Transactions on Image Processing ， 2017 ， 26 （ 7 ）： 3142 - 3155 . doi: 10.1109/tip.2017.2662206 http://dx.doi.org/10.1109/tip.2017.2662206

DONG C ， LOY C C ， TANG X O . Accelerating the super-resolution convolutional neural network ［C］// Proceedings of the 14th European Conference on Computer Vision . Amsterdam ： Springer ， 2016 ： 391 - 407 . doi: 10.1007/978-3-319-46475-6_25 http://dx.doi.org/10.1007/978-3-319-46475-6_25

ZHANG Y L ， LI K P ， LI K ， et al . Image super-resolution using very deep residual channel attention networks ［C］// Proceedings of the 15th European Conference on Computer Vision . Munich ： Springer ， 2018 ： 294 - 310 . doi: 10.1007/978-3-030-01234-2_18 http://dx.doi.org/10.1007/978-3-030-01234-2_18

ZHANG Y L ， TIAN Y P ， KONG Y ， et al . Residual dense network for image super-resolution ［C］// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City ： IEEE ， 2018 ： 2472 - 2481 . doi: 10.1109/cvpr.2018.00262 http://dx.doi.org/10.1109/cvpr.2018.00262

LIU J ， ZHANG W J ， TANG Y T ， et al . Residual feature aggregation network for image super-resolution ［C］// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle ： IEEE ， 2020 ： 2356 - 2365 . doi: 10.1109/cvpr42600.2020.00243 http://dx.doi.org/10.1109/cvpr42600.2020.00243

GOODFELLOW I J ， POUGET-ABADIE J ， MIRZA M ， et al . Generative adversarial nets ［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems . Montreal ： MIT Press ， 2014 ： 2672 - 2680 .

WANG X T ， YU K ， WU S X ， et al . ESRGAN： enhanced super-resolution generative adversarial networks ［C］// Proceedings of European Conference on Computer Vision . Munich ： Springer ， 2018 ： 63 - 79 . doi: 10.1007/978-3-030-11021-5_5 http://dx.doi.org/10.1007/978-3-030-11021-5_5

KONG X T ， ZHAO H Y ， QIAO Y ， et al . ClassSR： a general framework to accelerate super-resolution networks by data characteristic ［C］// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Nashville ： IEEE ， 2021 ： 12011 - 12020 . doi: 10.1109/cvpr46437.2021.01184 http://dx.doi.org/10.1109/cvpr46437.2021.01184

SONG D H ， WANG Y H ， CHEN H T ， et al . Addersr： towards energy efficient image super-resolution ［C］// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Nashville ： IEEE ， 2021 ： 15643 - 15652 . doi: 10.1109/cvpr46437.2021.01539 http://dx.doi.org/10.1109/cvpr46437.2021.01539

WANG L G ， DONG X Y ， WANG Y Q ， et al . Exploring sparsity in image super-resolution for efficient inference ［C］// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Nashville ： IEEE ， 2021 ： 4915 - 4924 . doi: 10.1109/cvpr46437.2021.00488 http://dx.doi.org/10.1109/cvpr46437.2021.00488

AGUSTSSON E ， TIMOFTE R . NTIRE 2017 challenge on single image super-resolution： dataset and study ［C］// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops . Honolulu ： IEEE ， 2017 ： 1122 - 1131 . doi: 10.1109/cvprw.2017.150 http://dx.doi.org/10.1109/cvprw.2017.150

ZEYDE R ， ELAD M ， PROTTER M . On single image scale-up using sparse-representations ［C］// Proceedings of the 7th International Conference on Curves and Surfaces . Avignon ： Springer ， 2010 ： 711 - 730 .

MARTIN D ， FOWLKES C ， TAL D ， et al . A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics ［C］// Proceedings of the Eighth IEEE International Conference on Computer Vision . Vancouver ： IEEE ， 2001 ： 416 - 423 .

BLAU Y ， MECHREZ R ， TIMOFTE R ， et al . The 2018 PIRM challenge on perceptual image super-resolution ［C］// Proceedings of European Conference on Computer Vision . Munich ： Springer ， 2018 ： 334 - 355 . doi: 10.1007/978-3-030-11021-5_21 http://dx.doi.org/10.1007/978-3-030-11021-5_21

ZHANG R ， ISOLA P ， EFROS A A ， et al . The unreasonable effectiveness of deep features as a perceptual metric ［C］// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City ： IEEE ， 2018 ： 586 - 595 . doi: 10.1109/cvpr.2018.00068 http://dx.doi.org/10.1109/cvpr.2018.00068

WANG Z ， BOVIK A C ， SHEIKH H R ， et al . Image quality assessment： from error visibility to structural similarity ［J］. IEEE Transactions on Image Processing ， 2004 ， 13 （ 4 ）： 600 - 612 . doi: 10.1109/tip.2003.819861 http://dx.doi.org/10.1109/tip.2003.819861

KINGMA D P ， BA J . Adam： a method for stochastic optimization ［J］. arXiv ， 2014 ： 1412 .6980.

WANG X T ， YU K ， DONG C ， et al . Recovering realistic texture in image super-resolution by deep spatial feature transform ［C］// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City ： IEEE ， 2018 ： 606 - 615 . doi: 10.1109/cvpr.2018.00070 http://dx.doi.org/10.1109/cvpr.2018.00070

SAJJADI M S M ， SCHÖLKOPF B ， HIRSCH M . Enhancenet： single image super-resolution through automated texture synthesis ［C］// Proceedings of 2017 IEEE International Conference on Computer Vision . Venice ： IEEE ， 2017 ： 4501 - 4510 . doi: 10.1109/iccv.2017.481 http://dx.doi.org/10.1109/iccv.2017.481

ZHANG K ， VAN GOOL L ， TIMOFTE R . Deep unfolding network for image super-resolution ［C］// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle ： IEEE ， 2020 ： 3214 - 3223 . doi: 10.1109/cvpr42600.2020.00328 http://dx.doi.org/10.1109/cvpr42600.2020.00328

CAI J R ， ZENG H ， YONG H W ， et al . Toward real-world single image super-resolution： a new benchmark and a new model ［C］// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision . Seoul ： IEEE ， 2019 ： 3086 - 3095 . doi: 10.1109/iccv.2019.00318 http://dx.doi.org/10.1109/iccv.2019.00318

Views

213

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Image super-resolution reconstruction using pyramid variance pooling network

Lens-free adversarial encoded imaging method based on convolutional attention mechanism

Single frame image super-resolution reconstruction based on improved generative adversarial network

Related Author

ZHOU Le

XU Long

ZHANG Xuan-de

ZHANG Xin-ze

PENG Yanfei

LI Yongxin

MENG Xin

CUI Yun

Related Institution

School of Electronic and Information Engineering， Liaoning Technical University

Optica Information Science and Technology Department， Jiangnan University

College of Physics and Information Engineering, Fuzhou University

Jinjiang RichSense Electronic Technology Co., Ltd., Jinjiang

Optica Information Science and Technology Department， Jiangnan University

AI问答

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China 130033 Postal code：130033
Tel：0431-86176059 Email：yjxs@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰