1.福州大学 电气工程与自动化学院, 福建 福州 350108
2.中国科学院 福建物质结构研究所, 福建 福州 350108
扫 描 看 全 文
ZHANG Yong, GUO Jie-long, WANG Fan, et al. Image rain removal algorithm based on multi-cascade progressive convolution structure. [J]. Chinese Journal of Liquid Crystals and Displays 38(10):1409-1422(2023)
ZHANG Yong, GUO Jie-long, WANG Fan, et al. Image rain removal algorithm based on multi-cascade progressive convolution structure. [J]. Chinese Journal of Liquid Crystals and Displays 38(10):1409-1422(2023) DOI: 10.37188/CJLCD.2022-0383.
雨天图像会影响计算机视觉任务的效果与精度。雨天图像常常包含来自不同方向、大小、形状的雨点或雨痕,在对这些雨点、雨痕进行去除时,现有的方法往往没有考虑到雨天图像不同精细尺度下的特征信息,仅采用单一尺度进行图像去雨存在很大缺陷,无法恢复出足够清晰的视觉任务图像。受益于卷积神经网络架构的强大特征提取能力,本文提出了一种端到端的多级联递进卷积结构算子,该算子包含4层卷积层,通过阶梯化连接构成一个整体模块,该模块可以针对多尺度场景下的雨天进行特征提取并整合。将该算子模块嵌入到渐进循环网络结构中,利用循环结构多次去除雨纹,最终有效还原出接近真实图像的无雨图像。该方法在现有的人工合成雨图数据集Rain100H、Rain100L、Rain800与自动驾驶领域合成雨图数据集BDD1000上进行了对比实验。实验结果表明,该算法在4个数据集上的PSNR值达到了30.70,37.91,27.63,35.74 dB,SSIM值达到了0.914,0.980,0.894,0.977。通过真实雨图数据集去雨结果的可视化展示,充分验证了本文方法在去雨任务上的有效性。
Rainy images can affect the performance and accuracy of computer vision tasks. Rainy images often contain raindrops or rain marks from different directions, sizes, and shapes. When removing these raindrops and rain marks, the existing methods often do not take into account the feature information of rainy images at different fine scales, and only use a single scale. There is a big defect in image deraining, and it is impossible to restore a clear enough image for visual tasks. Therefore, benefiting from the powerful feature extraction capability of the convolutional neural network architecture, an end-to-end multi-cascade progressive convolution structure operator is proposed, which consists of four convolutional layers connected through a ladder to form an overall module. This module can extract and integrate rainy weather features in multi-scale scenes. The operator module is embedded into the progressive recurrent network structure, the recurrent structure is used to remove rain streaks many times, and finally the rain-free image close to the real image is effectively restored. The method is compared with the existing artificially synthesized rain image datasets Rain100H, Rain100L, Rain800 and the synthetic rain image dataset BDD1000 in the field of automatic driving. The experiment results shows that the PSNR values of the algorithm on the four datasets reach 30.70 , 37.91, 27.63, 35.74 dB, and the SSIM values reach 0.914, 0.980, 0.894, 0.977. Through the visual display of the rain removal results of the real rain map dataset, the effectiveness of the method in this paper on the rain removal task is fully verified.
图像去雨多级联递进卷积结构卷积神经网络深度学习多尺度特征残差结构
image rain removalmulti-cascade progressive convolution structureconvolutional neural networkdeep learningmulti-scale featureresidual structure
ZUO C, QIAN J M, FENG S J, et al. Deep learning in optical metrology: a review[J]. Light: Science & Applications, 2022, 11(1): 1-54. doi: 10.1038/s41377-022-00714-xhttp://dx.doi.org/10.1038/s41377-022-00714-x
SITU G H. Deep holography [J]. Light: Advanced Manufacturing, 2022, 3(1): 1-23. doi: 10.37188/lam.2022.013http://dx.doi.org/10.37188/lam.2022.013
LUO Y, ZHAO Y F, Li J X, et al. Computational imaging without a computer: seeing through random diffusers at the speed of light [J]. eLight, 2022, 2(1): 1-16. doi: 10.1186/s43593-022-00012-4http://dx.doi.org/10.1186/s43593-022-00012-4
陈清江, 吴田田. 基于联结残差网络的单幅图像去雨[J]. 液晶与显示, 2021, 36(4):605-614. doi: 10.37188/CJLCD.2020-0173http://dx.doi.org/10.37188/CJLCD.2020-0173
CHEN Q J, WU T T. Single image deraining based on the concatenation residual network [J]. Chinese Journal of Liquid Crystals and Displays, 2021, 36(4):605-614. (in Chinese). doi: 10.37188/CJLCD.2020-0173http://dx.doi.org/10.37188/CJLCD.2020-0173
KANG L W, Lin C W, Fu Y H. Automatic single-image-based rain streaks removal via image decomposition [J]. IEEE Transactions on Image Processing, 2011, 21(4): 1742-1755. doi: 10.1109/tip.2011.2179057http://dx.doi.org/10.1109/tip.2011.2179057
LUO Y, XU Y, JI H. Removing rain from a single image via discriminative sparse coding [C]//Proceedings of the IEEE International Conference on Computer Vision.New York: IEEE, 2015: 3397-3405. doi: 10.1109/iccv.2015.388http://dx.doi.org/10.1109/iccv.2015.388
LI Y, TAN R T, GUO X J, et al. Rain streak removal using layer priors [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 2736-2744. doi: 10.1109/cvpr.2016.299http://dx.doi.org/10.1109/cvpr.2016.299
YANG W H, TAN R T, FENG J S, et al. Deep joint rain detection and removal from a single image [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017: 1357-1366. doi: 10.1109/cvpr.2017.183http://dx.doi.org/10.1109/cvpr.2017.183
LI X, WU J L, LIN Z C, et al. Recurrent squeeze-and-excitation context aggregation net for single image deraining [C]//Proceedings of the European Conference on Computer Vision (ECCV). Munich: ECCV, 2018: 254-269. doi: 10.1007/978-3-030-01234-2_16http://dx.doi.org/10.1007/978-3-030-01234-2_16
REN D W, ZUO W M, HU Q H, et al. Progressive image deraining networks: A better and simpler baseline [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 3937-3946. doi: 10.1109/cvpr.2019.00406http://dx.doi.org/10.1109/cvpr.2019.00406
WANG C, XING X Y, WU Y T, et al. Dcsfn: Deep cross-scale fusion network for single image rain removal [C]//Proceedings of the 28th ACM International Conference on Multimedia. Nice: ACM, 2020: 1643-1651. doi: 10.1145/3394171.3413820http://dx.doi.org/10.1145/3394171.3413820
ZHANG H, SINDAGI V, PATEL V M. Image de-raining using a conditional generative adversarial network [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 30(11): 3943-3956. doi: 10.1109/tcsvt.2019.2920407http://dx.doi.org/10.1109/tcsvt.2019.2920407
WEI W, MENG D Y, ZHAO Q, et al. Semi-supervised transfer learning for image rain removal [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach:IEEE, 2019: 3877-3886. doi: 10.1109/cvpr.2019.00400http://dx.doi.org/10.1109/cvpr.2019.00400
JIN X, CHEN Z B, LIN J X, et al. Unsupervised single image deraining with self-supervised constraints [C]// Proceedings of 2019 IEEE International Conference on Image Processing (ICIP). Taipei, China: IEE, 2019: 2761-2765. doi: 10.1109/icip.2019.8803238http://dx.doi.org/10.1109/icip.2019.8803238
LIU J Y, YANG W H, YANG S, et al. Erase or fill? deep joint recurrent rain removal and reconstruction in videos [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 3233-3242. doi: 10.1109/cvpr.2018.00341http://dx.doi.org/10.1109/cvpr.2018.00341
HU X W, Fu C W, ZHU L, et al. Depth-attentional features for single-image rain removal [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 8022-8031. doi: 10.1109/cvpr.2019.00821http://dx.doi.org/10.1109/cvpr.2019.00821
HUANG G, LIU Z, Van Der Maaten L, et al. Densely connected convolutional networks [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017: 4700-4708. doi: 10.1109/cvpr.2017.243http://dx.doi.org/10.1109/cvpr.2017.243
HUANG G, CHEN D L, LI T H, et al. Multi-scale dense networks for resource efficient image classification [J]. arXiv, 2017:1703.09844.
SUN K, XIAO B, LIU D, et al. Deep high-resolution representation learning for human pose estimation [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 5693-5703. doi: 10.1109/cvpr.2019.00584http://dx.doi.org/10.1109/cvpr.2019.00584
YU F, WANG D, SHELHAMER E, et al. Deep layer aggregation [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2403-2412. doi: 10.1109/cvpr.2018.00255http://dx.doi.org/10.1109/cvpr.2018.00255
LIN M, CHEN Q, YAN S C. Network in network [J/OL]. arXiv, 2013:1312.4400.
GAO S H, CHENG M M, ZHAO K, et al. Res2net: A new multi-scale backbone architecture [J]. IEEE Transactions on pattern analysis and machine intelligence, 2019, 43(2): 652-662.
CHEN Y P, FAN H Q, XU B, et al. Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Long Beach: IEEE, 2019: 3435-3444. doi: 10.1109/iccv.2019.00353http://dx.doi.org/10.1109/iccv.2019.00353
TAN M X, LE Q V. Mixconv: Mixed depthwise convolutional kernels[J/OL]. arXiv, 2019:1907.09595.
DUTA I C, LIU L, ZHU F, et al. Pyramidal convolution: Rethinking convolutional neural networks for visual recognition [J/OL]. arXiv, 2020: 2006.11538.
SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2015: 1-9. doi: 10.1109/cvpr.2015.7298594http://dx.doi.org/10.1109/cvpr.2015.7298594
LI Y, KUANG Z H, CHEN Y M, et al. Data-driven neuron allocation for scale aggregation networks [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 11526-11534. doi: 10.1109/cvpr.2019.01179http://dx.doi.org/10.1109/cvpr.2019.01179
HOCHREITER S, SCHMIDHUBER J. Long short-term memory [J]. Neural Computation, 1997, 9(8): 1735-1780. doi: 10.1162/neco.1997.9.8.1735http://dx.doi.org/10.1162/neco.1997.9.8.1735
ZHOU W, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity [J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. doi: 10.1109/tip.2003.819861http://dx.doi.org/10.1109/tip.2003.819861
汪帆, 魏宪, 郭杰龙, 等. 基于多通道分离整合的多尺度单幅图像去雨算法[J]. 计算机与现代化, 2021 (12): 72. doi: 10.3969/j.issn.1006-2475.2021.12.012http://dx.doi.org/10.3969/j.issn.1006-2475.2021.12.012
WANG F, WEI X, GUO J L, et al. Multi-scale single image rain removal based on multi-channel separation and integration [J]. Computer and Modernization, 2021 (12): 72. (in Chinese). doi: 10.3969/j.issn.1006-2475.2021.12.012http://dx.doi.org/10.3969/j.issn.1006-2475.2021.12.012
FU X Y, HUANG J B, ZENG D L, et al. Removing rain from single images via a deep detail network [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017: 3855-3863. doi: 10.1109/cvpr.2017.186http://dx.doi.org/10.1109/cvpr.2017.186
CUI X, SHANG W, Ren D W, et al. Semi-supervised single image deraining with discrete wavelet transform [C]//Pacific Rim International Conference on Artificial Intelligence. Cham: Springer, 2021: 265-278. doi: 10.1007/978-3-030-89370-5_20http://dx.doi.org/10.1007/978-3-030-89370-5_20
MITTAL A, SOUNDARARAJAN R, BOVIK A C. Making a “completely blind” image quality analyzer [J]. IEEE Signal Processing Letters, 2012, 20(3): 209-212. doi: 10.1109/lsp.2012.2227726http://dx.doi.org/10.1109/lsp.2012.2227726
0
Views
129
下载量
0
CSCD
Publicity Resources
Related Articles
Related Author
Related Institution