基于多尺度细节的孪生卷积神经网络图像融合算法

刘博; 韩广良; 罗惠元

doi:10.37188/CJLCD.2020-0339

您当前的位置：

首页 >

文章列表页 >

基于多尺度细节的孪生卷积神经网络图像融合算法

图像处理 | 更新时间：2021-09-10

- 基于多尺度细节的孪生卷积神经网络图像融合算法
- Image fusion algorithm based on multi-scale detail siamese convolutional neural network
- 液晶与显示 2021年36卷第9期页码：1283-1293
- 作者机构：
  
  1.中国科学院长春光学精密机械与物理研究所, 吉林长春 130033
  2.中国科学院大学, 北京 100049
- 作者简介：
  
  [ "刘博(1993-), 男, 吉林长春人, 硕士研究生, 2016年于长春大学获得学士学位, 主要从事计算机视觉、图像处理与信息处理方面的研究。E-mail: lbciomp@163.com" ]
  [ "韩广良(1968-), 男, 山东嘉祥人, 博士, 研究员, 2003年于中国科学院长春光学精密机构与物理研究所获得博士学位, 研究方向为图像和视频信息处理、计算机视觉、机器视觉和人工智能等。E-mail: hangl@ciomp.ac.cn" ]
- 基金信息：
  
  国家自然科学基金(61401425);国家自然科学基金青年科学基金项目(61602432)
- DOI：10.37188/CJLCD.2020-0339
  中图分类号： TP391.4
- 收稿日期：2020-12-09，
  
  修回日期：2021-01-26，
  
  纸质出版日期：2021-09
- 稿件说明：
移动端阅览
刘博, 韩广良, 罗惠元. 基于多尺度细节的孪生卷积神经网络图像融合算法[J]. 液晶与显示, 2021,36(9):1283-1293.

Bo LIU, Guang-liang HAN, Hui-yuan LUO. Image fusion algorithm based on multi-scale detail siamese convolutional neural network[J]. Chinese journal of liquid crystals and displays, 2021, 36(9): 1283-1293.
刘博, 韩广良, 罗惠元. 基于多尺度细节的孪生卷积神经网络图像融合算法[J]. 液晶与显示, 2021,36(9):1283-1293. DOI： 10.37188/CJLCD.2020-0339.

Bo LIU, Guang-liang HAN, Hui-yuan LUO. Image fusion algorithm based on multi-scale detail siamese convolutional neural network[J]. Chinese journal of liquid crystals and displays, 2021, 36(9): 1283-1293. DOI： 10.37188/CJLCD.2020-0339.

摘要

图像融合将来自不同捕获条件或不同传感器的互补图像进行融合以提高图像的视觉质量。针对这一任务，本文提出一种改进的滚动引导滤波与神经网络相结合的多尺度融合算法。首先，使用孪生卷积神经网络学习图像特征，并以此获得包含源图像显著特征的权值映射图。随后，使用改进的滚动引导滤波对图像进行多尺度分解，结合信息熵使滚动引导滤波权重参数自适应化来实现多尺度自适应分解，并结合非线性映射增强图像细节信息。最后，采用局部能量与权值图相结合的自适应调整融合模式对多尺度图像进行融合。经实验对比，所提方法能够避免出现图像边缘圆晕效应，且能够更好地突出图像边缘、细节纹理特征。另外，与其他算法相比，本文所提出的算法在平均梯度、信息熵、视觉信息保真度以及空间频率等客观评价指标项上均取得了更优的性能表现。

Abstract

Image fusion is to fuse the complementary images under different acquisition conditions or sensors for improving the visual quality of images. For this task

this paper proposes an improved multi-scale fusion algorithm based on the combination of rolling guided filtering and neural network. Firstly

the siamese convolutional neural network is used to learn the image features and further obtain the weight map containing the significant features of the source image. Then

the image is decomposed by the improved rolling guide filter

and the weight parameters of the rolling guide filter are adaptive to achieve multi-scale adaptive decomposition by combining the information entropy

and the details of the image are enhanced by combining the nonlinear mapping. Finally

based on the local energy and weight map

an adaptive fusing strategy is designed to combine the multi-scale images adaptively. The experimental comparison shows that the proposed method can avoid the halo effect of image edge and better highlight the edge and detail texture features of image. In addition

compared with other algorithms

our algorithm can achieve a better performance according to the objective evaluation index items

such as average gradient

information entropy

visual information fidelity and spatial frequency.

关键词

Keywords

references

武治国 , 王延杰 . 一种基于边缘的曲波变换图像融合方法 . 光学技术 , 2009 . 35 ( 5 ): 682 - 685, 690 . DOI: 10.3321/j.issn:1002-1582.2009.05.031 http://doi.org/10.3321/j.issn:1002-1582.2009.05.031 .

Z G WU , Y J WANG . Image fusion algorithm using Curvelet transform based on the edge detection . Optical Technique , 2009 . 35 ( 5 ): 682 - 685, 690 . DOI: 10.3321/j.issn:1002-1582.2009.05.031 http://doi.org/10.3321/j.issn:1002-1582.2009.05.031 .

A TOET . Image fusion by a ratio of low-pass pyramid . Pattern Recognition Letters , 1989 . 9 ( 4 ): 245 - 253 . DOI: 10.1016/0167-8655(89)90003-2 http://doi.org/10.1016/0167-8655(89)90003-2 .

LI M J, ZHOU J F, LI J. Research on image fusion rules based on contrast pyramid[C]//2018 Chinese Automation Congress . Xian, China: IEEE, 2018: 1361-1364.

李建林 , 俞建成 , 孙胜利 . 基于梯度金字塔图像融合的研究 . 科学技术与工程 , 2007 . 7 ( 22 ): 5818 - 5822 . DOI: 10.3969/j.issn.1671-1815.2007.22.018 http://doi.org/10.3969/j.issn.1671-1815.2007.22.018 .

J L LI , J C YU , S L SUN . Study of image fusion based on grad pyramid algorithm . Science Technology and Engineering , 2007 . 7 ( 22 ): 5818 - 5822 . DOI: 10.3969/j.issn.1671-1815.2007.22.018 http://doi.org/10.3969/j.issn.1671-1815.2007.22.018 .

万智萍 . 结合方向小波的多光谱与全色遥感图像融合算法 . 遥感技术与应用 , 2014 . 29 ( 4 ): 660 - 668 . DOI: 10.11873/j.issn.1004-0323.2014.4.0660 http://doi.org/10.11873/j.issn.1004-0323.2014.4.0660 .

Z P WAN . Multi-spectral and panchromatic remote sensing image fusion algorithm based on directional wavelet . Remote Sensing Technology and Application , 2014 . 29 ( 4 ): 660 - 668 . DOI: 10.11873/j.issn.1004-0323.2014.4.0660 http://doi.org/10.11873/j.issn.1004-0323.2014.4.0660 .

陈清江 , 魏冰蔗 , 柴昱洲 , 等 . 非抽样双树复小波域的BPP-NMF图像融合 . 液晶与显示 , 2016 . 31 ( 8 ): 784 - 792 . DOI: 10.3788/YJYXS20163108.0784 http://doi.org/10.3788/YJYXS20163108.0784 .

Q H CHEN , B Z WEI , Y Z CHAI , 等 . Image fusion based on block principal pivoting nonnegative matrix in undecimated dual-tree complex wavelet domain . Chinese Journal of Liquid Crystals and Displays , 2016 . 31 ( 8 ): 784 - 792 . DOI: 10.3788/YJYXS20163108.0784 http://doi.org/10.3788/YJYXS20163108.0784 .

殷明 , 段普宏 , 褚标 , 等 . 基于非下采样双树复轮廓波变换和稀疏表示的红外和可见光图像融合 . 光学精密工程 , 2016 . 24 ( 7 ): 1763 - 1771 . DOI: 10.3788/OPE.20162407.1763 http://doi.org/10.3788/OPE.20162407.1763 .

M YIN , P H DUAN , B CHU , 等 . Fusion of infrared and visible images combined with NSDTCT and sparse representation . Optics and Precision Engineering , 2016 . 24 ( 7 ): 1763 - 1771 . DOI: 10.3788/OPE.20162407.1763 http://doi.org/10.3788/OPE.20162407.1763 .

陈贞 , 邢笑雪 . 基于非下采样Shearlet变换与压缩感知的图像融合 . 液晶与显示 , 2015 . 30 ( 6 ): 1024 - 1031 . DOI: 10.3788/YJYXS20153006.1024 http://doi.org/10.3788/YJYXS20153006.1024 .

Z CHEN , X X XING . Image fusion algorithm based on non-subsampled Shearlet transform and compressed sensing . Chinese Journal of Liquid Crystals and Displays , 2015 . 30 ( 6 ): 1024 - 1031 . DOI: 10.3788/YJYXS20153006.1024 http://doi.org/10.3788/YJYXS20153006.1024 .

CUNHA A L DA , J ZHOU , M N DO . The nonsubsampled contourlet transform: theory, design, and applications . IEEE Transactions on Image Processing , 2006 . 15 ( 10 ): 3089 - 3101 . DOI: 10.1109/TIP.2006.877507 http://doi.org/10.1109/TIP.2006.877507 .

J F ZHAO , H J FENG , Z H XU , 等 . Detail enhanced multi-source fusion using visual weight map extraction based on multi scale edge preserving decomposition . Optics Communications , 2013 . 287 45 - 52 . DOI: 10.1016/j.optcom.2012.08.070 http://doi.org/10.1016/j.optcom.2012.08.070 .

赵程 , 黄永东 . 基于滚动导向滤波和混合多尺度分解的红外与可见光图像融合方法 . 激光与光电子学进展 , 2019 . 56 ( 14 ): 141007 DOI: 10.3788/LOP56.141007 http://doi.org/10.3788/LOP56.141007 .

C ZHAO , Y D HAUNG . Infrared and visible image fusion via rolling guidance filtering and hybrid multi-scale decomposition . Laser & Optoelectronics Progress , 2019 . 56 ( 14 ): 141007 DOI: 10.3788/LOP56.141007 http://doi.org/10.3788/LOP56.141007 .

H LI , X J WU . DenseFuse: a fusion approach to infrared and visible images . IEEE Transactions on Image Processing , 2019 . 28 ( 5 ): 2614 - 2623 . DOI: 10.1109/TIP.2018.2887342 http://doi.org/10.1109/TIP.2018.2887342 .

X C LIANG , P Y HU , L G ZHANG , 等 . MCFNet: multi-layer concatenation fusion network for medical images fusion . IEEE Sensors Journal , 2019 . 19 ( 16 ): 7107 - 7119 . DOI: 10.1109/JSEN.2019.2913281 http://doi.org/10.1109/JSEN.2019.2913281 .

Y LIU , X CHEN , H PENG , 等 . Multi-focus image fusion with a deep convolutional neural network . Information Fusion , 2017 . 36 191 - 207 . DOI: 10.1016/j.inffus.2016.12.001 http://doi.org/10.1016/j.inffus.2016.12.001 .

CHOPRA S, HADSELL R, LECUN Y. Learning a similarity metric discriminatively, with application to face verification[C]// IEEE Computer Society Conference on Computer Vision and Pattern Recognition . San Diego, CA: IEEE, 2005: 539-546.

ZHANG Q, SHEN X Y, XU L, et al . Rolling guidance filter[C]//13 th European Conference on Computer Vision (ECCV) . Zurich, Switzerland: Springer, 2014: 815-830.

S PARIS , S W HASINOFF , J KAUTZ . Local Laplacian filters: edge-aware image processing with a Laplacian pyramid . Communications of the ACM , 2015 . 58 ( 3 ): 81 - 91 . DOI: 10.1145/2723694 http://doi.org/10.1145/2723694 .

G Q HE , S Y XING , X J HE , 等 . Image fusion method based on simultaneous sparse representation with non-subsampled contourlet transform . IET Computer Vision , 2019 . 13 ( 2 ): 240 - 248 . DOI: 10.1049/iet-cvi.2018.5496 http://doi.org/10.1049/iet-cvi.2018.5496 .

赵春晖 , 郭蕴霆 . 一种快速的基于稀疏表示和非下采样轮廓波变换的图像融合算法 . 电子与信息学报 , 2016 . 38 ( 7 ): 1773 - 1780 . DOI: 10.11999/JEIT150933 http://doi.org/10.11999/JEIT150933 .

C H ZHAO , Y T GUO . Fast image fusion algorithm based on sparse representation and non-subsampled contourlet transform . Journal of Electronics & Information Technology , 2016 . 38 ( 7 ): 1773 - 1780 . DOI: 10.11999/JEIT150933 http://doi.org/10.11999/JEIT150933 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于结构纹理分解的红外与可见光图像融合算法

多干扰成像条件下的钻孔内壁图像鲁棒拼接方法

TFT-LCD表面Mura缺陷的AOI检测研究进展

基于OpenCV的双波长光纤导引光在高精度装调上的应用