Image fusion algorithm based on multi-scale detail siamese convolutional neural network

Bo LIU; Guang-liang HAN; Hui-yuan LUO

doi:10.37188/CJLCD.2020-0339

您当前的位置：

首页 >

文章列表页 >

Image fusion algorithm based on multi-scale detail siamese convolutional neural network

Image Processing | 更新时间：2021-09-10

- Image fusion algorithm based on multi-scale detail siamese convolutional neural network
- Chinese Journal of Liquid Crystals and Displays Vol. 36, Issue 9, Pages: 1283-1293(2021)
- 作者机构：
  
  1.中国科学院长春光学精密机械与物理研究所, 吉林长春 130033
  2.中国科学院大学, 北京 100049
- 作者简介：
- 基金信息：
- DOI：10.37188/CJLCD.2020-0339
  CLC： TP391.4
- Received：09 December 2020，
  
  Revised：26 January 2021，
  
  Published：2021-09
- 稿件说明：
移动端阅览
Bo LIU, Guang-liang HAN, Hui-yuan LUO. Image fusion algorithm based on multi-scale detail siamese convolutional neural network[J]. Chinese journal of liquid crystals and displays, 2021, 36(9): 1283-1293.
DOI：

Bo LIU, Guang-liang HAN, Hui-yuan LUO. Image fusion algorithm based on multi-scale detail siamese convolutional neural network[J]. Chinese journal of liquid crystals and displays, 2021, 36(9): 1283-1293. DOI： 10.37188/CJLCD.2020-0339.

摘要

图像融合将来自不同捕获条件或不同传感器的互补图像进行融合以提高图像的视觉质量。针对这一任务，本文提出一种改进的滚动引导滤波与神经网络相结合的多尺度融合算法。首先，使用孪生卷积神经网络学习图像特征，并以此获得包含源图像显著特征的权值映射图。随后，使用改进的滚动引导滤波对图像进行多尺度分解，结合信息熵使滚动引导滤波权重参数自适应化来实现多尺度自适应分解，并结合非线性映射增强图像细节信息。最后，采用局部能量与权值图相结合的自适应调整融合模式对多尺度图像进行融合。经实验对比，所提方法能够避免出现图像边缘圆晕效应，且能够更好地突出图像边缘、细节纹理特征。另外，与其他算法相比，本文所提出的算法在平均梯度、信息熵、视觉信息保真度以及空间频率等客观评价指标项上均取得了更优的性能表现。

Abstract

Image fusion is to fuse the complementary images under different acquisition conditions or sensors for improving the visual quality of images. For this task

this paper proposes an improved multi-scale fusion algorithm based on the combination of rolling guided filtering and neural network. Firstly

the siamese convolutional neural network is used to learn the image features and further obtain the weight map containing the significant features of the source image. Then

the image is decomposed by the improved rolling guide filter

and the weight parameters of the rolling guide filter are adaptive to achieve multi-scale adaptive decomposition by combining the information entropy

and the details of the image are enhanced by combining the nonlinear mapping. Finally

based on the local energy and weight map

an adaptive fusing strategy is designed to combine the multi-scale images adaptively. The experimental comparison shows that the proposed method can avoid the halo effect of image edge and better highlight the edge and detail texture features of image. In addition

compared with other algorithms

our algorithm can achieve a better performance according to the objective evaluation index items

such as average gradient

information entropy

visual information fidelity and spatial frequency.

关键词

Keywords

references

武治国 , 王延杰 . 一种基于边缘的曲波变换图像融合方法 . 光学技术 , 2009 . 35 ( 5 ): 682 - 685, 690 . DOI: 10.3321/j.issn:1002-1582.2009.05.031 http://doi.org/10.3321/j.issn:1002-1582.2009.05.031 .

Z G WU , Y J WANG . Image fusion algorithm using Curvelet transform based on the edge detection . Optical Technique , 2009 . 35 ( 5 ): 682 - 685, 690 . DOI: 10.3321/j.issn:1002-1582.2009.05.031 http://doi.org/10.3321/j.issn:1002-1582.2009.05.031 .

A TOET . Image fusion by a ratio of low-pass pyramid . Pattern Recognition Letters , 1989 . 9 ( 4 ): 245 - 253 . DOI: 10.1016/0167-8655(89)90003-2 http://doi.org/10.1016/0167-8655(89)90003-2 .

LI M J, ZHOU J F, LI J. Research on image fusion rules based on contrast pyramid[C]//2018 Chinese Automation Congress . Xian, China: IEEE, 2018: 1361-1364.

李建林 , 俞建成 , 孙胜利 . 基于梯度金字塔图像融合的研究 . 科学技术与工程 , 2007 . 7 ( 22 ): 5818 - 5822 . DOI: 10.3969/j.issn.1671-1815.2007.22.018 http://doi.org/10.3969/j.issn.1671-1815.2007.22.018 .

J L LI , J C YU , S L SUN . Study of image fusion based on grad pyramid algorithm . Science Technology and Engineering , 2007 . 7 ( 22 ): 5818 - 5822 . DOI: 10.3969/j.issn.1671-1815.2007.22.018 http://doi.org/10.3969/j.issn.1671-1815.2007.22.018 .

万智萍 . 结合方向小波的多光谱与全色遥感图像融合算法 . 遥感技术与应用 , 2014 . 29 ( 4 ): 660 - 668 . DOI: 10.11873/j.issn.1004-0323.2014.4.0660 http://doi.org/10.11873/j.issn.1004-0323.2014.4.0660 .

Z P WAN . Multi-spectral and panchromatic remote sensing image fusion algorithm based on directional wavelet . Remote Sensing Technology and Application , 2014 . 29 ( 4 ): 660 - 668 . DOI: 10.11873/j.issn.1004-0323.2014.4.0660 http://doi.org/10.11873/j.issn.1004-0323.2014.4.0660 .

陈清江 , 魏冰蔗 , 柴昱洲 , 等 . 非抽样双树复小波域的BPP-NMF图像融合 . 液晶与显示 , 2016 . 31 ( 8 ): 784 - 792 . DOI: 10.3788/YJYXS20163108.0784 http://doi.org/10.3788/YJYXS20163108.0784 .

Q H CHEN , B Z WEI , Y Z CHAI , 等 . Image fusion based on block principal pivoting nonnegative matrix in undecimated dual-tree complex wavelet domain . Chinese Journal of Liquid Crystals and Displays , 2016 . 31 ( 8 ): 784 - 792 . DOI: 10.3788/YJYXS20163108.0784 http://doi.org/10.3788/YJYXS20163108.0784 .

殷明 , 段普宏 , 褚标 , 等 . 基于非下采样双树复轮廓波变换和稀疏表示的红外和可见光图像融合 . 光学精密工程 , 2016 . 24 ( 7 ): 1763 - 1771 . DOI: 10.3788/OPE.20162407.1763 http://doi.org/10.3788/OPE.20162407.1763 .

M YIN , P H DUAN , B CHU , 等 . Fusion of infrared and visible images combined with NSDTCT and sparse representation . Optics and Precision Engineering , 2016 . 24 ( 7 ): 1763 - 1771 . DOI: 10.3788/OPE.20162407.1763 http://doi.org/10.3788/OPE.20162407.1763 .

陈贞 , 邢笑雪 . 基于非下采样Shearlet变换与压缩感知的图像融合 . 液晶与显示 , 2015 . 30 ( 6 ): 1024 - 1031 . DOI: 10.3788/YJYXS20153006.1024 http://doi.org/10.3788/YJYXS20153006.1024 .

Z CHEN , X X XING . Image fusion algorithm based on non-subsampled Shearlet transform and compressed sensing . Chinese Journal of Liquid Crystals and Displays , 2015 . 30 ( 6 ): 1024 - 1031 . DOI: 10.3788/YJYXS20153006.1024 http://doi.org/10.3788/YJYXS20153006.1024 .

CUNHA A L DA , J ZHOU , M N DO . The nonsubsampled contourlet transform: theory, design, and applications . IEEE Transactions on Image Processing , 2006 . 15 ( 10 ): 3089 - 3101 . DOI: 10.1109/TIP.2006.877507 http://doi.org/10.1109/TIP.2006.877507 .

J F ZHAO , H J FENG , Z H XU , 等 . Detail enhanced multi-source fusion using visual weight map extraction based on multi scale edge preserving decomposition . Optics Communications , 2013 . 287 45 - 52 . DOI: 10.1016/j.optcom.2012.08.070 http://doi.org/10.1016/j.optcom.2012.08.070 .

赵程 , 黄永东 . 基于滚动导向滤波和混合多尺度分解的红外与可见光图像融合方法 . 激光与光电子学进展 , 2019 . 56 ( 14 ): 141007 DOI: 10.3788/LOP56.141007 http://doi.org/10.3788/LOP56.141007 .

C ZHAO , Y D HAUNG . Infrared and visible image fusion via rolling guidance filtering and hybrid multi-scale decomposition . Laser & Optoelectronics Progress , 2019 . 56 ( 14 ): 141007 DOI: 10.3788/LOP56.141007 http://doi.org/10.3788/LOP56.141007 .

H LI , X J WU . DenseFuse: a fusion approach to infrared and visible images . IEEE Transactions on Image Processing , 2019 . 28 ( 5 ): 2614 - 2623 . DOI: 10.1109/TIP.2018.2887342 http://doi.org/10.1109/TIP.2018.2887342 .

X C LIANG , P Y HU , L G ZHANG , 等 . MCFNet: multi-layer concatenation fusion network for medical images fusion . IEEE Sensors Journal , 2019 . 19 ( 16 ): 7107 - 7119 . DOI: 10.1109/JSEN.2019.2913281 http://doi.org/10.1109/JSEN.2019.2913281 .

Y LIU , X CHEN , H PENG , 等 . Multi-focus image fusion with a deep convolutional neural network . Information Fusion , 2017 . 36 191 - 207 . DOI: 10.1016/j.inffus.2016.12.001 http://doi.org/10.1016/j.inffus.2016.12.001 .

CHOPRA S, HADSELL R, LECUN Y. Learning a similarity metric discriminatively, with application to face verification[C]// IEEE Computer Society Conference on Computer Vision and Pattern Recognition . San Diego, CA: IEEE, 2005: 539-546.

ZHANG Q, SHEN X Y, XU L, et al . Rolling guidance filter[C]//13 th European Conference on Computer Vision (ECCV) . Zurich, Switzerland: Springer, 2014: 815-830.

S PARIS , S W HASINOFF , J KAUTZ . Local Laplacian filters: edge-aware image processing with a Laplacian pyramid . Communications of the ACM , 2015 . 58 ( 3 ): 81 - 91 . DOI: 10.1145/2723694 http://doi.org/10.1145/2723694 .

G Q HE , S Y XING , X J HE , 等 . Image fusion method based on simultaneous sparse representation with non-subsampled contourlet transform . IET Computer Vision , 2019 . 13 ( 2 ): 240 - 248 . DOI: 10.1049/iet-cvi.2018.5496 http://doi.org/10.1049/iet-cvi.2018.5496 .

赵春晖 , 郭蕴霆 . 一种快速的基于稀疏表示和非下采样轮廓波变换的图像融合算法 . 电子与信息学报 , 2016 . 38 ( 7 ): 1773 - 1780 . DOI: 10.11999/JEIT150933 http://doi.org/10.11999/JEIT150933 .

C H ZHAO , Y T GUO . Fast image fusion algorithm based on sparse representation and non-subsampled contourlet transform . Journal of Electronics & Information Technology , 2016 . 38 ( 7 ): 1773 - 1780 . DOI: 10.11999/JEIT150933 http://doi.org/10.11999/JEIT150933 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Infrared and visible image fusion algorithm based on structure-texture decomposition

Robust stitching method for borehole wall images under multi-interference imaging conditions

Development of AOI inspection of Mura defects on TFT-LCD surface

Application of dual-wavelength fiber guided light based on OpenCV in high-precision assembly and alignment

Related Author

LI Qing-song

YANG Shen

WU Jin

HUANG Ze-feng

LI Qing-song

YANG Shen

HUANG Ze-feng

黄泽丰

Related Institution

School of Information Science and Engineering， Wuhan University of Science and Technology

College of Engineering， Shantou University

Guangdong Provincial Key Laboratory of Automotive Display and Touch Technologies

College of Mechanical and Electrical Engineering， Hainan University

Key Laboratory of Chemical Lasers， Dalian Institute of Chemical Physics， Chinese Academy of Sciences

AI问答

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China 130033 Postal code：130033
Tel：0431-86176059 Email：yjxs@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰