No reference image quality assessment based onfusion of multiple features and convolutional neural network

Peng LU; Kai-yun LIU; Guo-liang ZOU; Zhen-hua WANG; Zong-sheng ZHENG

doi:10.37188/CJLCD.2021-0175

您当前的位置：

首页 >

文章列表页 >

No reference image quality assessment based onfusion of multiple features and convolutional neural network

Image Processing | 更新时间：2022-01-19

- No reference image quality assessment based onfusion of multiple features and convolutional neural network
- Chinese Journal of Liquid Crystals and Displays Vol. 37, Issue 1, Pages: 66-76(2022)
- 作者机构：
  
  上海海洋大学信息学院, 上海 201306
- 作者简介：
- 基金信息：
  
  Shanghai Science and Technology Innovation Program(20dz203800);Shanghai Municipal Local Capacity Building Project(19050502100);Science and Technology Development Special Project of Shanghai Ocean University(A2-2006-20-200211)
- DOI：10.37188/CJLCD.2021-0175
  CLC： TP391.41
- Received：02 July 2021，
  
  Revised：06 September 2021，
  
  Published：2022-01
- 稿件说明：
移动端阅览
Peng LU, Kai-yun LIU, Guo-liang ZOU, et al. No reference image quality assessment based onfusion of multiple features and convolutional neural network[J]. Chinese journal of liquid crystals and displays, 2022, 37(1): 66-76.
DOI：

Peng LU, Kai-yun LIU, Guo-liang ZOU, et al. No reference image quality assessment based onfusion of multiple features and convolutional neural network[J]. Chinese journal of liquid crystals and displays, 2022, 37(1): 66-76. DOI： 10.37188/CJLCD.2021-0175.

摘要

为了更好地评价图像质量，解决在基于卷积神经网络的图像质量评价模型(CNN-IQA)上明显忽略的分块图像之间存在差异性的问题，提出了一种多特征融合的CNN模型。首先，将整幅图像进行不重叠分块，并提取每个分块图像的信息熵和纹理特征。然后，将提取计算的两特征相结合，计算各分块图像的重要性权重，以此衡量分块图像对失真图像质量的影响。最后，根据计算出的重要性权重修改损失函数，突出重要性高的分块图像在训练过程中发挥的作用。在LIVE数据集进行验证和对比实验发现，该算法的SROCC与LCC指标为0.962和0.960，相比原算法至少提升0.9%；在TID2008数据集上验证和对比实验发现，该算法获得的SROCC与LCC指标为0.922和0.926，相比原算法至少提升0.6%。并且在两数据集上的结果均优于其他对比算法。实验结果证明了其在预测图像质量方面具有良好的性能和泛化性。

Abstract

In order to evaluate the image quality better and solve the problem of the differences between the block images that are obviously ignored in the image quality evaluation model based on the convolutional neural network (CNN-IQA)

a CNN model of multi-feature fusion is proposed. Firstly

the whole image is divided into non-overlapping blocks

the information entropy and texture feature of each divided image are extracted. Then

the two features are combined to calculate the importance weight of each block image to measure the influence of the block image on the quality of the distorted image. Finally

the loss function is modified according to the calculated importance weight to highlight the role of the block image with high importance in the training process. Validation and comparison experiments on the LIVE data set found that the SROCC and LCC indicators of the algorithm are 0.962 and 0.960

which are higher than the original algorithm at least 0.9%. The validation and comparison experiments on the TID2008 data set show that the SROCC and LCC indicators obtained by the algorithm are 0.922 and 0.924

which are higher than the original algorithm at least 0.6%. And the results on the two data sets are better than other comparison algorithms. The experimental results prove that it has good performance and generalization in predicting image quality.

关键词

Keywords

references

方玉明 , 眭相杰 , 鄢杰斌 , 等 . 无参考图像质量评价研究进展 . 中国图象图形学报 , 2021 . 26 ( 2 ): 265 - 286 . https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB202102003.htm https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB202102003.htm , .

Y M FANG , X J SUI , J B YAN , 等 . Progress in noreference image quality assessment . Journal of Image and Graphics , 2021 . 26 ( 2 ): 265 - 286 . https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB202102003.htm https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB202102003.htm , .

周文舟 , 范晨 , 胡小平 , 等 . 多尺度奇异值分解的偏振图像融合去雾算法与实验 . 中国光学 , 2021 . 14 ( 2 ): 298 - 306 . https://www.cnki.com.cn/Article/CJFDTOTAL-ZGGA202102009.htm https://www.cnki.com.cn/Article/CJFDTOTAL-ZGGA202102009.htm , .

W Z ZHOU , C FAN , X P HU , 等 . Polarization image fusion defogging algorithm and experiment based on multiscale singular value decomposition . Chinese Optics , 2021 . 14 ( 2 ): 298 - 306 . https://www.cnki.com.cn/Article/CJFDTOTAL-ZGGA202102009.htm https://www.cnki.com.cn/Article/CJFDTOTAL-ZGGA202102009.htm , .

张申华 , 杨延西 , 秦峤孟 . 针对光栅图像的快速盲去噪方法 . 中国光学 , 2021 . 14 ( 3 ): 596 - 604 . https://www.cnki.com.cn/Article/CJFDTOTAL-ZGGA202103020.htm https://www.cnki.com.cn/Article/CJFDTOTAL-ZGGA202103020.htm , .

S H ZHANG , Y X YANG , Q M QIN . A fast blind denoising method for grating image . Chinese Optics , 2021 . 14 ( 3 ): 596 - 604 . https://www.cnki.com.cn/Article/CJFDTOTAL-ZGGA202103020.htm https://www.cnki.com.cn/Article/CJFDTOTAL-ZGGA202103020.htm , .

王春哲 , 安军社 , 姜秀杰 , 等 . 基于稀疏表示的SOM多失真图像质量评价方法 . 液晶与显示 , 2018 . 33 ( 10 ): 877 - 883 . DOI: 10.3788/YJYXS20183310.0877 http://doi.org/10.3788/YJYXS20183310.0877 .

C Z WANG , J S AN , X J JIANG , 等 . Multidistorted image quality assessment algorithm based on sparse representation and SOM . Chinese Journal of Liquid Crystals and Displays , 2018 . 33 ( 10 ): 877 - 883 . DOI: 10.3788/YJYXS20183310.0877 http://doi.org/10.3788/YJYXS20183310.0877 .

Z WANG , A C BOVIK , H R SHEIKH , 等 . Image quality assessment: from error visibility to structural similarity . IEEE Transactions on Image Processing , 2004 . 13 ( 4 ): 600 - 612 . DOI: 10.1109/TIP.2003.819861 http://doi.org/10.1109/TIP.2003.819861 .

L ZHANG , L ZHANG , X Q MOU , 等 . FSIM: a feature similarity index for image quality assessment . IEEE Transactions on Image Processing , 2011 . 20 ( 8 ): 2378 - 2386 . DOI: 10.1109/TIP.2011.2109730 http://doi.org/10.1109/TIP.2011.2109730 .

H R SHEIKH , A C BOVIK . Image information and visual quality . IEEE Transactions on Image Processing , 2006 . 15 ( 2 ): 430 - 444 . DOI: 10.1109/TIP.2005.859378 http://doi.org/10.1109/TIP.2005.859378 .

张薇 , 蔡浩 , 胡波 , 等 . 基于解析稀疏表示的图像模糊无参考快速评价算法 . 计算机应用研究 , 2016 . 33 ( 10 ): 3169 - 3172,3190 . DOI: 10.3969/j.issn.1001-3695.2016.10.066 http://doi.org/10.3969/j.issn.1001-3695.2016.10.066 .

W ZHANG , H CAI , B HU , 等 . Fast noreference image blur assessment based on analysis sparse representation . Application Research of Computers , 2016 . 33 ( 10 ): 3169 - 3172,3190 . DOI: 10.3969/j.issn.1001-3695.2016.10.066 http://doi.org/10.3969/j.issn.1001-3695.2016.10.066 .

R A MANAP , L SHAO . Nondistortionspecific noreference image quality assessment: a survey . Information Sciences , 2015 . 301 141 - 160 . DOI: 10.1016/j.ins.2014.12.055 http://doi.org/10.1016/j.ins.2014.12.055 .

A MITTAL , A K MOORTHY , A C BOVIK . Noreference image quality assessment in the spatial domain . IEEE Transactions on Image Processing , 2012 . 21 ( 12 ): 4695 - 4708 . DOI: 10.1109/TIP.2012.2214050 http://doi.org/10.1109/TIP.2012.2214050 .

余伟 , 徐晶晶 , 刘玉英 , 等 . 基于自然场景统计的色域映射图像无参考质量评价 . 激光与光电子学进展 , 2020 . 57 ( 14 ): 141006 https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202014007.htm https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202014007.htm , .

W YU , J J XU , Y Y LIU , 等 . Noreference quality evaluation for gamut mapping images based on natural scene statistics . Laser & Optoelectronics Progress , 2020 . 57 ( 14 ): 141006 https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202014007.htm https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202014007.htm , .

S BIANCO , L CELONA , P NAPOLETANO , 等 . On the use of deep learning for blind image quality assessment . Signal, Image and Video Processing , 2018 . 12 ( 2 ): 355 - 362 . DOI: 10.1007/s11760-017-1166-8 http://doi.org/10.1007/s11760-017-1166-8 .

叶雨琪 , 胡文瑾 . 基于多特征的彩色唐卡修复图像无参考质量评价方法 . 激光与光电子学进展 , 2020 . 57 ( 8 ): 081 - 105 . https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202008031.htm https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202008031.htm , .

Y Q YE , W J HU . Noreference quality assessment method for inpainting thangka image based on multiple features . Laser & Optoelectronics Progress , 2020 . 57 ( 8 ): 081 - 105 . https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202008031.htm https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202008031.htm , .

K F FAN , J Y LIANG , F LI , 等 . CNN based noreference hdr image quality assessment . Chinese Journal of Electronics , 2021 . 30 ( 2 ): 282 - 288 . DOI: 10.1049/cje.2021.01.008 http://doi.org/10.1049/cje.2021.01.008 .

KANG L, YE P, LI Y, et al . Convolutional neural networks for noreference image quality assessment [C]. //2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus: IEEE, 2014: 1733-1740.

王俊平 , 李锦 . 图像对比度增强研究的进展 . 电子科技 , 2013 . 26 ( 5 ): 160 - 165 . DOI: 10.3969/j.issn.1007-7820.2013.05.049 http://doi.org/10.3969/j.issn.1007-7820.2013.05.049 .

J P WANG , J LI . Development and prospect of image contrast enhancement . Electronic Science and Technology , 2013 . 26 ( 5 ): 160 - 165 . DOI: 10.3969/j.issn.1007-7820.2013.05.049 http://doi.org/10.3969/j.issn.1007-7820.2013.05.049 .

初录 , 刘斌 , 许亮 , 等 . 基于灰度共生矩阵的散斑质量评价 . 激光与光电子学进展 , 2021 . 58 ( 4 ): 0410019 https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202104028.htm https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202104028.htm , .

L CHU , B LIU , L XU , 等 . Speckle quality evaluation based on gray level cooccurrence matrix . Laser & Optoelectronics Progress , 2021 . 58 ( 4 ): 0410019 https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202104028.htm https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202104028.htm , .

X H JIANG , L Q SHEN , G R FENG , 等 . An optimized CNNbased quality assessment model for screen content image . Signal Processing: Image Communication , 2021 . 94 116 - 181 . http://www.sciencedirect.com/science/article/pii/S0923596521000291 http://www.sciencedirect.com/science/article/pii/S0923596521000291 , .

R M HARALICK , K SHANMUGAM , I DINSTEIN . Textural features for image classification . IEEE Transactions on Systems, Man, and Cybernetics , 1973 . SMC3 ( 6 ): 610 - 621 . http://ieeexplore.ieee.org/document/4309314 http://ieeexplore.ieee.org/document/4309314 , .

全永奇 , 李太君 , 邓家先 , 等 . 模糊集与非线性增益相结合的自适应图像增强算法 . 计算机应用研究 , 2016 . 33 ( 1 ): 311 - 315 . DOI: 10.3969/j.issn.1001-3695.2016.01.072 http://doi.org/10.3969/j.issn.1001-3695.2016.01.072 .

Y Q QUAN , T J LI , J X DENG , 等 . Image enhancement algorithm based on fuzzy sets and nonlinear gain . Application Research of Computers , 2016 . 33 ( 1 ): 311 - 315 . DOI: 10.3969/j.issn.1001-3695.2016.01.072 http://doi.org/10.3969/j.issn.1001-3695.2016.01.072 .

郑加苏. 基于图像信息熵的无参考图像质量评估算法的研究[D]. 北京: 北京交通大学, 2015.

ZHENG J S. An image information entropybased algorithm of noreference image quality assessment [D]. Beijing: Beijing Jiaotong University, 2015. (in Chinese)

卢鹏 , 林根巧 , 邹国良 . 基于信息熵和深度学习的无参考图像质量评价方法研究 . 计算机应用研究 , 2018 . 35 ( 11 ): 3508 - 3512 . DOI: 10.3969/j.issn.1001-3695.2018.11.074 http://doi.org/10.3969/j.issn.1001-3695.2018.11.074 .

P LU , G Q LIN , G L ZOU . Research on noreference image quality evaluation method based on information entropy and deep learning . Application Research of Computers , 2018 . 35 ( 11 ): 3508 - 3512 . DOI: 10.3969/j.issn.1001-3695.2018.11.074 http://doi.org/10.3969/j.issn.1001-3695.2018.11.074 .

E P SIMONCELLI , W T FREEMAN , E H ADELSON , 等 . Shiftable multiscale transforms . IEEE Transactions on Information Theory , 1992 . 38 ( 2 ): 587 - 607 . DOI: 10.1109/18.119725 http://doi.org/10.1109/18.119725 .

杨光义 , 叶凌轩 , 龙敏义 , 等 . 基于孪生网络的全参考彩色图像质量评价算法 . 武汉大学学报(理学版) , 2020 . 66 ( 6 ): 589 - 596 . https://www.cnki.com.cn/Article/CJFDTOTAL-WHDY202006009.htm https://www.cnki.com.cn/Article/CJFDTOTAL-WHDY202006009.htm , .

G Y YANG , L X YE , M Y LONG , 等 . Fullreference color image quality assessment algorithm based on siamese network . Journal of Wuhan University (Natural Science Edition) , 2020 . 66 ( 6 ): 589 - 596 . https://www.cnki.com.cn/Article/CJFDTOTAL-WHDY202006009.htm https://www.cnki.com.cn/Article/CJFDTOTAL-WHDY202006009.htm , .

YE P, KUMAR J, KANG L, et al . Unsupervised feature learning framework for noreference image quality assessment [C]. //2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence: IEEE, 2012: 1098-1105.

MITTAL A, MOORTHY A K, BOVIK A C. Blind/referenceless image spatial quality evaluator [C]. //2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR). Pacific Grove: IEEE, 2011: 723-727.

A K MOORTHY , A C BOVIK . Blind image quality assessment: from natural scene statistics to perceptual quality . IEEE Transactions on Image Processing , 2011 . 20 ( 12 ): 3350 - 3364 . DOI: 10.1109/TIP.2011.2147325 http://doi.org/10.1109/TIP.2011.2147325 .

高方远 , 何立火 . 基于深度网络和视觉特性的无参考图像质量评价方法 . 南京师大学报(自然科学版) , 2019 . 42 ( 3 ): 20 - 26 . DOI: 10.3969/j.issn.1001-4616.2019.03.003 http://doi.org/10.3969/j.issn.1001-4616.2019.03.003 .

F Y GAO , L H HE . Noreference image quality assessment based on deep network and visual characteristics . Journal of Nanjing Normal University (Natural Science Edition) , 2019 . 42 ( 3 ): 20 - 26 . DOI: 10.3969/j.issn.1001-4616.2019.03.003 http://doi.org/10.3969/j.issn.1001-4616.2019.03.003 .

Y MA , X B CAI , F M SUN . Towards noreference image quality assessment based on multiscale convolutional neural network . Computer Modeling in Engineering & Sciences , 2020 . 123 ( 1 ): 201 - 216 . http://www.ingentaconnect.com/content/tsp/cmes/2020/00000123/00000001/art00009 http://www.ingentaconnect.com/content/tsp/cmes/2020/00000123/00000001/art00009 , .

X B GAO , F GAO , D C TAO , 等 . Universal blind image quality assessment metrics via natural scene statistics and multiple kernel learning . IEEE Transactions on Neural Networks and Learning Systems , 2013 . 24 ( 12 ): 2013 - 2026 . DOI: 10.1109/TNNLS.2013.2271356 http://doi.org/10.1109/TNNLS.2013.2271356 .

Views

122

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Design of heterogeneous FPGA hardware accelerator based on CNN

Development of AOI inspection of Mura defects on TFT-LCD surface

Application of dual-wavelength fiber guided light based on OpenCV in high-precision assembly and alignment

Design and implementation of piecewise approximation bilateral filtering algorithm based on FPGA

Related Author

Peng LU

Kai-yun LIU

Guo-liang ZOU

Zhen-hua WANG

Zong-sheng ZHENG

JI Haolin

XU Wei

PIAO Yongjie

Related Institution

College of Information， Shanghai Ocean University

Changchun Institute of Optics， Fine Mechanics and Physics， Chinese Academy of Sciences

University of Chinese Academy of Sciences

Key Laboratory of Space-based Dynamic & Rapid Optical Imaging Technology， Chinese Academy of Sciences

College of Engineering， Shantou University

AI问答

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China 130033 Postal code：130033
Tel：0431-86176059 Email：yjxs@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰