Hybrid distortion image correction method based on improved U-Net networks

SONG Wei; SHI Li-biao; GENG Li-jia; MA Zhen-ling; DU Yan-ling

doi:10.37188/CJLCD.2022-0387

您当前的位置：

首页 >

文章列表页 >

Hybrid distortion image correction method based on improved U-Net networks

Image Enhancement | 更新时间：2023-11-03

- Hybrid distortion image correction method based on improved U-Net networks
- Chinese Journal of Liquid Crystals and Displays Vol. 38, Issue 11, Pages: 1580-1589(2023)
- 作者机构：
  
  1.上海海洋大学信息学院，上海 201306
  2.国家海洋局东海标准计量中心，上海 201306
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(42101443;61972240)
- DOI：10.37188/CJLCD.2022-0387
  CLC： TP391
- Received：21 November 2022，
  
  Revised：15 January 2023，
  
  Published：05 November 2023
- 稿件说明：
移动端阅览
SONG Wei, SHI Li-biao, GENG Li-jia, et al. Hybrid distortion image correction method based on improved U-Net networks[J]. Chinese journal of liquid crystals and displays, 2023, 38(11): 1580-1589.
DOI：

SONG Wei, SHI Li-biao, GENG Li-jia, et al. Hybrid distortion image correction method based on improved U-Net networks[J]. Chinese journal of liquid crystals and displays, 2023, 38(11): 1580-1589. DOI： 10.37188/CJLCD.2022-0387.

摘要

图像几何畸变校正是许多计算机视觉应用的关键预处理步骤。当前基于深度学习的几何畸变校正方法主要解决图像的单一畸变校正问题，为此，本文提出一种改进U-Net网络的图像混合畸变校正方法。首先，提出了一种构建混合畸变的图像数据集的方法，解决了训练数据集稀缺以及畸变类型单一的问题。其次，采用U-Net网络结合空间注意力机制进行图像特征提取及畸变坐标图的重建，将图像校正问题转为畸变图像的逐像素点坐标位移变化的预测问题，并设计了结合坐标差损失和图像重采样损失的损失函数，有效提高校正的准确性。最后，通过消融实验验证了本文方法各模块的性能。与最新基于深度学习的畸变校正方法对比。实验结果表明，本文方法在定量指标和主观评价方面都有较好的表现，对畸变图像的空间坐标校正的平均绝对误差为0.251 9。本文还对GoPro相机获取的光学影像开展了校正实验，进一步验证了本文方法对畸变图像校正的有效性。

Abstract

Image geometric distortion correction is a key pre-processing step for many computer vision applications. Current geometric distortion correction methods based on deep learning mainly solve the problems of single distortion correction of images. For this reason， this paper proposes a hybrid distortion correction method for images with improved U-Net networks. Firstly， a method is proposed to construct hybrid distortion image datasets， which solves the problems of sparse training dataset and single distortion type. Secondly， the U-Net with spatial attention for image feature extraction and reconstruction of the distortion coordinate map is used to turn the image correction problem into a prediction problem of the pixel-by-pixel coordinate displacement change of the distorted image， and a loss function combining coordinate difference loss and image resampling loss is designed to effectively improve the correction accuracy. Finally， the performance of each module of the method in this paper is verified by ablation experiments. Compared with the latest deep learning-based distortion correction methods， the experimental results show that the method in this paper has better performance in terms of quantitative indexes and subjective evaluation， and reaches 0.251 9 of the mean absolute error for coordinate correction of distorted images. Calibration experiments on the optical images acquired by GoPro cameras have further verified the effectiveness of the proposed method in practice.

关键词

Keywords

references

陈苑锋 . 视觉深度估计与点云建图研究进展［J］. 液晶与显示， 2021 ， 36 （ 6 ）： 896 - 911 ． doi: 10.37188/CJLCD.2020-0047 http://dx.doi.org/10.37188/CJLCD.2020-0047

CHEN Y F . Progress of visual depth estimation and point cloud mapping ［J］. Chinese Journal of Liquid Crystals and Displays ， 2021 ， 36 （ 6 ）： 896 - 911 . （in Chinese） . doi: 10.37188/CJLCD.2020-0047 http://dx.doi.org/10.37188/CJLCD.2020-0047

LEE Y G ， TANG Z ， HWANG J N . Online-learning-based human tracking across non-overlapping cameras ［J］. IEEE Transactions on Circuits and Systems for Video Technology ， 2018 ， 28 （ 10 ）： 2870 - 2883 . doi: 10.1109/tcsvt.2017.2707399 http://dx.doi.org/10.1109/tcsvt.2017.2707399

LI G B ， GAN Y K ， WU H J ， et al . Cross-modal attentional context learning for RGB-D object detection ［J］. IEEE Transactions on Image Processing ， 2019 ， 28 （ 4 ）： 1591 - 1601 . doi: 10.1109/tip.2018.2878956 http://dx.doi.org/10.1109/tip.2018.2878956

ZHANG Z Y . A flexible new technique for camera calibration ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence ， 2000 ， 22 （ 11 ）： 1330 - 1334 . doi: 10.1109/34.888718 http://dx.doi.org/10.1109/34.888718

MA Z L ， WANG J L ， CUI C C ， et al . Geometric distortion correction for the underwater images ［C］. OCEANS 2018 MTS/IEEE Charleston. Charleston ： IEEE ， 2018 ： 1 - 5 . doi: 10.1109/oceans.2018.8604579 http://dx.doi.org/10.1109/oceans.2018.8604579

TSAI R . A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses ［J］. IEEE Journal on Robotics and Automation ， 1987 ， 3 （ 4 ）： 323 - 344 . doi: 10.1109/jra.1987.1087109 http://dx.doi.org/10.1109/jra.1987.1087109

SUN J ， WANG H P ， ZHU X L . A fast underwater calibration method based on vanishing point optimization of two orthogonal parallel lines ［J］. Measurement ， 2021 ， 178 ： 109305 . doi: 10.1016/j.measurement.2021.109305 http://dx.doi.org/10.1016/j.measurement.2021.109305

WORKMAN S ， ZHAI M H ， JACOBS N . Horizon lines in the wild ［C］// Proceedings of the British Machine Vision Conference 2016 . York ： BMVC ， 2016 . doi: 10.5244/c.30.20 http://dx.doi.org/10.5244/c.30.20

赵亚凤，胡峻峰 . 一种双正交消隐点的双目相机标定方法［J］. 液晶与显示， 2016 ， 31 （ 10 ）： 958 - 966 ． doi: 10.3788/yjyxs20163110.0958 http://dx.doi.org/10.3788/yjyxs20163110.0958

ZHAO Y F ， HU J F . Binocular self calibration using two pairs of orthogonal vanishing points ［J］. Chinese Journal of Liquid Crystals and Displays ， 2016 ， 31 （ 10 ）： 958 - 966 . （in Chinese） . doi: 10.3788/yjyxs20163110.0958 http://dx.doi.org/10.3788/yjyxs20163110.0958

RONG J P ， HUANG S Y ， SHANG Z Y ， et al . Radial lens distortion correction using convolutional neural networks trained with synthesized images ［C］. 13th Asian Conference on Computer Vision . Taipei， China ： Springer ， 2017 ： 35 - 49 . doi: 10.1007/978-3-319-54187-7_3 http://dx.doi.org/10.1007/978-3-319-54187-7_3

FITZGIBBON A W . Simultaneous linear estimation of multiple view geometry and lens distortion ［C］// Proceedings of 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition . Kauai ： IEEE ， 2001 ： I .

陈文艺，许洁，杨辉，等 . 基于深度学习的单张图像畸变校正［J］. 激光与光电子学进展， 2020 ， 57 （ 24 ）： 241504 ． doi: 10.3788/lop57.241504 http://dx.doi.org/10.3788/lop57.241504

CHEN W Y ， XU J ， YANG H ， et al . Distortion correction of single image based on deep learning ［J］. Laser & Optoelectronics Progress ， 2020 ， 57 （ 24 ）： 241504 . （in Chinese） . doi: 10.3788/lop57.241504 http://dx.doi.org/10.3788/lop57.241504

LIAO K ， LIN C Y ， ZHAO Y ， et al . DR-GAN： automatic radial distortion rectification using conditional GAN in real-time ［J］. IEEE Transactions on Circuits and Systems for Video Technology ， 2020 ， 30 （ 3 ）： 725 - 733 . doi: 10.1109/tcsvt.2019.2897984 http://dx.doi.org/10.1109/tcsvt.2019.2897984

LI X Y ， ZHANG B ， SANDER P V ， et al . Blind geometric distortion correction on images through deep learning ［C］// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach ： IEEE ， 2019 ： 4850 - 4859 . doi: 10.1109/cvpr.2019.00499 http://dx.doi.org/10.1109/cvpr.2019.00499

ZHAO H ， SHI Y J ， TONG X ， et al . A simple yet effective pipeline for radial distortion correction ［C］. 2020 IEEE International Conference on Image Processing （ICIP）. Abu Dhabi ： IEEE ， 2020 ： 878 - 882 . doi: 10.1109/icip40778.2020.9191107 http://dx.doi.org/10.1109/icip40778.2020.9191107

LIAO K ， LIN C Y ， LIAO L X ， et al . Multi-level curriculum for training a distortion-aware barrel distortion rectification model ［C］// Proceedings of 2021 IEEE/CVF International Conference on Computer Vision . Montreal ： IEEE ， 2021 ： 4369 - 4378 . doi: 10.1109/iccv48922.2021.00435 http://dx.doi.org/10.1109/iccv48922.2021.00435

LIAO K ， LIN C Y ， ZHAO Y . A deep ordinal distortion estimation approach for distortion rectification ［J］. IEEE Transactions on Image Processing ， 2021 ， 30 ： 3362 - 3375 . doi: 10.1109/tip.2021.3061283 http://dx.doi.org/10.1109/tip.2021.3061283

REMONDINO F ， FRASER C . Digital camera calibration methods： considerations and comparisons ［J］. International Archives of the Photogrammetry， Remote Sensing and Spatial Information Sciences ， 2006 ， 36 （ 5 ）： 266 - 272 .

TANG R F ， FRITSCH D ， CRAMER M ， et al . A flexible mathematical method for camera calibration in digital aerial photogrammetry ［J］. Photogrammetric Engineering & Remote Sensing ， 2012 ， 78 （ 10 ）： 1069 - 1077 . doi: 10.14358/pers.78.10.1069 http://dx.doi.org/10.14358/pers.78.10.1069

ZHOU B L ， LAPEDRIZA A ， KHOSLA A ， et al . Places： a 10 million image database for scene recognition ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence ， 2018 ， 40 （ 6 ）： 1452 - 1464 . doi: 10.1109/tpami.2017.2723009 http://dx.doi.org/10.1109/tpami.2017.2723009

RONNEBERGER O ， FISCHER P ， BROX T . U-Net： convolutional networks for biomedical image segmentation ［C］. 18th International Conference on Medical Image Computing and Computer-Assisted Intervention . Munich ： Springer ， 2015 ： 234 - 241 . doi: 10.1007/978-3-319-24574-4_28 http://dx.doi.org/10.1007/978-3-319-24574-4_28

WOO S ， PARK J ， LEE J Y ， et al . CBAM： convolutional block attention module ［C］// Proceedings of the 15th European Conference on Computer Vision . Munich ： Springer ， 2018 ： 3 - 19 . doi: 10.1007/978-3-030-01234-2_1 http://dx.doi.org/10.1007/978-3-030-01234-2_1

YIN X Q ， WANG X C ， YU J ， et al . FishEyeRecNet： a multi-context collaborative deep network for fisheye image rectification ［C］// Proceedings of the 15th European Conference on Computer Vision . Munich ： Springer ， 2018 ： 475 - 490 . doi: 10.1007/978-3-030-01249-6_29 http://dx.doi.org/10.1007/978-3-030-01249-6_29

SHI Y J ， ZHANG D F ， WEN J S ， et al . Radial lens distortion correction by adding a weight layer with inverted foveal models to convolutional neural networks ［C］. 24th International Conference on Pattern Recognition （ICPR） . Beijing ： IEEE ， 2018 ： 1 - 6 . doi: 10.1109/icpr.2018.8545218 http://dx.doi.org/10.1109/icpr.2018.8545218

HOSONO M ， SIMO-SERRA E ， SONODA T . Self-supervised deep fisheye image rectification approach using coordinate relations ［C］. 17th International Conference on Machine Vision and Applications （MVA） . Aichi ： IEEE ， 2021 ： 1 - 5 . doi: 10.23919/mva51890.2021.9511349 http://dx.doi.org/10.23919/mva51890.2021.9511349

ZHAO H S ， SHI J P ， QI X J ， et al . Pyramid scene parsing network ［C］// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu ： IEEE ， 2017 ： 6230 - 6239 . doi: 10.1109/cvpr.2017.660 http://dx.doi.org/10.1109/cvpr.2017.660

CHEN L C ， ZHU Y K ， PAPANDREOU G ， et al . Encoder-decoder with atrous separable convolution for semantic image segmentation ［C］// Proceedings of the 15th European Conference on Computer Vision . Munich ： Springer ， 2018 ： 833 - 851 . doi: 10.1007/978-3-030-01234-2_49 http://dx.doi.org/10.1007/978-3-030-01234-2_49

ITU-T . Recommendation P . 913 ： methods for the subjective assessment of video quality， audio quality and audiovisual quality of Internet video and distribution quality television in any environment ［EB/OL］. ［ 2021-06-03 ］. https：//www.itu.int/rec/T-REC-P.913-202106-I/en https://www.itu.int/rec/T-REC-P.913-202106-I/en .

Views

201

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Depth estimation of light field based on correction occlusion perception

Blind super-resolution model based on degradation-aware

Segmentation and recognition of magnetic tile surface defects based on deep learning

Multi-atlaslabel fusion based on U-Net

Related Author

WU Jin

XIANG Sen

DENG Huiping

NI Jing

JIANG Nian-de

CAI Jian-feng

Qun YAN

Zhi-xian LIN

Related Institution

School of Information Science and Engineering， Wuhan University of Science and Technology

School of Information Engineering， East China University of Technology

College of Physics and Information Engineering, Fuzhou University

Jinjiang RichSense Electronic Technology Co., Ltd., Jinjiang

School of Physics and Electronic-Electrical Engineering, Ningxia University

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China 130033 Postal code：130033
Tel：0431-86176059 Email：yjxs@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰