Light field reconstruction based on spatial-angular decouple and fuse network

Hong-ji ZHANG; Hui-ping DENG; Sen XIANG; Jin WU

doi:10.37188/CJLCD.2022-0117

您当前的位置：

首页 >

文章列表页 >

Light field reconstruction based on spatial-angular decouple and fuse network

Image Processing | 更新时间：2022-09-26

- Light field reconstruction based on spatial-angular decouple and fuse network
- Chinese Journal of Liquid Crystals and Displays Vol. 37, Issue 10, Pages: 1345-1354(2022)
- 作者机构：
  
  武汉科技大学信息科学与工程学院，湖北武汉 430081
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(61702384;61502357)
- DOI：10.37188/CJLCD.2022-0117
  CLC： TP391
- Received：11 April 2022，
  
  Revised：19 April 2022，
  
  Published：05 October 2022
- 稿件说明：
移动端阅览
Hong-ji ZHANG, Hui-ping DENG, Sen XIANG, et al. Light field reconstruction based on spatial-angular decouple and fuse network[J]. Chinese journal of liquid crystals and displays, 2022, 37(10): 1345-1354.
DOI：

Hong-ji ZHANG, Hui-ping DENG, Sen XIANG, et al. Light field reconstruction based on spatial-angular decouple and fuse network[J]. Chinese journal of liquid crystals and displays, 2022, 37(10): 1345-1354. DOI： 10.37188/CJLCD.2022-0117.

摘要

密集采样的光场能提供沉浸式的视觉体验，在三维重建和虚拟现实等领域有着广泛的应用。光场重建技术可以从稀疏采样光场获取密集采样光场，但当光场图像的基线增大时，会造成重建视图中边缘的遮挡和纹理重复区域的模糊。本文提出了一种基于空间角度解耦融合的光场重建网络，充分利用光场图像蕴含的深度信息以及空间信息解决上述复杂区域的重建难题。采用多路输入方式来获取光场多方向信息以解决遮挡问题，在每条支路通过空洞空间卷积池化金字塔模块来提取多尺度特征，以获取大接受域的视差信息和上下文信息；设计了空间角度解耦融合模块，利用空间信息指导光场角度超分辨率重建，使重建视图在纹理重复区域更加清晰。实验结果表明，该网络在HCI、HCI old和30 scenes数据集上重建光场的平均PSNR和SSIM分别达到了39.73 dB和0.965，表现出较高的准确性，优于所比较的方法。

Abstract

The densely sampled light field can provide immersive visual experience and have extensive applications in areas such as 3D reconstruction and virtual reality. Light field reconstruction technology can obtain densely sampled light fields from sparsely sampled light fields， but when the baseline of the light field image increases， it will cause occlusion on edges and textured duplicate areas blur in the reconstructed view. In this paper， we propose a light field reconstruction network based on spatial-anglular decouple and fuse， it can make full use of the depth information and spatial information contained in the light field image and solve the reconstruction problems of the complex regions. The multi-branch input is used to supply the multi-directional information of the light field to solve the occlusion problem. In each branch， multi-scale features are extracted by atrous spatial pyramid pooling module to obtain disparity information and context information for the large receptive fields. The spatial-angular decouple and fuse module is designed to explore the spatial information， guiding the light field angle super-resolution reconstruction so as to obtain the reconstruction view with clearer texture duplicate region. Experimental results show that the average PSNR and SSIM reached 39.73 dB and 0.965 on the HCI， HCI old and 30 scenes datasets，showing high accuracy and outperforming the compared method.

关键词

Keywords

references

陈苑锋 . 视觉深度估计与点云建图研究进展［J］. 液晶与显示， 2021 ， 36 （ 6 ）： 896 - 911 ． doi: 10.37188/CJLCD.2020-0047 http://dx.doi.org/10.37188/CJLCD.2020-0047

CHEN Y F . Progress of visual depth estimation and point cloud mapping ［J］. Chinese Journal of Liquid Crystals and Displays ， 2021 ， 36 （ 6 ）： 896 - 911 . （in Chinese） . doi: 10.37188/CJLCD.2020-0047 http://dx.doi.org/10.37188/CJLCD.2020-0047

MENG N ， LI K ， LIU J Z ， et al . Light field view synthesis via aperture disparity and warping confidence map ［J］. IEEE Transactions on Image Processing ， 2021 ， 30 ： 3908 - 3921 . doi: 10.1109/tip.2021.3066293 http://dx.doi.org/10.1109/tip.2021.3066293

WU G C ， WANG Y Q ， LIU Y B ， et al . Spatial-angular attention network for light field reconstruction ［J］. IEEE Transactions on Image Processing ， 2021 ， 30 ： 8999 - 9013 . doi: 10.1109/tip.2021.3122089 http://dx.doi.org/10.1109/tip.2021.3122089

KO K ， KOH Y J ， CHANG S ， et al . Light field super-resolution via adaptive feature remixing ［J］. IEEE Transactions on Image Processing ， 2021 ， 30 ： 4114 - 4128 . doi: 10.1109/tip.2021.3069291 http://dx.doi.org/10.1109/tip.2021.3069291

CHEN A P ， XU Z X ， ZHAO F Q ， et al . MVSNeRF： fast generalizable radiance field reconstruction from multi-view stereo ［C］// Proceedings of 2021 IEEE/CVF International Conference on Computer Vision . Montreal ： IEEE ， 2021 ： 14104 - 14113 . doi: 10.1109/iccv48922.2021.01386 http://dx.doi.org/10.1109/iccv48922.2021.01386

SHI L X ， HASSANIEH H ， DAVIS A ， et al . Light field reconstruction using sparsity in the continuous Fourier domain ［J］. ACM Transactions on Graphics ， 2014 ， 34 （ 1 ）： 12 . doi: 10.1145/2682631 http://dx.doi.org/10.1145/2682631

VAGHARSHAKYAN S ， BREGOVIC R ， GOTCHEV A . Light field reconstruction using shearlet transform ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence ， 2018 ， 40 （ 1 ）： 133 - 147 . doi: 10.1109/tpami.2017.2653101 http://dx.doi.org/10.1109/tpami.2017.2653101

WU G C ， ZHAO M D ， WANG L Y ， et al . Light field reconstruction using deep convolutional network on EPI ［C］// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu ： IEEE ， 2017 ： 1638 - 1646 . doi: 10.1109/cvpr.2017.178 http://dx.doi.org/10.1109/cvpr.2017.178

PENDU M L ， GUILLEMOT C ， SMOLIC A . A Fourier disparity layer representation for light fields ［J］. IEEE Transactions on Image Processing ， 2019 ， 28 （ 11 ）： 5740 - 5753 . doi: 10.1109/tip.2019.2922099 http://dx.doi.org/10.1109/tip.2019.2922099

YEUNG H W F ， HOU J H ， CHEN J ， et al . Fast light field reconstruction with deep coarse-to-fine modeling of spatial-angular clues ［C］// Proceedings of the 15th European Conference on Computer Vision （ECCV） . Munich ： Springer ， 2018 ： 138 - 154 . doi: 10.1007/978-3-030-01231-1_9 http://dx.doi.org/10.1007/978-3-030-01231-1_9

MENG N ， SO H K H ， SUN X ， et al . High-dimensional dense residual convolutional neural network for light field reconstruction ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence ， 2021 ， 43 （ 3 ）： 873 - 886 . doi: 10.1109/tpami.2019.2945027 http://dx.doi.org/10.1109/tpami.2019.2945027

MENG N ， WU X F ， LIU J Z ， et al . High-order residual network for light field super-resolution ［J］. Proceedings of the AAAI Conference on Artificial Intelligence ， 2020 ， 34 （ 7 ）： 11757 - 11764 . doi: 10.1609/aaai.v34i07.6847 http://dx.doi.org/10.1609/aaai.v34i07.6847

JEON H G ， PARK J ， CHOE G ， et al . Accurate depth map estimation from a lenslet light field camera ［C］// Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition . Boston ： IEEE ， 2015 ： 1547 - 1555 . doi: 10.1109/cvpr.2015.7298762 http://dx.doi.org/10.1109/cvpr.2015.7298762

WANNER S ， GOLDLUECKE B . Variational light field analysis for disparity estimation and super-resolution ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence ， 2014 ， 36 （ 3 ）： 606 - 619 . doi: 10.1109/tpami.2013.147 http://dx.doi.org/10.1109/tpami.2013.147

FLYNN J ， NEULANDER I ， PHILBIN J ， et al . Deep stereo： learning to predict new views from the world's imagery ［C］// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas ： IEEE ， 2016 ： 5515 - 5524 . doi: 10.1109/cvpr.2016.595 http://dx.doi.org/10.1109/cvpr.2016.595

KALANTARI N K ， WANG T C ， RAMAMOORTHI R . Learning-based view synthesis for light field cameras ［J］. ACM Transactions on Graphics ， 2016 ， 35 （ 6 ）： 193 . doi: 10.1145/2980179.2980251 http://dx.doi.org/10.1145/2980179.2980251

JIN J ， HOU J H ， YUAN H ， et al . Learning light field angular super-resolution via a geometry-aware network ［J］. Proceedings of the AAAI Conference on Artificial Intelligence ， 2020 ， 34 （ 7 ）： 11141 - 11148 . doi: 10.1609/aaai.v34i07.6771 http://dx.doi.org/10.1609/aaai.v34i07.6771

WU G C ， LIU Y B ， DAI Q H ， et al . Learning sheared EPI structure for light field reconstruction ［J］. IEEE Transactions on Image Processing ， 2019 ， 28 （ 7 ）： 3261 - 3273 . doi: 10.1109/tip.2019.2895463 http://dx.doi.org/10.1109/tip.2019.2895463

ZHOU T H ， TUCKER R ， FLYNN J ， et al . Stereo magnification： learning view synthesis using multiplane images ［J］. ACM Transactions on Graphics ， 2018 ， 37 （ 4 ）： 65 . doi: 10.1145/3197517.3201323 http://dx.doi.org/10.1145/3197517.3201323

MILDENHALL B ， SRINIVASAN P P ， ORTIZ-CAYON R ， et al . Local light field fusion： practical view synthesis with prescriptive sampling guidelines ［J］. ACM Transactions on Graphics ， 2019 ， 38 （ 4 ）： 29 . doi: 10.1145/3306346.3322980 http://dx.doi.org/10.1145/3306346.3322980

HONAUER K ， JOHANNSEN O ， KONDERMANN D ， et al . A dataset and evaluation methodology for depth estimation on 4D light fields ［C］// Proceedings of the 13th Asian Conference on Computer Vision . Taipei， China ： Springer ， 2017 ： 19 - 34 . doi: 10.1007/978-3-319-54187-7_2 http://dx.doi.org/10.1007/978-3-319-54187-7_2

WANNER S ， MEISTER S ， GOLDLUECKE B . Datasets and benchmarks for densely sampled 4D light fields ［M］//BRONSTEIN M， FAVRE J， HORMANN K. Vision， Modeling & Visualization. Geneva ： The Eurographics Association ， 2013 ： 225 - 226 .

KINGMA D P ， BA J . Adam： a method for stochastic optimization ［J］. arXiv ， 2014 ： 1412 .6980.

Views

297

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Edge-guided light field image saliency detection

Research progress on absolute visual localization of unmanned aerial vehicles

3D hand reconstruction method based on adaptive occlusion recovery and topology-pose bidirectional perception

Related Author

ZHANG Hong-ji

DENG Hui-ping

XIANG Sen

WU Jin

LIANG Xiao

XIANG Sen

WU Jin

Deng Huiping

Related Institution

Engineering Research Center for Metallurgical Automation and Measurement Technology of Ministry of Education， Wuhan University of Science and Technology

Technological Institute of Materials & Energy Science （TIMES）， School of Electronic Information， Xijing University

School of Artificial Intelligence， Optics and Electronics （iOPEN）， Northwestern Polytechnical University

School of Automation， Nanjing University of Information Science & Technology

School of Computer and Computing Science， Hangzhou City University

AI问答

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China 130033 Postal code：130033
Tel：0431-86176059 Email：yjxs@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰