Virtual viewpoint image generation method based on deep convolutional neural network in autostereoscopic display

Ao-wei FU; Min ZHAO; Ling LUO; Yan XING; Huan DENG; Qiong-hua WANG

doi:10.3788/YJYXS20193411.1031

您当前的位置：

首页 >

文章列表页 >

Virtual viewpoint image generation method based on deep convolutional neural network in autostereoscopic display

Materials and Devices | 更新时间：2020-07-05

- Virtual viewpoint image generation method based on deep convolutional neural network in autostereoscopic display
- Chinese Journal of Liquid Crystals and Displays Vol. 34, Issue 11, Pages: 1031-1036(2019)
- 作者机构：
  
  1.四川大学电子信息学院, 四川成都 610065
  2.北京航空航天大学仪器科学与光电工程学院, 北京 100191
- 作者简介：
- 基金信息：
- DOI：10.3788/YJYXS20193411.1031
  CLC： TB133
- Received：22 August 2019，
  
  Accepted：02 September 2019，
  
  Published：05 November 2019
- 稿件说明：
移动端阅览
Ao-wei FU, Min ZHAO, Ling LUO, et al. Virtual viewpoint image generation method based on deep convolutional neural network in autostereoscopic display[J]. Chinese journal of liquid crystals and displays, 2019, 34(11): 1031-1036.
DOI：

Ao-wei FU, Min ZHAO, Ling LUO, et al. Virtual viewpoint image generation method based on deep convolutional neural network in autostereoscopic display[J]. Chinese journal of liquid crystals and displays, 2019, 34(11): 1031-1036. DOI： 10.3788/YJYXS20193411.1031.

摘要

传统虚拟视点生成采用像素填充法对生成的虚拟视点图像进行空洞填充和伪影修复，其修复效果无法满足自由立体显示需求。为了获取高质量的虚拟视点图像，提出了一种基于深度卷积神经网络的虚拟视点生成方法。该方法采用随机初始化的深度卷积神经网络作为图像先验，经过卷积神经网络结构的不断迭代，对虚拟视点图像的空洞和伪影进行修复，并将得到的高质量虚拟视点图像合成为自由立体图像，用于自由立体显示。修复后的虚拟视点图像的PSNR均值为25.6，相比传统像素填充方法有明显提升。实验结果表明，所提方法能够实现高质量的自由立体显示效果。

Abstract

The conventional virtual viewpoint generation methods use the pixel filling algorithm to fill the image holes and restore artifacts on the generated virtual viewpoint images. However

the repaired virtual viewpoint images cannot meet the requirements of the autostereoscopic display. In order to obtain virtual viewpoint images with high quality

we propose a virtual viewpoint image generation method for autostereoscopic display based on deep convolutional neural network (DCNN). The method uses a randomly initialized deep convolutional neural network as an image prior

iterates through the structure of the convolutional neural network

and effectively repairs the image holes and artifacts of the virtual viewpoint image. The repaired virtual viewpoint images are then synthesized into the autostereoscopic image for autostereoscopic 3D display. The average value of PSNR of the virtual viewpoint images is 25.6

which is higher than that of the conventional pixel filling method. The experimental results demonstrate the proposed method can obtain high quality autostereoscopic 3D images.

关键词

Keywords

references

王琼华.3D显示技术与器件[M].北京:科学出版社, 2011.

WANG Q H. 3 D Display Technology and Device [M]. Beijing:Science Press, 2011. (in Chinese)

王琼华, 王爱红.三维立体显示综述[J].计算机应用, 2010, 30(3):579-581, 588.

WANG Q H, WANG A H. Survey on stereoscopic three-dimensional display[J]. Journal of Computer Applications , 2010, 30(3):579-581, 588. (in Chinese)

ZHANG Y Q, WANG Q H, WANG A H, et al . Depth-map generation for multi-view autostereoscopic 3-D displays based on the sift algorithm constrained by boundary[J]. Journal of the Society for Information Display , 2010, 18(7):513-518.

XIONG Z L, WANG Q H, LI S L, et al . Partially-overlapped viewing zone based integral imaging system with super wide viewing angle[J]. Optics Express , 2014, 22(19):22268-22277.

DENG H, WANG Q H, Li D H. Method of generating orthoscopic elemental image array from sparse camera array[J]. Chinese Optics Letters , 2012, 10(6):061102.

XIAO F, XIONG Z L, DENG H, et al . P-100:Approach on two-step 3D warping with background filling for multi-view autostereoscopic display[J]. SID Symposium Digest of Technical Papers , 2017, 48(1):1624-1626.

赵悟翔, 王琼华, 李大海, 等.阶梯光栅多视点自由立体显示的子像素排列[J].四川大学学报(工程科学版), 2009, 41(6):216-218, 236.

ZHAO W X, WANG Q H, LI D H, et al . Sub-pixel arrangement for multi-view autostereoscopic display based on step parallax barrier[J]. Journal of Sichuan University ( Engineering Science Edition ), 2009, 41(6):216-218, 236. (in Chinese)

FEHN C. Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV[C]// Proceedings of SPIE Stereoscopic Displays and Virtual Reality Systems XI . San Jose, California, United States: SPIE, 2004: 93-104. https://www.researchgate.net/publication/228747651_Depth-image-based_rendering_DIBR_compression_and_transmission_for_a_new_approach_on_3D-TV

MORI Y, FUKUSHIMA N, YENDO T, et al . View generation with 3D warping using depth information for FTV[J]. Signal Processing : Image Communication , 2009, 24(1/2):65-72.

LEMPITSKY V, VEDALDI A, ULYANOV D. Deep image prior[C]// Proceedings of 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City, UT, USA: IEEE, 2018: 9446-9454. https://www.researchgate.net/publication/329745621_Deep_Image_Prior

CHAWLA N V, BOWYER K W, HALL L O, et al . SMOTE:synthetic minority over-sampling technique[J]. Journal of Artificial Intelligence Research , 2002, 16(1):321-357.

BERTALMIO M, VESE L, SAPIRO G, et al . Simultaneous structure and texture image inpainting[J]. IEEE Transactions on Image Processing , 2003, 12(8):882-889.

PATHAK D, KRÄHENBUHL P, DONAHUE J, et al . Context encoders: feature learning by inpainting[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas, NV, USA: IEEE, 2016: 2536-2544. https://www.researchgate.net/publication/301816789_Context_Encoders_Feature_Learning_by_Inpainting

YANG C, LU X, LIN Z, et al . High-resolution image inpainting using multi-scale neural patch synthesis[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Honolulu, HI, USA: IEEE, 2017: 4076-4084. https://www.researchgate.net/publication/320964396_High-Resolution_Image_Inpainting_Using_Multi-scale_Neural_Patch_Synthesis

黄冠婷, 韩学辉, 龚晓婷, 等.基于图像分割和区域匹配的灰度图像彩色化算法[J].液晶与显示, 2019, 34(6):619-626.

HUANG G T, HAN X H, GONG X T, et al . Gray image colorization algorithm based on image segmentation and region matching[J]. Chinese Journal of Liquid Crystals and Displays , 2019, 34(6):619-626. (in Chinese)

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Second-order image restoration based on gated convolution and attention transfer

Effect of the vertical/slanted staggered barrier on autostereoscopic displays

Multi-feature fusion gesture recognition based ondeep convolutional neural network

Criminisi algorithm based on marginal structure for image inpainting

Problems about Number of Views and Crosstalk Tolerance in Projective Auto-Stereoscopic Display Based on Lenticular Grating

Related Author

WANG Gang

GU Li-rui

PENG Yan-fei

Yan ZHAO

En-guo CHEN

Yun YE

Yu CHEN

Xiang-yao ZENG

Related Institution

School of Electronic and Information Engineering， Liaoning Technical University

Bohai Shipbuilding Vocational College

National & Local United Engineering Laboratory of Flat Panel Display Technology, School of Physics and Information Engineering, Fuzhou University

School of Information and Control Engineering, Xi'an University of Architecture and Technology

College of Information Science and Technology, Wuhan University of Science and Techology

AI问答

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China 130033 Postal code：130033
Tel：0431-86176059 Email：yjxs@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰