1.广东工业大学 集成电路学院, 广东 广州 510006
2.广东工业大学 信息工程学院, 广东 广州 510006
[ "叶武剑(1987—),男,广东韶关人,博士,讲师,2004年于檀国大学获得博士学位,主要从事计算机视觉与深度学习、机器学习应用等方面的研究。E-mail: yewjian@126.com" ]
扫 描 看 全 文
叶武剑, 林振溢, 刘怡俊, 等. 图像感知引导CycleGAN网络的背景虚化方法[J]. 液晶与显示, 2023,38(9):1248-1261.
YE Wu-jian, LIN Zhen-yi, LIU Yi-jun, et al. Background defocus method of image perception guided CycleGAN network[J]. Chinese Journal of Liquid Crystals and Displays, 2023,38(9):1248-1261.
叶武剑, 林振溢, 刘怡俊, 等. 图像感知引导CycleGAN网络的背景虚化方法[J]. 液晶与显示, 2023,38(9):1248-1261. DOI: 10.37188/CJLCD.2022-0403.
YE Wu-jian, LIN Zhen-yi, LIU Yi-jun, et al. Background defocus method of image perception guided CycleGAN network[J]. Chinese Journal of Liquid Crystals and Displays, 2023,38(9):1248-1261. DOI: 10.37188/CJLCD.2022-0403.
现有生成对抗网络在背景虚化处理过程中,往往是无差别地提取整张输入图像的特征,导致网络难以区分图像的前后景,从而容易出现图像失真的现象。本文提出了图像感知引导CycleGAN网络的背景虚化方法,通过引入图像感知信息以提升模型性能。图像感知信息包括注意力信息和景深信息,前者用于引导网络关注不同的前后景区域,从而区分前后景;而后者用于增强前景目标的感知信息,能够实现有效的智能定焦并减少图像出现失真的现象,使背景虚化效果更佳。通过最小化生成对抗损失及循环一致性损失,可以避免丢失过多景深信息,提高图片的生成质量。实验结果及数据表明,提出的方法在背景虚化过程中能有效区分前后景并改善图像失真的现象,使生成的效果更加真实。此外,在与现有方法生成的图像效果对比中,通过采用问卷调查的方式进行了评估。本文提出的图像感知引导CycleGAN网络的背景虚化方法与SOTA相比,生成的图像质量更好,模型大小与生成图像的速率也具有明显的优势,分别为56.10 MB及47 ms。
The existing image conversion algorithms based on generative adversarial network often extract the features of the whole input image indiscriminately in the process of background defocus, which makes it difficult for the network to distinguish the front and back scenes of the image, so it is easy to lead to the phenomenon of image distortion. We propose a background virtualization method of image perception guided CycleGAN network. The image perception information is introduced to improve the performance of the model. The image perception information includes attention information and depth of field information. The former is used to guide the network to pay attention to different foreground and background areas, so as to distinguish the foreground and background. The latter is used to enhance the perception information of foreground targets, achieve effective intelligent focusing, and reduce image distortion, making the background defocus better. The experimental results and data show that the method proposed in this paper can effectively distinguish the foreground and background in the process of background defocus, reduce the phenomenon of image distortion, and make the generated effect more real. In addition, in comparison with the image effect generated by the existing methods, the questionnaire survey is used for evaluation. A background virtualization method for image perception guided CycleGAN network is proposed, comparing with SOTA, the image quality generated is the best, and its model size and image generation rate also have obvious advantages of 56.10 MB and 47 ms, respectively.
背景虚化图像感知CycleGAN网络智能定焦
background defocusimage perceptionCycleGAN networkintelligent focusing
肖进胜,杜康华,涂超平,等.基于多聚焦图像深度信息提取的背景虚化显示[J].自动化学报,2015,41(2):304-311. doi: 10.16383/j.aas.2015.c140059http://dx.doi.org/10.16383/j.aas.2015.c140059
XIAO J S, DU K H, TU C P, et al. Bokeh display based on depth information extraction of multi-focus images [J]. Acta Automatica Sinica, 2015, 41(2): 304-311. (in Chinese). doi: 10.16383/j.aas.2015.c140059http://dx.doi.org/10.16383/j.aas.2015.c140059
张婉佳.基于光场相机图像的超分辨率重建研究[D].西安:西安电子科技大学,2020. doi: 10.1109/ccdc49329.2020.9164705http://dx.doi.org/10.1109/ccdc49329.2020.9164705
ZHANG W J. Research on super-resolution reconstruction based on light field camera image [D]. Xi'an: Xidian University, 2020. (in Chinese). doi: 10.1109/ccdc49329.2020.9164705http://dx.doi.org/10.1109/ccdc49329.2020.9164705
ZUO C, QIAN J M, FENG S J, et al. Deep learning in optical metrology: a review [J]. Light: Science & Applications, 2022, 11(1): 39. doi: 10.1038/s41377-022-00714-xhttp://dx.doi.org/10.1038/s41377-022-00714-x
POTMESIL M, CHAKRAVARTY I. A lens and aperture camera model for synthetic image generation [J]. ACM SIGGRAPH Computer Graphics, 1981, 15(3): 297-305. doi: 10.1145/965161.806818http://dx.doi.org/10.1145/965161.806818
LEE S, KIM G J, CHOI S. Real-time depth-of-field rendering using anisotropically filtered mipmap interpolation [J]. IEEE Transactions on Visualization and Computer Graphics, 2009, 15(3): 453-464. doi: 10.1109/tvcg.2008.106http://dx.doi.org/10.1109/tvcg.2008.106
XIE G F, SUN X, WANG W C. Interactive depth-of-field rendering with secondary rays [J]. Journal of Computer Science and Technology, 2013, 28(3): 490-498. doi: 10.1007/s11390-013-1350-4http://dx.doi.org/10.1007/s11390-013-1350-4
LIU X, ROKNE J G. Depth of field synthesis from sparse views [J]. Computers & Graphics, 2016, 55: 21-32. doi: 10.1016/j.cag.2015.10.015http://dx.doi.org/10.1016/j.cag.2015.10.015
李晓颖,周卫星,吴孙槿,等.基于单目深度估计方法的图像分层虚化技术[J].华南师范大学学报(自然科学版),2018,50(1):124-128. doi: 10.6054/j.jscnun.2018038http://dx.doi.org/10.6054/j.jscnun.2018038
LI X Y, ZHOU W X, WU S J, et al. Hierarchical image blurring technology based on monocular depth estimation method [J]. Journal of South China Normal University (Natural Science Edition), 2018, 50(1): 124-128. (in Chinese). doi: 10.6054/j.jscnun.2018038http://dx.doi.org/10.6054/j.jscnun.2018038
SHEN X Y, HERTZMANN A, JIA J Y, et al. Automatic portrait segmentation for image stylization [J]. Computer Graphics Forum, 2016, 35(2): 93-102. doi: 10.1111/cgf.12814http://dx.doi.org/10.1111/cgf.12814
MOK Y, KIM H, SHIN J, et al. Semantic segmentation-based real-time shallow depth-of-field imaging using a single smartphone camera [C]. 2019 IEEE 9th International Conference on Consumer Electronics (ICCE). Berlin: IEEE, 2019: 248-250. doi: 10.1109/icce-berlin47944.2019.8966188http://dx.doi.org/10.1109/icce-berlin47944.2019.8966188
HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition [C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE, 2016: 770-778. doi: 10.1109/cvpr.2016.90http://dx.doi.org/10.1109/cvpr.2016.90
WADHWA N, GARG R, JACOBS D E, et al. Synthetic depth-of-field with a single-camera mobile phone [J]. ACM Transactions on Graphics, 2018, 37(4): 64. doi: 10.1145/3197517.3201329http://dx.doi.org/10.1145/3197517.3201329
PUROHIT K, SUIN M, KANDULA P, et al. Depth-guided dense dynamic filtering network for bokeh effect rendering [C]. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). Seoul: IEEE, 2019: 3417-3426. doi: 10.1109/iccvw.2019.00424http://dx.doi.org/10.1109/iccvw.2019.00424
IGNATOV A, PATEL J, TIMOFTE R. Rendering natural camera bokeh effect with deep learning [C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Seattle: IEEE, 2020: 1676-1686. doi: 10.1109/cvprw50498.2020.00217http://dx.doi.org/10.1109/cvprw50498.2020.00217
DUTTA S, DAS S D, SHAH N A, et al Stacked deep multi-scale hierarchical network for fast bokeh effect rendering from a single image [C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Nashville: IEEE, 2021: 2398-2407. doi: 10.1109/cvprw53098.2021.00272http://dx.doi.org/10.1109/cvprw53098.2021.00272
刘成民,叶武剑,刘怡俊.基于图像感知与分割的自动背景虚化算法[J].激光与光电子学进展,2022,59(4):0410002. doi: 10.3788/LOP202259.0410002http://dx.doi.org/10.3788/LOP202259.0410002
LIU C M, YE W J, LIU Y J. Automatic background blurring algorithm based on image perception and segmentation [J]. Laser & Optoelectronics Progress, 2022, 59(4): 0410002. (in Chinese). doi: 10.3788/LOP202259.0410002http://dx.doi.org/10.3788/LOP202259.0410002
ZHENG B L, CHEN Q, YU S X, et al. Constrained predictive filters for single image bokeh rendering [J]. IEEE Transactions on Computational Imaging, 2022, 8: 346-357. doi: 10.1109/tci.2022.3171417http://dx.doi.org/10.1109/tci.2022.3171417
JEONG Y, BAEK S Y, SEOK Y, et al. Real-time dynamic bokeh rendering with efficient look-up table sampling [J]. IEEE Transactions on Visualization and Computer Graphics, 2022, 28(2): 1373-1384. doi: 10.1109/tvcg.2020.3014474http://dx.doi.org/10.1109/tvcg.2020.3014474
LUO X R, PENG J W, XIAN K, et al. Defocus to focus: photo-realistic bokeh rendering by fusing defocus and radiance priors [J]. Information Fusion, 2023, 89: 320-335. doi: 10.1016/j.inffus.2022.08.023http://dx.doi.org/10.1016/j.inffus.2022.08.023
ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks [C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE, 2017: 5967-5976. doi: 10.1109/cvpr.2017.632http://dx.doi.org/10.1109/cvpr.2017.632
ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks [C]. 2017 IEEE International Conference on Computer Vision (ICCV). Venice: IEEE, 2017: 2242-2251. doi: 10.1109/iccv.2017.244http://dx.doi.org/10.1109/iccv.2017.244
QIAN M, QIAO C Y, LIN J M, et al. BGGAN: bokeh-glass generative adversarial network for rendering realistic bokeh [C]. 16th European Conference on Computer Vision. Glasgow: Springer, 2020: 229-244. doi: 10.1007/978-3-030-67070-2_14http://dx.doi.org/10.1007/978-3-030-67070-2_14
PIZZATI F, CERRI P, DE CHARETTE R. CoMoGAN: continuous model-guided image-to-image translation [C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville: IEEE, 2021: 14283-14293. doi: 10.1109/cvpr46437.2021.01406http://dx.doi.org/10.1109/cvpr46437.2021.01406
WANG L, DENG H, ZHONG F Y, et al. Integral imaging display with enhanced depth of field based on bifocal lens array [J]. Journal of the Society for Information Display, 2021, 29(9): 689-696. doi: 10.1002/jsid.1018http://dx.doi.org/10.1002/jsid.1018
XIE X H, YU X B, GAO X, et al. Extended depth of field method with a designed diffraction optical element based on multi-depth fusion and end-to-end optimization [J]. Optics Communications, 2022, 517: 128317. doi: 10.1016/j.optcom.2022.128317http://dx.doi.org/10.1016/j.optcom.2022.128317
SITU G H. Deep holography [J]. Light: Advanced Manufacturing, 2022, 3(2): 278-300. doi: 10.37188/lam.2022.013http://dx.doi.org/10.37188/lam.2022.013
LUO Y, ZHAO Y F, LI J X, et al. Computational imaging without a computer: seeing through random diffusers at the speed of light [J]. eLight, 2022, 2: 4. doi: 10.1186/s43593-022-00012-4http://dx.doi.org/10.1186/s43593-022-00012-4
KIM J, KIM M, KANG H, et al. U-GAT-IT: unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation [C]. 8th International Conference on Learning Representations (ICLR). Addis Ababa: ICLR, 2020.
WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module [C]//Proceedings of the 15th European Conference on Computer Vision. Munich: Springer, 2018: 3-19. doi: 10.1007/978-3-030-01234-2_1http://dx.doi.org/10.1007/978-3-030-01234-2_1
CHEN W F, FU Z, YANG D W, et al. Single-image depth perception in the wild [C]//Proceedings of the 30th International Conference on Neural Information Processing Systems. Barcelona: Curran Associates Inc., 2016: 730-738.
CHATTOPADHAY A, SARKAR A, HOWLADER P, et al. Grad-CAM++: generalized gradient-based visual explanations for deep convolutional networks [C]. 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Tahoe: IEEE, 2018: 839-847. doi: 10.1109/wacv.2018.00097http://dx.doi.org/10.1109/wacv.2018.00097
XIE S N, TU Z W. Holistically-nested edge detection [C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago: IEEE, 2015: 1395-1403. doi: 10.1109/iccv.2015.164http://dx.doi.org/10.1109/iccv.2015.164
TANG H, XU D, SEBE N, et al. Attention-guided generative adversarial networks for unsupervised image-to-image translation [C]. 2019 International Joint Conference on Neural Networks (IJCNN). Budapest: IEEE, 2019: 1-8. doi: 10.1109/ijcnn.2019.8851881http://dx.doi.org/10.1109/ijcnn.2019.8851881
劳俊明,叶武剑,刘怡俊,等.基于双专用注意力机制引导的循环生成对抗网络[J].液晶与显示,2022,37(6):746-757. doi: 10.37188/CJLCD.2021-0293http://dx.doi.org/10.37188/CJLCD.2021-0293
LAO J M, YE W J, LIU Y J, et al. A cycle generative adversarial network guided by dual special attention mechanism [J]. Chinese Journal of Liquid Crystals and Displays, 2022, 37(6): 746-757. (in Chinese). doi: 10.37188/CJLCD.2021-0293http://dx.doi.org/10.37188/CJLCD.2021-0293
0
浏览量
36
下载量
0
CSCD
关联资源
相关文章
相关作者
相关机构