Optical flow estimation <italic style="font-style: italic">via</italic> fusing sequence image intensity correlation information

AN Tong; JIA Di; ZHANG Jia-bao; CAI Peng

doi:10.37188/CJLCD.2022-0384

您当前的位置：

首页 >

文章列表页 >

Optical flow estimation via fusing sequence image intensity correlation information

Image Processing | 更新时间：2023-10-10

- Optical flow estimation via fusing sequence image intensity correlation information
- Chinese Journal of Liquid Crystals and Displays Vol. 38, Issue 10, Pages: 1434-1444(2023)
- 作者机构：
  
  1.辽宁工程技术大学电子与信息工程学院，辽宁葫芦岛 125105
  2.辽宁工程技术大学电气与控制工程学院，辽宁葫芦岛 125105
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(61601213);Supported by National Natural Science Foundation of Liaoning Province(LJ2020FWL004);Supported by China Postdoctoral Science Foundation(2017M61125)
- DOI：10.37188/CJLCD.2022-0384
  CLC：
扫描看全文
AN Tong, JIA Di, ZHANG Jia-bao, et al. Optical flow estimation via fusing sequence image intensity correlation information. [J]. Chinese Journal of Liquid Crystals and Displays 38(10):1434-1444(2023)
DOI：

AN Tong, JIA Di, ZHANG Jia-bao, et al. Optical flow estimation via fusing sequence image intensity correlation information. [J]. Chinese Journal of Liquid Crystals and Displays 38(10):1434-1444(2023) DOI： 10.37188/CJLCD.2022-0384.

摘要

针对现有光流估计方法在目标轮廓分割不清晰、缺乏细粒度的问题，本文提出融合序列影像相关区域信息的光流估计网络。通过特征编码器和全局编码器分别提取图像的编码特征和上下文特征，并通过下采样处理缩减特征尺寸。在构建4D相关体前，对输入的连续两帧特征图进行分区处理，以强弱相关的方式计算稠密的视觉相似度，建立更为精细的4D相关体积。在迭代更新阶段，提出残差卷积滤波器和细粒度模块，分别应用于处理相关体和光流传递，使得在融合相关体信息和光流信息前保留更多的局部小位移信息。在KITTI-2015数据集和MPI-Sintel数据集上与其他方法进行对比，光流估计评价指标分别提升了8.2%和6.15%。本文给出的网络模型可以更好地提高光流估计的准确性，有效解决了光流场过于平滑、缺乏细粒度和忽略小物体运动等问题。

Abstract

Aiming at the problems of unclear target contour segmentation and poor granularity in existing optical flow estimation methods， an optical flow estimation ,via, fusing sequence image intensity correlation information is proposed. First， The coding features and contextual features of the images are extracted by the feature encoder and the global encoder， respectively， and the feature sizes are reduced by downsampling processing. Then， before constructing 4D correlation volume， the input two consecutive frames of feature maps are divided into regions to calculate dense visual similarity in the form of strong and weak correlation to build a more refined 4D correlation volume. Finally， in the iterative update stage， the residual convolution filter and the fine-grained module are proposed to be applied to process the correlation volume and optical flow transmission， respectively， which allows to retain more local small displacement information before fusing the correlation volume information and optical flow information. In comparison with other methods on the KITTI-2015 and MPI-Sintel， the optical flow estimation evaluation metric （Endpoint error， EPE） is improved by 8.2% and 6.15%， respectively. The network model given in this paper can better improve the accuracy of optical flow estimation and effectively solve the problems of the optical flow prediction field being over smooth， lacking of fine granularity and ignoring of small object motion.

关键词

计算机视觉深度学习光流区域匹配迭代更新

Keywords

computer visiondeep learningoptical flowregion matchingiterative update

references

DÉRIAN P， ALMAR R. Wavelet-based optical flow estimation of instant surface currents from shore-based and UAV videos ［J］. IEEE Transactions on Geoscience and Remote Sensing， 2017， 55（10）： 5790-5797. doi: 10.1109/tgrs.2017.2714202http://dx.doi.org/10.1109/tgrs.2017.2714202

李亚玮，金立左，孙长银，等. 基于光流约束自编码器的动作识别［J］. 东南大学学报（自然科学版），2017，47（4）：691-696． doi: 10.3969/j.issn.1001-0505.2017.04.011http://dx.doi.org/10.3969/j.issn.1001-0505.2017.04.011

LI Y W， JIN L Z， SUN C Y， et al. Action recognition based on optical flow constrained auto-encoder ［J］. Journal of Southeast University （Natural Science Edition）， 2017， 47（4）： 691-696. （in Chinese）. doi: 10.3969/j.issn.1001-0505.2017.04.011http://dx.doi.org/10.3969/j.issn.1001-0505.2017.04.011

张博，龙慧，刘刚. 基于特征约束与光流场模型的多通道视频目标跟踪算法［J］. 液晶与显示，2021，36（11）：1554-1564． doi: 10.37188/CJLCD.2021-0113http://dx.doi.org/10.37188/CJLCD.2021-0113

ZHANG B， LONG H， LIU G. Multi-channel video target tracking algorithm based on feature constraint and optical flow field model ［J］. Chinese Journal of Liquid Crystals and Displays， 2021， 36（11）： 1554-1564. （in Chinese）. doi: 10.37188/CJLCD.2021-0113http://dx.doi.org/10.37188/CJLCD.2021-0113

蒋欣兰，王胜春，罗四维，等. 车载前向运动视频的实时全景成像方法［J］. 光学学报，2017，37（5）：0515003． doi: 10.3788/aos201737.0515003http://dx.doi.org/10.3788/aos201737.0515003

JIANG X L， WANG S C， LUO S W， et al. Real-time panoramic imaging method for train-borne forward motion video ［J］. Acta Optica Sinica， 2017， 37（5）： 0515003. （in Chinese）. doi: 10.3788/aos201737.0515003http://dx.doi.org/10.3788/aos201737.0515003

HORN B K P， SCHUNCK B G. Determining optical flow ［J］. Artificial Intelligence， 1981， 17（1/3）： 185-203. doi: 10.1016/0004-3702(81)90024-2http://dx.doi.org/10.1016/0004-3702(81)90024-2

BLACK M J， ANANDAN P. A framework for the robust estimation of optical flow ［C］//Proceedings of the 4th International Conference on Computer Vision. Berlin： IEEE， 1993： 231-236.

ZACH C， POCK T， BISCHOF H. A duality based approach for realtime TV-L1 optical flow ［C］//Proceedings of the 29th Pattern Recognition Symposium. Heidelberg： Springer， 2007： 214-223. doi: 10.1007/978-3-540-74936-3_22http://dx.doi.org/10.1007/978-3-540-74936-3_22

WEINZAEPFEL P， REVAUD J， HARCHAOUI Z， et al. DeepFlow： large displacement optical flow with deep matching ［C］//Proceedings of 2013 IEEE International Conference on Computer Vision. Sydney： IEEE， 2013： 1385-1392. doi: 10.1109/iccv.2013.175http://dx.doi.org/10.1109/iccv.2013.175

SIMONYAN K， ZISSERMAN A. Two-stream convolutional networks for action recognition in videos ［C］//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal： MIT Press， 2014： 568-576.

ILG E， MAYER N， SAIKIA T， et al. FlowNet 2.0： Evolution of optical flow estimation with deep networks ［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu： IEEE， 2017： 1647-1655. doi: 10.1109/cvpr.2017.179http://dx.doi.org/10.1109/cvpr.2017.179

RANJAN A， BLACK M J. Optical flow estimation using a spatial pyramid network ［C］//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu： IEEE， 2017： 2720-2729. doi: 10.1109/cvpr.2017.291http://dx.doi.org/10.1109/cvpr.2017.291

SUN D Q， YANG X D， LIU M Y， et al. PWC-Net： CNNs for optical flow using pyramid， warping， and cost volume ［C］//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City： IEEE， 2018： 8934-8943. doi: 10.1109/cvpr.2018.00931http://dx.doi.org/10.1109/cvpr.2018.00931

YANG G S， RAMANAN D. Volumetric correspondence networks for optical flow ［C］//Proceedings of the 33rd International Conference on Neural Information Processing Systems. Vancouver： Curran Associates Inc.， 2019： 72.

HUR J， ROTH S. Iterative residual refinement for joint optical flow and occlusion estimation ［C］//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach： IEEE， 2019： 5747-5756. doi: 10.1109/cvpr.2019.00590http://dx.doi.org/10.1109/cvpr.2019.00590

LU Y， VALMADRE J， WANG H， et al. Devon： deformable volume network for learning optical flow ［C］//2020 IEEE Winter Conference on Applications of Computer Vision （WACV）. Snowmass： IEEE， 2020： 2694-2702. doi: 10.1109/wacv45572.2020.9093590http://dx.doi.org/10.1109/wacv45572.2020.9093590

AMOS B， KOLTER J Z. OptNet： differentiable optimization as a layer in neural networks ［C］//Proceedings of the 34th International Conference on Machine Learning. Sydney： JMLR.org， 2017： 136-145.

AGRAWAL A， AMOS B， BARRATT S， et al. Differentiable convex optimization layers ［C］//Proceedings of the 33rd International Conference on Neural Information Processing Systems. Vancouver： Curran Associates Inc.， 2019： 858.

TEED Z， DENG J. RAFT： Recurrent all-pairs field transforms for optical flow ［C］//Proceedings of the 16th European Conference on Computer Vision. Glasgow： Springer， 2020： 402-419. doi: 10.1007/978-3-030-58536-5_24http://dx.doi.org/10.1007/978-3-030-58536-5_24

GEIGER A， LENZ P， STILLER C， et al. Vision meets robotics： the KITTI dataset ［J］. International Journal of Robotics Research， 2013， 32（11）：1231-1237. doi: 10.1177/0278364913491297http://dx.doi.org/10.1177/0278364913491297

BUTLER D J， WULFF J， STANLEY G B， et al. A naturalistic open source movie for optical flow evaluation ［C］// Proceedings of European Conference on Computer Vision. Berlin， Heidelberg： ECCV， 2012： 611-625. doi: 10.1007/978-3-642-33783-3_44http://dx.doi.org/10.1007/978-3-642-33783-3_44

WANG J Y， ZHONG Y R， DAI Y C， et al. Displacement-invariant matching cost learning for accurate optical flow estimation ［C］//Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver： Curran Associates Inc.， 2020： 1276.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Single-objective tracking algorithm based on Siamese networks

Advances in twin network research in visual tracking technology

Infrared dim-small target detection under complex background based on attention mechanism

Related Author

No data

Related Institution

Xi'an Key Laboratory of Advanced Photo-electronics Materials and Energy Conversion Device， Xijing University

School of Artificial Intelligence， Optics and Electronics， Northwestern Polytechnical University

Institute of System Engineering， Academy of Military Sciences

Technological Institute of Materials & Energy Science （TIMES）， Xijing University

Beijing Xinghang Electromechanical Equipment Co.， Ltd.

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China 130033 Postal code：130033
Tel：0431-86176059 Email：yjxs@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰

Optical flow estimation via fusing sequence image intensity correlation information

DOI：10.37188/CJLCD.2022-0384

摘要

Abstract

关键词

Keywords

references