基于光流传感器的视频稳像技术

周鹏威, 季元吉, 董超, 等. 基于光流传感器的视频稳像技术[J]. 光电工程, 2019, 46(11): 180581. doi: 10.12086/oee.2019.180581
引用本文: 周鹏威, 季元吉, 董超, 等. 基于光流传感器的视频稳像技术[J]. 光电工程, 2019, 46(11): 180581. doi: 10.12086/oee.2019.180581
Zhou Pengwei, Ji Yuanji, Dong Chao, et al. Video stabilization technique based on optical flow sensor[J]. Opto-Electronic Engineering, 2019, 46(11): 180581. doi: 10.12086/oee.2019.180581
Citation: Zhou Pengwei, Ji Yuanji, Dong Chao, et al. Video stabilization technique based on optical flow sensor[J]. Opto-Electronic Engineering, 2019, 46(11): 180581. doi: 10.12086/oee.2019.180581

基于光流传感器的视频稳像技术

  • 基金项目:
    国家自然科学基金资助项目(61505192);浙江省自然科学基金资助项目(LQ15F050004,LY20F050008);国家海洋局南海维权技术与应用重点实验室开放基金项目(1705)
详细信息
    作者简介:
    通讯作者: 董超(1982-),男,博士,副研究员,主要从事海洋无人艇研制与应用、海上运动目标跟踪与识别等的研究。E-mail:dongchaoxj888@126.com
  • 中图分类号: TP391.4

Video stabilization technique based on optical flow sensor

  • Fund Project: Supported by National Natural Science Foundation of China (61505192), Natural Science Foundation of Zhejiang Province (LQ15F050004, LY20F050008), and Open Foundation of Key Laboratory of Technology and Application for Safeguarding of Marine Rights and Interests, SOA (1705)
More Information
  • 针对平台运动导致的视频抖动问题,提出了一种基于光流传感器的视频稳像技术。该方案首先通过对一般光流传感器的改进,使其具有旋转运动下输出准确运动矢量的能力,然后利用光流传感器获得相邻两帧图像之间的运动矢量,并通过坐标变换计算出主相机的实时平移和旋转信息;其次,对原视频图像序列进行运动补偿,以获得稳定的图像序列,最终实现了视频稳像。实验结果表明,稳像后的图像序列与未稳像之前相比峰值信噪比提高了11.86 dB。该方案在视频抖动较大的情况下,能够明显减小图像序列的抖动现象,具有稳像效果好的特点,满足视频稳像的性能要求,对提高平台抗扰能力有着较高的实用性。

  • Overview: Video is one of the most intuitive and effective information carrier in the field of machine vision. Through the analysis of video information, the machine can perceive the surrounding environment and make relevant decisions. However, in the field of camera platform, such as shipborne, on-vehicle and hand-held, the bumping and jitter of the camera system which is due to the attitude change and vibration of the platform will cause the instability of the video sequences. It will seriously affect the information collection and decision for surrounding environment situation. Therefore, the video stabilization has important significance and value in practical engineering applications. The traditional video stabilization technique has mechanical image stabilization, optical image stabilization and electronic image stabilization. Mechanical image stabilization detects motion information and compensates the offset of video sequences by motor calibration system; optical image stabilization achieves image stabilization by adjusting the structure of the optical element to change the optical path and compensate the offset of the optical axis. The first two image stabilization technique are often expensive, difficult, and complicated to operate, which limits their application in small-sized and low-cost field of camera. The electronic image stabilization technique obtains the motion vectors of the video sequences by motion analysis and compensates the motion to obtain stable video sequences. However, this method tends to have a large amount of calculation and low precision, which limits its application range. In allusion of the above video stabilization problem, a video stabilization technique based on optical flow sensor is presented. Firstly, the scheme makes the general optical flow sensor overcome the shortcomings of the SAD algorithm under rotational motion and output accurate motion vectors. In addition, since the main camera and the optical flow sensor have a fixed spatial positional relationship, the real-time translation and rotation information of the main camera is calculated through coordinate transformation with the motion vectors obtained by the optical flow sensor. Then, the original unstable video sequences are compensated by frame, and finally the video stabilization is realized. Experimental results indicate that, compared with the unstable image, the peak signal-to-noise ratio (PSNR) is increased by 11.86 dB, and compared with the electronic image stabilization algorithm based on feature point matching, the processing time is decreased by 54.65% with similar image stabilization effect. In the case of obvious video jitter, the scheme can significantly reduce the jitter between video sequences. The method which has the characteristics of salutary video stabilization and high speed can meet the performance requirements of video stabilization and improve the capacity of disturbance resistance for platform.

  • 加载中
  • 图 1  光流传感器结构框图

    Figure 1.  Block diagram of optical flow sensor

    图 2  光流传感器改进原理示意图。(a)改进之前图像;(b)旋转校正后图像;(c)剪切后图像

    Figure 2.  Framework of improvement principle of optical flow sensor. (a) Before improvement; (b) After rotation correction; (c) After sheared

    图 3  光流传感器改进前后的输出图像。(a)改进之前的图像;(b)改进之后的图像

    Figure 3.  Image before and after improvement of optical flow sensor. (a) Before improvement; (b) After improvement

    图 4  光流传感器的采样数据结果

    Figure 4.  Results of sampling data for optical flow sensor

    图 5  实验装置实物图

    Figure 5.  Physical diagram of experimental device

    图 6  两相机的运动和计算模型。(a)相机实际运动模型;(b)平移量计算模型

    Figure 6.  Motion and computation model of two cameras. (a) Camera real motion model; (b) Translation computation model

    图 7  实验稳像的结果图。(a)第1帧图像;(b)第9帧图像;(c)第10帧图像;(d)第13帧图像;(e)第16帧图像;(f)第9帧补偿后图像;(g)第10帧补偿后图像;(h)第13帧补偿后图像;(i)第16帧补偿后图像

    Figure 7.  Results of experimental image stabilization. (a) First frame of image; (b) Ninth frame of image; (c) Tenth frame of image; (d) Thirteenth frame of image; (e) Sixteenth frame of image; (f) Ninth frame compensated image; (g) Tenth frame compensated image; (h) Thirteenth frame compensated image; (i) Sixteenth frame compensated image

    图 8  灰度列投影曲线。(a)第1帧和第16帧截取图像的灰度投影曲线;(b)第1帧和经旋转校正后的第16帧截取图像的灰度投影曲线;(c)第1帧和经旋转平移补偿后的第16帧截取图像的灰度投影曲线

    Figure 8.  Gray column projection curve. (a) Gray column projection curve of intercepted image in first and sixteenth frames; (b) Gray column projection curve of intercepted image in first frame and sixteenth frame after rotation correction; (c) Gray column projection curve of intercepted image in first frame and sixteenth frame after rotation translation compensation

    图 9  差值图像比较。(a)未稳像差值图;(b)稳像后差值图

    Figure 9.  Comparison of difference image. (a) Original difference image; (b) Stabilized difference image

    表 1  视频图像序列的运动补偿量

    Table 1.  Motion compensation data of video image sequence

    帧号 旋转角度/rad x轴平移量 y轴平移量
    2 0.00043 0 0
    5 0.00109 4 -2
    10 0.01932 29 40
    15 0.17906 -118 311
    20 0.11421 -68 210
    25 0.01871 22 31
    30 0.05589 -2 108
    35 0.09805 -32 243
    40 0.00885 36 19
    45 0.09153 -25 188
    50 0.18182 -112 320
    55 0.01479 36 32
    60 0.06017 6 134
    65 0.12485 -35 253
    70 0.00593 35 14
    75 0.13505 -49 307
    80 0.05299 -8 103
    85 0.00925 22 20
    90 0.19287 -129 348
    95 0.03074 2 7
    100 0.02645 6 64
    下载: 导出CSV

    表 2  算法性能比较

    Table 2.  Performance comparison of algorithm

    特征点匹配算法 本文方法 未稳像前
    时间/s 标准差 时间/s 标准差 标准差
    8.569 12.44 3.886 11.73 21.18
    下载: 导出CSV
  • [1]

    刘文.船载移动视频电子稳像方法研究[D].大连: 大连海事大学, 2017.

    Liu W. Research on the methods of electronic image stabilization for shipborne mobile video[D]. Dalian: Dalian Maritime University, 2017.http://cdmd.cnki.com.cn/Article/CDMD-10151-1017209269.htm

    [2]

    张贝.基于特征提取与灰度投影联合稳像的算法研究[D].哈尔滨: 哈尔滨工程大学, 2010.

    Zhang B. Research on united image stabilization based on feature tracking and gray projection[D]. Harbin: Harbin Engineering University, 2010.

    [3]

    冯筠, 延瑜瑜, 赵妍, 等.基于学习不变特征变换的兵马俑图像分区匹配[J].光学 精密工程, 2018, 26(7): 1774–1783. http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201807024

    Feng J, Yan Y Y, Zhao Y, et al. A terracotta image partition matching method based on learned invariant feature transform[J]. Optics and Precision Engineering, 2018, 26(7): 1774–1783. http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201807024

    [4]

    曾峦, 翟优, 谭久彬.基于SIFT的自动匹配策略[J].光电工程, 2011, 38(2): 65–70. doi: 10.3969/j.issn.1003-501X.2011.02.011

    Zeng L, Zhai Y, Tan J B. Automatic matching strategy based on SIFT[J]. Opto-Electronic Engineering, 2011, 38(2): 65–70. doi: 10.3969/j.issn.1003-501X.2011.02.011

    [5]

    Honegger D, Meier L, Tanskanen P, et al. An open source and open hardware embedded metric optical flow CMOS camera for indoor and outdoor applications[C]//2013 IEEE International Conference on Robotics and Automation, Karlsruhe, 2013: 1736–1741.

    [6]

    龙达峰, 刘俊, 张晓明, 等.基于椭球拟合的三轴陀螺仪快速标定方法[J].仪器仪表学报, 2013, 34(6): 1299–1305. doi: 10.3969/j.issn.0254-3087.2013.06.015

    Long D F, Liu J, Zhang X M, et al. Triaxial gyroscope fast calibration method based on ellipsoid fitting[J]. Chinese Journal of Scientific Instrument, 2013, 34(6): 1299–1305. doi: 10.3969/j.issn.0254-3087.2013.06.015

    [7]

    纪慧中, 贾大宇, 董恩清, 等.基于图像特征和光流场的非刚性图像配准[J].光学 精密工程, 2017, 25(9): 2469–2482. http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201709026

    Ji H Z, Jia D Y, Dong E Q, et al. Non-rigid registrations based on image characteristics and optical flows[J]. Optics and Precision Engineering, 2017, 25(9): 2469–2482. http://d.old.wanfangdata.com.cn/Periodical/gxjmgc201709026

    [8]

    Lan J H, Li J, Hu G D, et al. Vehicle speed measurement based on gray constraint optical flow algorithm[J]. Optik, 2014, 125(1): 289–295. doi: 10.1016/j.ijleo.2013.06.036

    [9]

    Huang R M, Li J Q, Shi M Y, et al. Quadrotor hovering scheme based on improved optical flow[C]//2017 Chinese Automation Congress (CAC), Jinan, 2017: 2357–2362.

    [10]

    Tsai S E, Zhuang S H. Optical flow sensor integrated navigation system for quadrotor in GPS-denied environment[C]//2016 International Conference on Robotics and Automation Engineering (ICRAE), Jeju, 2016: 87–91.

    [11]

    熊晶莹.基于特征提取与匹配的车载电子稳像方法研究[D].长春: 中国科学院长春光学精密机械与物理研究所, 2017.

    Xiong J Y. Study of electronic image stabilization based on feature detection and description for vehicles[D]. Changchun: Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, 2017.http://cdmd.cnki.com.cn/Article/CDMD-80139-1017064025.htm

    [12]

    郑晓锋, 陈跃庭, 徐之海, 等.一种补偿平移与旋转运动的快速电子稳像算法[J].光子学报, 2008, 37(9): 1890–1894. http://d.old.wanfangdata.com.cn/Periodical/gzxb200809040

    Zheng X F, Chen Y T, Xu Z H, et al. A fast electronic image stabilization algorithm for translational and rotational motion compensation[J]. Acta Photonica Sinica, 2008, 37(9): 1890–1894. http://d.old.wanfangdata.com.cn/Periodical/gzxb200809040

    [13]

    吕高杰, 张国华, 车宏.电子稳像算法精度评价方法研究[J].电光与控制, 2011, 18(3): 76–79. doi: 10.3969/j.issn.1671-637X.2011.03.019

    Lv G J, Zhang G H, Che H. A method for evaluating the accuracy of electronic image stabilization system and algorithm[J]. Electronics Optics & Control, 2011, 18(3): 76–79. doi: 10.3969/j.issn.1671-637X.2011.03.019

  • 加载中

(9)

(2)

计量
  • 文章访问数:  7465
  • PDF下载数:  2413
  • 施引文献:  0
出版历程
收稿日期:  2018-11-12
修回日期:  2019-05-09
刊出日期:  2019-11-01

目录

/

返回文章
返回