点云和视觉特征融合的增强现实装配系统三维跟踪注册方法
|
摘要
为了提高三维跟踪注册方法面向机械产品增强现实装配引导的适用性和鲁棒性,提出了一种点云和视觉特征融合的三维跟踪注册方法。首先利用参考模型点云对三维跟踪注册绝对坐标系进行定义,从而确定虚拟装配引导信息的定位基准。然后在迭代最近点法点云数据配准基础上,结合深度传感器彩色图像信息,通过视觉特征匹配,提高深度传感器快速移动时的跟踪注册过程鲁棒性。为了在此过程获取足够数量的视觉特征匹配点对,提出了一种基于方向向量一致性的视觉特征匹配策略。最后在跟踪注册过程加入基于关键帧的回环检测和全局位姿优化。实验结果表明:新方法精确性、实时性好,能达到每秒30帧。而且在相机快速移动时仍能表现出较好的鲁棒性,其综合性能优于基于点云的Kinect Fusion方法。
To improve the robustness and applicability of 3 D tracking and registration for augmented reality( AR)aided mechanical assembly system,a 3 D registration and tracking method based on the point cloud and visual features is proposed. Firstly,the reference model point cloud is used to definite absolute tracking coordinate system,thus the locating datum of the virtual assembly guidance information is determined. Then by adding visual features matching to the iterative closest points( ICP) registration process,the robustness of tracking and registration is improved. In order to obtain sufficient number of visual feature matching points in this process,a visual feature matching strategy based on orientation vector consistency is proposed. Finally,the loop closure detection and global pose optimization from key frames are added in the tracking registration process. The experimental result shows that the proposed method has good real-time performance and accuracy,and the running speed can reach 30 frames per second. Moreover,it also shows good robustness when the camera is moving fast and the depth information is inaccurate,and the comprehensive performance of the proposed method is better than the KinectFusion method.
To improve the robustness and applicability of 3 D tracking and registration for augmented reality( AR)aided mechanical assembly system,a 3 D registration and tracking method based on the point cloud and visual features is proposed. Firstly,the reference model point cloud is used to definite absolute tracking coordinate system,thus the locating datum of the virtual assembly guidance information is determined. Then by adding visual features matching to the iterative closest points( ICP) registration process,the robustness of tracking and registration is improved. In order to obtain sufficient number of visual feature matching points in this process,a visual feature matching strategy based on orientation vector consistency is proposed. Finally,the loop closure detection and global pose optimization from key frames are added in the tracking registration process. The experimental result shows that the proposed method has good real-time performance and accuracy,and the running speed can reach 30 frames per second. Moreover,it also shows good robustness when the camera is moving fast and the depth information is inaccurate,and the comprehensive performance of the proposed method is better than the KinectFusion method.
引文
[1]刘检华.军工数字化装配技术[J].国防制造技术,2011(4):5-7LIU Jianhua. Digital Assembly Technology in Military Industry[J]. Defense Manufacturing Technology,2011(4):5-7(in Chinese)
[2]肖鸿.面向复杂产品装配现场的移动三维模型关键技术研究[D].西安:西北工业大学,2014XIAO Hong. Research on the Key Technologies of Mobile Three-Dimensional Model for Assembly Site of Complex Product[D].Xi'an,Northwestern Polytechnical University,2014(in Chinese)
[3]陈祥辉.大部件人工装配作业现场信息投影技术研究[D].上海:华中科技大学,2016CHEN Xianghui. Research of Satellite Assembly Induced System Based on Augmented Reality[D]. Shanghai,Huazhong University of Science and Technology,2016(in Chinese)
[4] TOMBARI F,FRANCHI A,STEFANO L D. BOLD Features to Detect Textureless Objects[C]∥IEEE International Conference on Computer Vision Sydney,2014:1265-1272
[5] DAMEN D,BUNNUN P,CALWAY A,et al. Real-Time Learning and Detection of 3D Textureless Objects:a Scalable Approach[C]∥The British Machine Vision Conference,Guildford,2012:1-12
[6] WANG G,WANG B,ZHONG F,et al. Global Optimal Searching for Textureless 3D Object Tracking[J]. Visual Computer,2015,31(6/7/8):979-988
[7] WANG Y,ZHANG S S,YANG S,et al. A LINE-MOD-Based Markerless Tracking Approach for AR Applications[J]. International Journal of Advanced Manufacturing Technology,2017,89(5/6/7/8):1699-1707
[8]徐迟.增强现实中的三维物体注册方法及其应用研究[D].上海:华中科技大学,2011XU Chi. Research on the 3D Objeet Registration Method in Augmented Reality and Its Application[D]. Shanghai,Huazhong University of Science and Technology,2011(in Chinese)
[9] ENGEL J,STCKLER J,CREMERS D. Large-Scale Direct SLAM with Stereo Cameras[C]∥IEEE/RSJ International Conference on Intelligent Robots and Systems,Hamburg,2015:1935-1942
[10]MUR-ARTAL R,MONTIEL J M M,TARDS J D. ORB-SLAM:a Versatile and Accurate Monocular SLAM System[J]. IEEE Trans on Robotics,2015,31(5):1147-1163
[11]付梦印,吕宪伟,刘彤,等.基于RGB-D数据的实时SLAM算法[J].机器人,2015,37(6):683-692FU Mengyin,LXianwei,LIU Tong,et al. Real-time SLAM Algorithm Based on RGB-D Data[J]. Robot,2015,37(6):683-692
[12]RADKOWSKI R. Object Tracking with a Range Camera for Augmented Reality Assembly Assistance[J]. Journal of Computing and Information Science in Engineering,2016,16(1):011004
[13]刘鑫辰,傅慧源,马华东.基于RGB-D摄像头的实时手指跟踪注册与手势识别[J].计算机科学,2014,41(10):50-52LIU Xinchen,FU Huiyuan,MA Huadong,et al. Real-time Fingertip Tracking and Gesture Recognition Using RGB-D Camera[J]. Computer Science,2014,41(10):50-52(in Chinese)
[14]MATSUO T,FUKUSHIMA N,ISHIBASHI Y. Weighted Joint Bilateral Filter with Slope Depth Compensation Filter for Depth Map Refinement[C]∥International Conference on Computer Vision Theory and Applications,Berlin,2015:313-320
[15]BIAN J W,LIN W Y,MATSUSHITA Y,et al. GMS:Grid-Based Motion Statistics for Fast,Ultra-Robust Feature Correspondence[C]∥2017 IEEE Conference on Computer Vision and Pattern Recognition,Puerto Rico,2017:2828-2837
[16]沈跃,潘成凯,刘慧,等.基于改进SIFT-ICP算法的Kinect植株点云配准方法[J].农业机械学报,2017(12):183-189SHEN Yue,PAN Chengkai,LIU Hui,et al. Method of Plant Point Cloud Registration Based on Kinect of Improved SIFT-ICP[J]. Transactions of the Chinese Society of Agricultural,2017(12):183-189(in Chinese)
[17]HENRY P,KRAININ M,HERBST E,et al. RGB-D Mapping:Using Depth Cameras for Dense 3D Modeling of Indoor Environments[M]. Berlin Heidebery,Springer,2014:647-663
[18]李同,张奇志.基于ORB词袋模型的SLAM回环检测研究[J].信息通信,2017(10):20-25LI Tong,ZHANG Qizhi. Research of SLAM Loop-Closure Based on ORB Bag of Words[J]. Information&Communications,2017(10):20-25(in Chinese)
[19]NEWCOMBE R A,IZADI S,HILLIGES O,et al. Kinect Fusion:Real-Time Dense Surface Mapping and Tracking[C]∥IEEE International Symposium on Mixed and Augmented Reality,Basel,2011:127-136
[20]WANG Y,ZHANG S S,YANG S,et al. Mechanical Assembly Assistance Using Markerless Augmented Reality System[J]. Assembly Automation,2018,38(1):77-87
[21]徐迟,李世其,王峻峰,等.面向增强现实装配的虚实遮挡技术研究[J].机械设计与制造,2009(12):256-258XU Chi,LI Shiqi,WANG Junfeng,et al. Ocelusion Handling in Augmented Reality Based Virtuecl Assembly[J]. Machinery Oesign&Manufaeture,2009(12):256-258(in Chinese)
[2]肖鸿.面向复杂产品装配现场的移动三维模型关键技术研究[D].西安:西北工业大学,2014XIAO Hong. Research on the Key Technologies of Mobile Three-Dimensional Model for Assembly Site of Complex Product[D].Xi'an,Northwestern Polytechnical University,2014(in Chinese)
[3]陈祥辉.大部件人工装配作业现场信息投影技术研究[D].上海:华中科技大学,2016CHEN Xianghui. Research of Satellite Assembly Induced System Based on Augmented Reality[D]. Shanghai,Huazhong University of Science and Technology,2016(in Chinese)
[4] TOMBARI F,FRANCHI A,STEFANO L D. BOLD Features to Detect Textureless Objects[C]∥IEEE International Conference on Computer Vision Sydney,2014:1265-1272
[5] DAMEN D,BUNNUN P,CALWAY A,et al. Real-Time Learning and Detection of 3D Textureless Objects:a Scalable Approach[C]∥The British Machine Vision Conference,Guildford,2012:1-12
[6] WANG G,WANG B,ZHONG F,et al. Global Optimal Searching for Textureless 3D Object Tracking[J]. Visual Computer,2015,31(6/7/8):979-988
[7] WANG Y,ZHANG S S,YANG S,et al. A LINE-MOD-Based Markerless Tracking Approach for AR Applications[J]. International Journal of Advanced Manufacturing Technology,2017,89(5/6/7/8):1699-1707
[8]徐迟.增强现实中的三维物体注册方法及其应用研究[D].上海:华中科技大学,2011XU Chi. Research on the 3D Objeet Registration Method in Augmented Reality and Its Application[D]. Shanghai,Huazhong University of Science and Technology,2011(in Chinese)
[9] ENGEL J,STCKLER J,CREMERS D. Large-Scale Direct SLAM with Stereo Cameras[C]∥IEEE/RSJ International Conference on Intelligent Robots and Systems,Hamburg,2015:1935-1942
[10]MUR-ARTAL R,MONTIEL J M M,TARDS J D. ORB-SLAM:a Versatile and Accurate Monocular SLAM System[J]. IEEE Trans on Robotics,2015,31(5):1147-1163
[11]付梦印,吕宪伟,刘彤,等.基于RGB-D数据的实时SLAM算法[J].机器人,2015,37(6):683-692FU Mengyin,LXianwei,LIU Tong,et al. Real-time SLAM Algorithm Based on RGB-D Data[J]. Robot,2015,37(6):683-692
[12]RADKOWSKI R. Object Tracking with a Range Camera for Augmented Reality Assembly Assistance[J]. Journal of Computing and Information Science in Engineering,2016,16(1):011004
[13]刘鑫辰,傅慧源,马华东.基于RGB-D摄像头的实时手指跟踪注册与手势识别[J].计算机科学,2014,41(10):50-52LIU Xinchen,FU Huiyuan,MA Huadong,et al. Real-time Fingertip Tracking and Gesture Recognition Using RGB-D Camera[J]. Computer Science,2014,41(10):50-52(in Chinese)
[14]MATSUO T,FUKUSHIMA N,ISHIBASHI Y. Weighted Joint Bilateral Filter with Slope Depth Compensation Filter for Depth Map Refinement[C]∥International Conference on Computer Vision Theory and Applications,Berlin,2015:313-320
[15]BIAN J W,LIN W Y,MATSUSHITA Y,et al. GMS:Grid-Based Motion Statistics for Fast,Ultra-Robust Feature Correspondence[C]∥2017 IEEE Conference on Computer Vision and Pattern Recognition,Puerto Rico,2017:2828-2837
[16]沈跃,潘成凯,刘慧,等.基于改进SIFT-ICP算法的Kinect植株点云配准方法[J].农业机械学报,2017(12):183-189SHEN Yue,PAN Chengkai,LIU Hui,et al. Method of Plant Point Cloud Registration Based on Kinect of Improved SIFT-ICP[J]. Transactions of the Chinese Society of Agricultural,2017(12):183-189(in Chinese)
[17]HENRY P,KRAININ M,HERBST E,et al. RGB-D Mapping:Using Depth Cameras for Dense 3D Modeling of Indoor Environments[M]. Berlin Heidebery,Springer,2014:647-663
[18]李同,张奇志.基于ORB词袋模型的SLAM回环检测研究[J].信息通信,2017(10):20-25LI Tong,ZHANG Qizhi. Research of SLAM Loop-Closure Based on ORB Bag of Words[J]. Information&Communications,2017(10):20-25(in Chinese)
[19]NEWCOMBE R A,IZADI S,HILLIGES O,et al. Kinect Fusion:Real-Time Dense Surface Mapping and Tracking[C]∥IEEE International Symposium on Mixed and Augmented Reality,Basel,2011:127-136
[20]WANG Y,ZHANG S S,YANG S,et al. Mechanical Assembly Assistance Using Markerless Augmented Reality System[J]. Assembly Automation,2018,38(1):77-87
[21]徐迟,李世其,王峻峰,等.面向增强现实装配的虚实遮挡技术研究[J].机械设计与制造,2009(12):256-258XU Chi,LI Shiqi,WANG Junfeng,et al. Ocelusion Handling in Augmented Reality Based Virtuecl Assembly[J]. Machinery Oesign&Manufaeture,2009(12):256-258(in Chinese)