基于相位信息的立体图像匹配研究
|
摘要
三维立体视频系统作为最近出现的一种能够增强视频观赏体验的新兴媒体介质,成为视频应用的重点。其中,立体匹配是3D视频系统研究中的关键问题,通过像点获取对应点深度信息,实现三维立体再现的重要途径。带通相位信息可以反映图像本身结构的位置信息,对景物具有更强、更全面的解释能力。基于相位的立体匹配是立体视频系统研究中的一个重要方向。本论文的主要内容为对当今基于相位信息的立体匹配的研究总结以及算法的改进。
基于相位的立体匹配以傅立叶变换的平移不变性为依据,利用复数的带通时频滤波器,提取图像不同频段的相位响应为其本质特征。首先,本文选择具有平移不变性的双树复小波作为提取多个频段相位信息的工具。基于多分辨率相位信息的由粗到精的匹配模式,针对单一通道,提出将视差计算归结为一种求解匹配程度方程极大值的形式,隐式地抑制了相位匹配中最常见的两个问题——相位奇异与相位卷绕,我们将这种方法称作“BestMatch”匹配方法。BestMatch匹配可以基本反映立体图像对之间的视差信息,但对于边缘区域,以及纹理缺乏区域,则需要进一步的修正与改善。
由于单纯基于相位的匹配方法无法取得全局优化的视差结果,而且存在图像的色彩平面往往包含在同一视差平面里的假设,因此,我们将匹配问题转化为在视差空间中求解像素深度坐标的问题,通过引入了左右一致性检测,以及图像彩色分割算法和平面拟合算法,提出了一个完整的一维立体匹配框架,对BestMatch算法中得到的初始视差图进行判断、检测、填补、修正。大大改善了边缘区域与缺乏纹理区域的匹配精度,与近年其他的相位匹配算法结果相比较,也有比较好的提升。在视差图边缘区域,克服了基于相位的匹配方法中通常出现的毛刺、不清晰等问题。
最后,我们根据四元数以及四元小波的构造准则,利用双树复小波的张量积构造了四元小波,可以用以提取像对的四元相位,进而求取二维视差。我们给出了一个二维相位匹配的框架,来计算未校正图像对的稀疏视差场。实验结果证明,此方法可以比较准确的估计平移、旋转以及运动图像对之间的稀疏视差,在自由视立体匹配以及运动估计等方面均可以加以运用。
3D video system has recently emerged as a way to enhance the watching experience, which is becoming a focus of video applications. And stereo matching is one of the most active research areas in 3D video system, a very important way to obtain the depth information of the scene in the stereo images. Local phase could not only reflect the original image structure completely, but also resist the geometry and noises distortion. Therefore stereo matching approach based on phases is a key research direction. In this thesis, we investigated the main algorithms and presented some improved solutions on phase-based stereo matching.
Phase-based matching method is developed in recent two decades, which is based on Fourier shift invariance theorem. By using complex band-pass filters, we can get the phase information in different frequency bands. In the first part of this thesis, we choose Dual-Tree Complex Wavelet Transform (DT CWT), which is approximate shift invariance, to extract phases. A coarse-to-fine matching solution based on phase of different resolutions is presented, effectively inhibited the most two common problems in phase-based matching——singularities and phase wrap. We call it“BestMatch approach”. The results of this solution show the general disparity map between the stereo images. However, for the disparity edge areas and texture lacking areas, some improvements should be done to amend the bad estimated pixels.
As we mentioned, there are plenty of matching mistakes at edge areas and texture-lacking areas by just using phase information. To overcome this disadvantage, we introduced color segmentation and plane fitting, on the assumption that the scene structure can be approximated by a set of non-overlapping planes in the disparity space and that each plane is coincident with at least one homogeneous color segment in the reference image. Here, we present a whole stereo matching scheme, which includes two parts: phase-based matching and initial disparity amending. This new approach shows higher accuracy than most of the other recent phase-based algorithms, and the disparity at edge areas could be effectively shaped and clearly shown in the map.
At the last part of the thesis, we introduced the conceptions of quaternion and quaternion wavelets. And we constructed a kind of quaternion wavelets by using the tensor product of DT CWT to derive 2 dimensional disparity maps based on quaternion phases. A 2D sparse disparity estimation approach is proposed for uncalibrated stereo images in this part. The experimental results show that this method has accurate estimation and could play a role in free view disparity estimation as well as motion estimation.
基于相位的立体匹配以傅立叶变换的平移不变性为依据,利用复数的带通时频滤波器,提取图像不同频段的相位响应为其本质特征。首先,本文选择具有平移不变性的双树复小波作为提取多个频段相位信息的工具。基于多分辨率相位信息的由粗到精的匹配模式,针对单一通道,提出将视差计算归结为一种求解匹配程度方程极大值的形式,隐式地抑制了相位匹配中最常见的两个问题——相位奇异与相位卷绕,我们将这种方法称作“BestMatch”匹配方法。BestMatch匹配可以基本反映立体图像对之间的视差信息,但对于边缘区域,以及纹理缺乏区域,则需要进一步的修正与改善。
由于单纯基于相位的匹配方法无法取得全局优化的视差结果,而且存在图像的色彩平面往往包含在同一视差平面里的假设,因此,我们将匹配问题转化为在视差空间中求解像素深度坐标的问题,通过引入了左右一致性检测,以及图像彩色分割算法和平面拟合算法,提出了一个完整的一维立体匹配框架,对BestMatch算法中得到的初始视差图进行判断、检测、填补、修正。大大改善了边缘区域与缺乏纹理区域的匹配精度,与近年其他的相位匹配算法结果相比较,也有比较好的提升。在视差图边缘区域,克服了基于相位的匹配方法中通常出现的毛刺、不清晰等问题。
最后,我们根据四元数以及四元小波的构造准则,利用双树复小波的张量积构造了四元小波,可以用以提取像对的四元相位,进而求取二维视差。我们给出了一个二维相位匹配的框架,来计算未校正图像对的稀疏视差场。实验结果证明,此方法可以比较准确的估计平移、旋转以及运动图像对之间的稀疏视差,在自由视立体匹配以及运动估计等方面均可以加以运用。
3D video system has recently emerged as a way to enhance the watching experience, which is becoming a focus of video applications. And stereo matching is one of the most active research areas in 3D video system, a very important way to obtain the depth information of the scene in the stereo images. Local phase could not only reflect the original image structure completely, but also resist the geometry and noises distortion. Therefore stereo matching approach based on phases is a key research direction. In this thesis, we investigated the main algorithms and presented some improved solutions on phase-based stereo matching.
Phase-based matching method is developed in recent two decades, which is based on Fourier shift invariance theorem. By using complex band-pass filters, we can get the phase information in different frequency bands. In the first part of this thesis, we choose Dual-Tree Complex Wavelet Transform (DT CWT), which is approximate shift invariance, to extract phases. A coarse-to-fine matching solution based on phase of different resolutions is presented, effectively inhibited the most two common problems in phase-based matching——singularities and phase wrap. We call it“BestMatch approach”. The results of this solution show the general disparity map between the stereo images. However, for the disparity edge areas and texture lacking areas, some improvements should be done to amend the bad estimated pixels.
As we mentioned, there are plenty of matching mistakes at edge areas and texture-lacking areas by just using phase information. To overcome this disadvantage, we introduced color segmentation and plane fitting, on the assumption that the scene structure can be approximated by a set of non-overlapping planes in the disparity space and that each plane is coincident with at least one homogeneous color segment in the reference image. Here, we present a whole stereo matching scheme, which includes two parts: phase-based matching and initial disparity amending. This new approach shows higher accuracy than most of the other recent phase-based algorithms, and the disparity at edge areas could be effectively shaped and clearly shown in the map.
At the last part of the thesis, we introduced the conceptions of quaternion and quaternion wavelets. And we constructed a kind of quaternion wavelets by using the tensor product of DT CWT to derive 2 dimensional disparity maps based on quaternion phases. A 2D sparse disparity estimation approach is proposed for uncalibrated stereo images in this part. The experimental results show that this method has accurate estimation and could play a role in free view disparity estimation as well as motion estimation.
引文
[1] A. Redert, M. Op de Beeck, C. Fehn, W. IJsselsteijn, M. Pollefeys, L. Van Gool, E. Ofek, I. Sexton, and P. Surman,“ATTEST—advanced three-dimensional television system techniques,”in Proc. 3DPVT’02, Padova, Italy, Jun. 2002, pp. 313–319.
[2] C. Fehn, R. Barre, and R. S. Pastoor,“Interactive 3-D TV-Concepts and key technologies,”Proc. IEEE, vol. 94, no. 3, pp. 524–538, Mar.2006.
[3] D.Marr.Vision:A Computational Investigation into the Human Representation and Procession of Visual Information.W.H.Freeman,San Francisco,USA,1982
[4]M. Ziegler, L. Falkenhagen, R. Horst, and D. Kalivas,“Evolution of stereoscopic and three-dimensional video,”Signal Processing: Image Communication, vol. 14, pp. 173–194, 1998.
[5] C. Fehn,“A 3D-TV Approach Using Depth-Image-Based Rendering,”In Proceedings of Picture Coding Symposium, Dec 2004.
[6] Fleet D J, Jepson A D, Jenkin M R M. Phase-based disparity measurement [J]. CVGIP: Image Understanding, 1991, 53(2):198-210
[7] Fleet D J, Jepson A D. Stability of phase matching [J]. IEEE Trans PAMI, 1993, PAMI-15(12): 1253-1268.
[8] Maimone M.W. Characterizing stereo matching problems using local spatial frequency [D]. USA: Carnegie Mellon University, 1996.
[9] El-Etriby, S. Al-Hamadi, A, K. Michaelis, B. Dense Stereo Correspondence with Slanted Surface using Phase-based Algorithm [J]. IEEE ISIE, 2007, pp. 1807-1813
[10] U.R. Dhond, J.K. Aggarwal, Structure from stereo—a review, IEEE Transactions on Systems Management and Cybernetics [J]. 19,06(1989) pp.1489-1510
[11] Bamard S,Fischler M,Computational stereo[M].ACM Computer Surveys,1982,14(4): 553-572
[12] Faugeraus O,Laveau S,Robert L G,et al.3D reconstruction of urban scenesfrom sequences of images[J].Jun,1995,Technical Report,2572,INRIA
[13] Koschan, A. What is New in Computational Stereo Since 1989: A survey of Current Stereo Papers[R], Technical Report93-22. Univ. of Berlin, 1993
[14] D. Scharstein, R. Szeliski. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms. International Journal of Computer Vision, 2002, vol 47, pp.7–42
[15] M. Z. Brown, D Burschka and G D Hager. Advances in Computational Stereo [J]. TPAMI, 2005, 25(8).
[16] J. G. Robson, Receptive fields: Neural representation of the spatial and intensive attributes of the visual image [M], in Handbook of Perception, edited by EC Carterette and MP Friedman (Academic, New York, 1975), Vol. V. 9D.
[17] Julesz T B. Stereoscopic vision. Vision Res, 1986, 26: 1601-1012
[18] M.R.M. Jenkin, A. D. Jepson, J. K. Tsotsos. Techniques for disparity measurement [J]. CVGIP: image Understanding, 1991, 53(1): 14-30
[19] Sanger T. D. Stereo disparity computation using Gabor filters[J]. Biological Cybernetics, 1988, 59: 405-418
[20] J. Weng, Image matching using the windowed Fourier phase [J], International Journal of Computer Vision, vol. 11, no. 3, pp. 211-236, 1993
[21]游素亚,柳健,万发贯,基于小波变换相位基元的立体匹配系统[J];信息与控制;1994年05期
[22]徐彦君,杜利民,侯自强等.基于相位的尺度自适应立体匹配方法[J].电子学报, 1999, 27 (07): 38-41
[23] Yi Xu, Jun Zhou, Yuanhua Zhou. Disparity estimation and occlusion detection based on the relationship between phase-based stereo and area-based stereo [J]. Chinese Optics Letters, 2003, 1(8):494-496
[24]徐奕,周军,周源华.基于动态规划的相位匹配和遮挡检测[J].电子学报, 2004, 32(4): 591-595
[25]章毓晋.图像理解与计算机视觉[M].北京:清华大学出版社, 2000.
[26]徐奕,周军,周源华.立体视觉中的图像匹配技术[T].计算机工程与应用,2003, 39(5):1-5
[27] C. D. Kuglin, D. C. Hines. The phase correlation image alignment method[A]. In: Proc. of IEEE Int. Conf. on Cybernetics and Society[C]. New Yourk, 1975, 163-165.
[28] A. V. Oppenheim, J.S. Lim. The importance of phase in signals [A]. In: Proc. of the IEEE[C]. Gong, 1981, 69(5): 529-541.
[29]游素亚.立体视觉研究的现状与进展.中国图象图形学报, 1997, 2 (01): 17-24. 1996.
[30]吴立德.计算机视觉.上海:复旦大学出版社, 1993
[31] D.J.Fleet. Disparity from local weighted phase-correlation [A]. In: IEEE International Conference on Systems, Man and Cybernetics [C]. SanA ntonio, 1994, 48-56.
[32]余武荣,周军,周源华.采用小波变换的立体匹配:一种基于相位的方法[J].信号处理,1999, 15(4):321-324
[33] N.G. Kingsbury, The dual-tree complex wavelet transform: A new efficient tool for image restoration and enhancement [J], in Proc. European Signal Processing Conf., Rhodes, Sept. 1998, pp. 319–322.
[34] N.G. Kingsbury. The dual-tree complex wavelet transform: A new technique for shift invariance and directional filters [J], in Proc. 8th IEEE DSP Workshop, Utah, Aug. 9–12, 1998, paper no.86
[35] N.G. Kingsbury. Complex wavelets for shift invariant analysis and filtering of signals [J], Journal of Applied and Computational Harmonic Analysis, 2001, 10(3), pp.234-253
[36] N.G. Kingsbury. Design of q-shift complex wavelets for image processing using frequency domain energy minimization [C]. Proceedings of IEEE International Conference on Image Processing. Barcelona: IEEE, 2003: 1013-1016.
[37] Kingsbury,N.G. Shift Invariant Properties of the Dual-Tree Complex Wavelet Transformation [C], Proc. IEEE Conf. on Acoustics,Speech and Signal Processing, Phoenix, AZ,Paper SPTM 3.6,March 16-19, 1999.
[38] M.H. Ouali, D. Ziou, C. Laurgeau. Density and accuracy improvement of phase-based disparity[A]. In: Vision Interface’99[C]. Trois-Rivieres, Canada, 1999, 439-444
[39] L.D. Cai, J. Mayhew. A note on some phase differencing algorithms for disparity estimation [J]. Int. Journal of Computer Vision, 1997, 22(2): 111-124.
[40] F. Solari, S.P. Sabatini and G.M. Bisio. A fast technique for phase-based disparity estimation with no explicit calculation of phase. Electronics Letters 37, 1382 , 2001
[41] http://vision.middlebury.edu/stereo/
[42] D. Comaniciu and P. Meer. Mean shift: A robust approach toward feature space analysis. IEEE:PAMI, 24(5):603–619, May 2002.
[43] Fischler M, Bolles R. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography [J]. Graphics and Image Processing, 1981, 26(4): 381-395
[44] A software published by Triaxes Ltd, www.triaxes.com
[45] Thomas Bulow. Hypercomplex Spectral Signal Representations for the Processing and Analysis of Images [D]. Kiel: Institue for Informatik und Praktische Mathematik der Christian-Albrechts-Universitat zu Kiel, 1999
[46] E.B. Corrochano. The theory and use of Quaternion wavelet transform[EB/OL]. edb@gdl.cinvestav.mx, 2002
[47] Xu Y, Zhou J and Zhai G.T., 2D phase-based matching in uncalibrated images, IEEE workshop on signal processing systems-design and implementation, pp.325-330, Athens, Greece, 2005.
[48] Zhou J, Xu Y, Yang X.K., Quaternion wavelet phase based stereo matching for uncalibrated images, Pattern Recognition Letters, 2007, 28, pp.1509-1522
[49]徐奕,四元小波在图像分析与处理中的应用研究
[50] http://vasc.ri.cmu.edu//idb/html/stereo/
[51] http://i21www.ira.uka.de/image_sequences/
[2] C. Fehn, R. Barre, and R. S. Pastoor,“Interactive 3-D TV-Concepts and key technologies,”Proc. IEEE, vol. 94, no. 3, pp. 524–538, Mar.2006.
[3] D.Marr.Vision:A Computational Investigation into the Human Representation and Procession of Visual Information.W.H.Freeman,San Francisco,USA,1982
[4]M. Ziegler, L. Falkenhagen, R. Horst, and D. Kalivas,“Evolution of stereoscopic and three-dimensional video,”Signal Processing: Image Communication, vol. 14, pp. 173–194, 1998.
[5] C. Fehn,“A 3D-TV Approach Using Depth-Image-Based Rendering,”In Proceedings of Picture Coding Symposium, Dec 2004.
[6] Fleet D J, Jepson A D, Jenkin M R M. Phase-based disparity measurement [J]. CVGIP: Image Understanding, 1991, 53(2):198-210
[7] Fleet D J, Jepson A D. Stability of phase matching [J]. IEEE Trans PAMI, 1993, PAMI-15(12): 1253-1268.
[8] Maimone M.W. Characterizing stereo matching problems using local spatial frequency [D]. USA: Carnegie Mellon University, 1996.
[9] El-Etriby, S. Al-Hamadi, A, K. Michaelis, B. Dense Stereo Correspondence with Slanted Surface using Phase-based Algorithm [J]. IEEE ISIE, 2007, pp. 1807-1813
[10] U.R. Dhond, J.K. Aggarwal, Structure from stereo—a review, IEEE Transactions on Systems Management and Cybernetics [J]. 19,06(1989) pp.1489-1510
[11] Bamard S,Fischler M,Computational stereo[M].ACM Computer Surveys,1982,14(4): 553-572
[12] Faugeraus O,Laveau S,Robert L G,et al.3D reconstruction of urban scenesfrom sequences of images[J].Jun,1995,Technical Report,2572,INRIA
[13] Koschan, A. What is New in Computational Stereo Since 1989: A survey of Current Stereo Papers[R], Technical Report93-22. Univ. of Berlin, 1993
[14] D. Scharstein, R. Szeliski. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms. International Journal of Computer Vision, 2002, vol 47, pp.7–42
[15] M. Z. Brown, D Burschka and G D Hager. Advances in Computational Stereo [J]. TPAMI, 2005, 25(8).
[16] J. G. Robson, Receptive fields: Neural representation of the spatial and intensive attributes of the visual image [M], in Handbook of Perception, edited by EC Carterette and MP Friedman (Academic, New York, 1975), Vol. V. 9D.
[17] Julesz T B. Stereoscopic vision. Vision Res, 1986, 26: 1601-1012
[18] M.R.M. Jenkin, A. D. Jepson, J. K. Tsotsos. Techniques for disparity measurement [J]. CVGIP: image Understanding, 1991, 53(1): 14-30
[19] Sanger T. D. Stereo disparity computation using Gabor filters[J]. Biological Cybernetics, 1988, 59: 405-418
[20] J. Weng, Image matching using the windowed Fourier phase [J], International Journal of Computer Vision, vol. 11, no. 3, pp. 211-236, 1993
[21]游素亚,柳健,万发贯,基于小波变换相位基元的立体匹配系统[J];信息与控制;1994年05期
[22]徐彦君,杜利民,侯自强等.基于相位的尺度自适应立体匹配方法[J].电子学报, 1999, 27 (07): 38-41
[23] Yi Xu, Jun Zhou, Yuanhua Zhou. Disparity estimation and occlusion detection based on the relationship between phase-based stereo and area-based stereo [J]. Chinese Optics Letters, 2003, 1(8):494-496
[24]徐奕,周军,周源华.基于动态规划的相位匹配和遮挡检测[J].电子学报, 2004, 32(4): 591-595
[25]章毓晋.图像理解与计算机视觉[M].北京:清华大学出版社, 2000.
[26]徐奕,周军,周源华.立体视觉中的图像匹配技术[T].计算机工程与应用,2003, 39(5):1-5
[27] C. D. Kuglin, D. C. Hines. The phase correlation image alignment method[A]. In: Proc. of IEEE Int. Conf. on Cybernetics and Society[C]. New Yourk, 1975, 163-165.
[28] A. V. Oppenheim, J.S. Lim. The importance of phase in signals [A]. In: Proc. of the IEEE[C]. Gong, 1981, 69(5): 529-541.
[29]游素亚.立体视觉研究的现状与进展.中国图象图形学报, 1997, 2 (01): 17-24. 1996.
[30]吴立德.计算机视觉.上海:复旦大学出版社, 1993
[31] D.J.Fleet. Disparity from local weighted phase-correlation [A]. In: IEEE International Conference on Systems, Man and Cybernetics [C]. SanA ntonio, 1994, 48-56.
[32]余武荣,周军,周源华.采用小波变换的立体匹配:一种基于相位的方法[J].信号处理,1999, 15(4):321-324
[33] N.G. Kingsbury, The dual-tree complex wavelet transform: A new efficient tool for image restoration and enhancement [J], in Proc. European Signal Processing Conf., Rhodes, Sept. 1998, pp. 319–322.
[34] N.G. Kingsbury. The dual-tree complex wavelet transform: A new technique for shift invariance and directional filters [J], in Proc. 8th IEEE DSP Workshop, Utah, Aug. 9–12, 1998, paper no.86
[35] N.G. Kingsbury. Complex wavelets for shift invariant analysis and filtering of signals [J], Journal of Applied and Computational Harmonic Analysis, 2001, 10(3), pp.234-253
[36] N.G. Kingsbury. Design of q-shift complex wavelets for image processing using frequency domain energy minimization [C]. Proceedings of IEEE International Conference on Image Processing. Barcelona: IEEE, 2003: 1013-1016.
[37] Kingsbury,N.G. Shift Invariant Properties of the Dual-Tree Complex Wavelet Transformation [C], Proc. IEEE Conf. on Acoustics,Speech and Signal Processing, Phoenix, AZ,Paper SPTM 3.6,March 16-19, 1999.
[38] M.H. Ouali, D. Ziou, C. Laurgeau. Density and accuracy improvement of phase-based disparity[A]. In: Vision Interface’99[C]. Trois-Rivieres, Canada, 1999, 439-444
[39] L.D. Cai, J. Mayhew. A note on some phase differencing algorithms for disparity estimation [J]. Int. Journal of Computer Vision, 1997, 22(2): 111-124.
[40] F. Solari, S.P. Sabatini and G.M. Bisio. A fast technique for phase-based disparity estimation with no explicit calculation of phase. Electronics Letters 37, 1382 , 2001
[41] http://vision.middlebury.edu/stereo/
[42] D. Comaniciu and P. Meer. Mean shift: A robust approach toward feature space analysis. IEEE:PAMI, 24(5):603–619, May 2002.
[43] Fischler M, Bolles R. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography [J]. Graphics and Image Processing, 1981, 26(4): 381-395
[44] A software published by Triaxes Ltd, www.triaxes.com
[45] Thomas Bulow. Hypercomplex Spectral Signal Representations for the Processing and Analysis of Images [D]. Kiel: Institue for Informatik und Praktische Mathematik der Christian-Albrechts-Universitat zu Kiel, 1999
[46] E.B. Corrochano. The theory and use of Quaternion wavelet transform[EB/OL]. edb@gdl.cinvestav.mx, 2002
[47] Xu Y, Zhou J and Zhai G.T., 2D phase-based matching in uncalibrated images, IEEE workshop on signal processing systems-design and implementation, pp.325-330, Athens, Greece, 2005.
[48] Zhou J, Xu Y, Yang X.K., Quaternion wavelet phase based stereo matching for uncalibrated images, Pattern Recognition Letters, 2007, 28, pp.1509-1522
[49]徐奕,四元小波在图像分析与处理中的应用研究
[50] http://vasc.ri.cmu.edu//idb/html/stereo/
[51] http://i21www.ira.uka.de/image_sequences/