面向增强现实的移动视点下室外视频的阴影检测
|
摘要
针对增强现实中移动视点下的室外场景检测问题,提出一种基于光流跟踪的阴影在线检测算法.首先将当前帧中待检测的阴影分为将前一帧阴影跟踪过来的阴影部分和因视点改变出现的新阴影部分;然后在GraphCut框架下对跟踪结果进行优化,以消除光流跟踪产生的累积误差;再利用阴影区域的光照特点,如太阳光与天空光强度比值等,提取优化后的跟踪阴影区域的阴影特征,在线地检测新进的阴影区域;最后再次利用Graph Cut算法对检测到的初始新阴影区域进行优化,消除基于阈值的方法检测独立新进阴影时产生的误差.在Matlab平台下对不同复杂度的室外场景进行实验的结果表明,该算法不仅能有效地减少跟踪阴影的误差,而且可准确地检测到室外场景中经常存在的软影.
To detect shadows from a live video captured with a free moving camera in augmented reality(AR), an online algorithm based on optical flow tracking is proposed in this paper. Firstly, the algorithm divides shadows in the current frame into the shadows tracked from previous frame with the optical flow tracking algorithm, and the new shadow caused by the camera movement. To alleviate the inevitably cumulative errors produced in tracking process, we refine tracking results under the framework of Graph Cut optimization. Observing that the tracked and new shadow of a frame should have the same illumination condition such as the same magnitude ratio between the sunlight and skylight, the shadow features from tracked shadows after optimization is extracted to detect new shadows. Finally, to eliminate the error of detecting isolated new shadows based on the thresholding method, the Graph Cut optimization is utilized again to refine new detected shadows. The experimental results demonstrate that the algorithm significantly reduces the cumulative error of tracking. Meanwhile, it is able to accurately detect the soft shadows frequently occurring in outdoor scenes.
To detect shadows from a live video captured with a free moving camera in augmented reality(AR), an online algorithm based on optical flow tracking is proposed in this paper. Firstly, the algorithm divides shadows in the current frame into the shadows tracked from previous frame with the optical flow tracking algorithm, and the new shadow caused by the camera movement. To alleviate the inevitably cumulative errors produced in tracking process, we refine tracking results under the framework of Graph Cut optimization. Observing that the tracked and new shadow of a frame should have the same illumination condition such as the same magnitude ratio between the sunlight and skylight, the shadow features from tracked shadows after optimization is extracted to detect new shadows. Finally, to eliminate the error of detecting isolated new shadows based on the thresholding method, the Graph Cut optimization is utilized again to refine new detected shadows. The experimental results demonstrate that the algorithm significantly reduces the cumulative error of tracking. Meanwhile, it is able to accurately detect the soft shadows frequently occurring in outdoor scenes.
引文
[1]Jacobs K,Loscos C.Classification of illumination methods for mixed reality[J].Computer Graphics Forum,2006,25(1):29-51
[2]Zhao Q P.A survey on virtual reality[J].Science in China Series F:Information Sciences,2009,52(3):348-400
[3]Xu Weipeng,Wang Yongtian,Liu Yue,et al.Survey on occlusion handling in augmented reality[J].Journal of Computer-Aided Design&Computer Graphics,2013,25(11):1635-1642(in Chinese)(徐维鹏,王涌天,刘越,等.增强现实中的虚实遮挡处理综述[J].计算机辅助设计与图形学学报,2013,25(11):1635-1642)
[4]Dai J Y,Han D Y.Region-based moving shadow detection using affinity propagation[J].International Journal of Signal Processing,Image Processing and Pattern Recognition,2015,8(3):65-74
[5]Bullkich E,Ilan I,Moshe Y,et al.Moving shadow detection by nonlinear Tone-Mapping[C]//Proceedings of the 19th International Conference on Systems,Signals and Image Processing.Los Alamitos:IEEE Computer Society Press,2012:146-149
[6]Nadimi S,Bhanu B.Physical models for moving shadow and object detection in video[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2004,26(8):1079-1087
[7]Jacobs K,Nahmias J D,Angus C,et al.Automatic generation of consistent shadows for augmented reality[C]//Proceedings of Graphics Interface.Waterloo:Canadian Human-Computer Communications Society,2005:113-120
[8]Lu B V,Kakuta T,Kawakami R,et al.Foreground and shadow occlusion handling for outdoor augmented reality[C]//Proceedings of the IEEE International Symposium on Mixed and Augmented Reality.Los Alamitos:IEEE Computer Society Press,2010:109-119
[9]Cornelis K,Pollefeys M,Vergauwen M,et al.Augmented reality using uncalibrated video sequences[C]//Proceedings of EuEuropean Workshop on 3D Structure from Multiple Images of Large-Scale Environments.Heidelberg:Springer,2001,2018:144-160
[10]Debevec P.Rendering synthetic objects into real scenes:bridging traditional and image-based graphics with global illumination and high dynamic range photography[C]//Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques.New York:ACM Press,1998:189-198
[11]Kanbara M,Yokoya N.Real-time estimation of light source environment for photorealistic augmented reality[C]//Proceedings of the 17th International Conference on Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2004:911-914
[12]Martel-Brisson N,Zaccarin A.Moving cast shadow detection from a Gaussian mixture shadow model[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2005,(2):643-648
[13]Nadimi S,Bhanu B.Physical models for moving shadow and object detection in video[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2004,26(8):1079-1087
[14]Leone A,Distante C.Shadow detection for moving objects based on texture analysis[J].Pattern Recognition,2007,40(4):1222-1233
[15]Stander J,Mech R,Ostermann J.Detection of moving cast shadows for object segmentation[J].IEEE Transactions on Multimedia,1999,1(1):65-76
[16]Finlayson G,Fredembach C,Drew M S.Detecting illumination in images[C]//Proceedings of the 11th IEEE International Conference on Computer Vision.Los Alamitos:IEEE Computer Society Press,2007:1-8
[17]Finlayson G D,Hordley S D,Drew M S.Removing shadows from images[C]//Proceedings of the 7th European Conference on Computer Vision.Heidelberg:Springer,2002:823-836
[18]Khan S H,Bennamoun M,Sohel F,et al.Automatic feature learning for robust shadow detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2014:1939-1946
[19]Khan S H,Bennamoun M,Sohel F,et al.Automatic shadow detection and removal from a single image[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(3):431-446
[20]Macedo M C F,Nascimento V P,Souza A C S.Real-time shadow detection using multi-channel binarization and noise removal[J].Journal of Real-Time Image Processing,2018:1-14
[21]Newey C C,Jones O D,Dee H M.Shadow detection for mobile robots:features,evaluation,and datasets[J].Spatial Cognition&Computation,2018,18(2):115-137
[22]Bao L C,Yang Q X,Jin H L.Fast edge-preserving PatchMatch for large displacement optical flow[J].IEEE Transactions on Image Processing,2014,23(12):4996-5006
[23]Barrow H G,Tenenbaum J M.Recovering intrinsic scene characteristics from images[M]//Hanson A,Riseman E.Computer Vision Systems.New York:Academic Press,1978:3-26
[24]Levin A,Lischinski D,Weiss Y.A closed-form solution to natural image matting[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2008,30(2):228-242
[25]Xing G Y,Zhou X H,Peng Q S,et al.Lighting simulation of augmented outdoor scene based on a legacy photograph[J].Computer Graphics Forum,2013,32(7):101-110
[2]Zhao Q P.A survey on virtual reality[J].Science in China Series F:Information Sciences,2009,52(3):348-400
[3]Xu Weipeng,Wang Yongtian,Liu Yue,et al.Survey on occlusion handling in augmented reality[J].Journal of Computer-Aided Design&Computer Graphics,2013,25(11):1635-1642(in Chinese)(徐维鹏,王涌天,刘越,等.增强现实中的虚实遮挡处理综述[J].计算机辅助设计与图形学学报,2013,25(11):1635-1642)
[4]Dai J Y,Han D Y.Region-based moving shadow detection using affinity propagation[J].International Journal of Signal Processing,Image Processing and Pattern Recognition,2015,8(3):65-74
[5]Bullkich E,Ilan I,Moshe Y,et al.Moving shadow detection by nonlinear Tone-Mapping[C]//Proceedings of the 19th International Conference on Systems,Signals and Image Processing.Los Alamitos:IEEE Computer Society Press,2012:146-149
[6]Nadimi S,Bhanu B.Physical models for moving shadow and object detection in video[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2004,26(8):1079-1087
[7]Jacobs K,Nahmias J D,Angus C,et al.Automatic generation of consistent shadows for augmented reality[C]//Proceedings of Graphics Interface.Waterloo:Canadian Human-Computer Communications Society,2005:113-120
[8]Lu B V,Kakuta T,Kawakami R,et al.Foreground and shadow occlusion handling for outdoor augmented reality[C]//Proceedings of the IEEE International Symposium on Mixed and Augmented Reality.Los Alamitos:IEEE Computer Society Press,2010:109-119
[9]Cornelis K,Pollefeys M,Vergauwen M,et al.Augmented reality using uncalibrated video sequences[C]//Proceedings of EuEuropean Workshop on 3D Structure from Multiple Images of Large-Scale Environments.Heidelberg:Springer,2001,2018:144-160
[10]Debevec P.Rendering synthetic objects into real scenes:bridging traditional and image-based graphics with global illumination and high dynamic range photography[C]//Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques.New York:ACM Press,1998:189-198
[11]Kanbara M,Yokoya N.Real-time estimation of light source environment for photorealistic augmented reality[C]//Proceedings of the 17th International Conference on Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2004:911-914
[12]Martel-Brisson N,Zaccarin A.Moving cast shadow detection from a Gaussian mixture shadow model[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2005,(2):643-648
[13]Nadimi S,Bhanu B.Physical models for moving shadow and object detection in video[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2004,26(8):1079-1087
[14]Leone A,Distante C.Shadow detection for moving objects based on texture analysis[J].Pattern Recognition,2007,40(4):1222-1233
[15]Stander J,Mech R,Ostermann J.Detection of moving cast shadows for object segmentation[J].IEEE Transactions on Multimedia,1999,1(1):65-76
[16]Finlayson G,Fredembach C,Drew M S.Detecting illumination in images[C]//Proceedings of the 11th IEEE International Conference on Computer Vision.Los Alamitos:IEEE Computer Society Press,2007:1-8
[17]Finlayson G D,Hordley S D,Drew M S.Removing shadows from images[C]//Proceedings of the 7th European Conference on Computer Vision.Heidelberg:Springer,2002:823-836
[18]Khan S H,Bennamoun M,Sohel F,et al.Automatic feature learning for robust shadow detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2014:1939-1946
[19]Khan S H,Bennamoun M,Sohel F,et al.Automatic shadow detection and removal from a single image[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(3):431-446
[20]Macedo M C F,Nascimento V P,Souza A C S.Real-time shadow detection using multi-channel binarization and noise removal[J].Journal of Real-Time Image Processing,2018:1-14
[21]Newey C C,Jones O D,Dee H M.Shadow detection for mobile robots:features,evaluation,and datasets[J].Spatial Cognition&Computation,2018,18(2):115-137
[22]Bao L C,Yang Q X,Jin H L.Fast edge-preserving PatchMatch for large displacement optical flow[J].IEEE Transactions on Image Processing,2014,23(12):4996-5006
[23]Barrow H G,Tenenbaum J M.Recovering intrinsic scene characteristics from images[M]//Hanson A,Riseman E.Computer Vision Systems.New York:Academic Press,1978:3-26
[24]Levin A,Lischinski D,Weiss Y.A closed-form solution to natural image matting[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2008,30(2):228-242
[25]Xing G Y,Zhou X H,Peng Q S,et al.Lighting simulation of augmented outdoor scene based on a legacy photograph[J].Computer Graphics Forum,2013,32(7):101-110