基于空间金字塔池化的深度卷积神经网络多聚焦图像融合
|
摘要
针对传统方法需要人工设定特征和融合准则来完成融合任务,未能充分利用源图像中其他潜在有用信息的缺陷,提出一种基于空间金字塔池化网络的深度学习方法.首先,设计了一种孪生双通道卷积神经网络,并使用金字塔池化代替最大池化,学习多聚焦图像的特征.然后,为了有效训练该网络,采用高斯滤波器合成一个大规模具有金标准的多聚焦数据集.给定一幅多聚焦图像作为输入,训练好的模型可以输出一个指示源图像中聚焦性质的得分图.此外,为了进一步提高融合效果,将得分图进一步分割为二值掩模图,并使用形态学方法对其进行优化.最后,通过在优化的二值掩模图及源图像之间使用点乘运算,将可以得到最终融合图像.实验结果表明,算法在测试集上平均量化指标提高了0.78%.
Aiming at the defect of making use of hand-crafted features and fusion criterions to fulfill the fusion task by traditional methods, which does not consider efficiently other potentially useful information in source images, we propose a deep learning method based on the spatial pyramid pooling(SPP). First, we design a Siamese network and replace the average pooling with SPP to learn the features of multi-focus images. Then, to train the network effectively, we synthesize a large-scale multi-focus image dataset with ground truth through a Gaussian filter. Given a pair of multi-focus image as input, the trained model can generate a score map indicating the focus property of source images. Moreover, to further enhance the fusion effects, we segment the score map into a binary mask image, which is refined using morphological technique. Finally, the fused image is gained by employing dot multiplication operation between source images and the refined binary mask image. Experimental results reveal that the average quantitative score on test images achieved by the proposed method is increased by 0.78%.
Aiming at the defect of making use of hand-crafted features and fusion criterions to fulfill the fusion task by traditional methods, which does not consider efficiently other potentially useful information in source images, we propose a deep learning method based on the spatial pyramid pooling(SPP). First, we design a Siamese network and replace the average pooling with SPP to learn the features of multi-focus images. Then, to train the network effectively, we synthesize a large-scale multi-focus image dataset with ground truth through a Gaussian filter. Given a pair of multi-focus image as input, the trained model can generate a score map indicating the focus property of source images. Moreover, to further enhance the fusion effects, we segment the score map into a binary mask image, which is refined using morphological technique. Finally, the fused image is gained by employing dot multiplication operation between source images and the refined binary mask image. Experimental results reveal that the average quantitative score on test images achieved by the proposed method is increased by 0.78%.
引文
[1]Li S,Kang X,Hu J.Image fusion with guided filtering[J].IEEE Transactions on Image Processing,2013,22(7):2 864-2 875.DOI:10.1109/TIP.2013.2244222.
[2]Li S,Kwok J T,Wang Y.Combination of images with diverse focuses using the spatial frequency[J].Information Fusion,2001,2(3):169-176.DOI:10.1016/S1566-2535(01)00038-0.
[3]Wang Z,Ma Y,Gu J.Multi-focus image fusion using PCNN[J].Pattern Recognition,2010,43(6):2003-2016.DOI:10.1016/j.patcog.2010.01.011.
[4]Aslantas V,Kurban R.Fusion of multi-focus images using differential evolution algorithm[J].Expert Systems with Applications,2010,37(12):8 861-8 870.DOI:10.1016/j.eswa.2010.06.011.
[5]郭敏,任娜.基于小波变换与块分割的多聚焦图像融合[J].云南大学学报:自然科学版,2008,30(3):251-255.DOI:10.3321/j.issn:0258-7971.2008.03.006.Guo M,Ren N.Multi-focus image fusion based on wavelet transform and block segment[J].Journal of Yunnan University:Natural Sciences Edition,2008,30(3):251-255.
[6]叶智勇,施继红,裴以建.基于小波变换的图像融合算法[J].云南大学学报:自然科学版,2009,29(S1):23-26.Ye Z Y,Shi J H,Pei Y J.The algorithm of image fusion based wavelet transform[J].Journal of Yunnan University:Natural Sciences Edition,2009,29(S1):23-26.
[7]Liu Y,Liu S,Wang Z.A general framework for image fusion based on multi-scale transform and sparse representation[J].Information Fusion,2015,24:147-164.DOI:10.1016/j.inffus.2014.09.004.
[8]Kumar B K S.Multi-focus and multispectral image fusion based on pixel significance using discrete cosine harmonic wavelet transform[J].Signal Image&Video Processing,2013,7(6):1 125-1 143.
[9]Kumar B K S.Image fusion based on pixel significance using cross bilateral filter[J].Signal Image&Video Processing,2015,9(5):1 193-1 204.
[10]Cao Y,Li S,Hu J.Multi-focus image fusion by nonsubsampled shearlet transform[C]//Sixth International Conference on Image and Graphics,IEEE Computer Society,2011:17-21.
[11]Hyv?rinen A,Hurri J,Hoyer P O.Independent component analysis[J].IEEE Transactions on Neural Networks,2014,15(2):529-529.
[12]Zagoruyko S,Komodakis N.Learning to compare image patches via convolutional neural networks[C]//Computer Vision and Pattern Recognition,IEEE,2015:4 353-4 361.
[13]He K,Zhang X,Ren S,et al.Spatial pyramid pooling in deep convolutional networks for visual recognition[J].IEEE Transactions on Pattern Analysis&machine intelligence,2015,37(9):1 904-1 916.
[14]Krizhevsky A,Hinton G.Learning multiple layers of features from tiny images[R].University of Toronto,2009.
[15]Jia Y,Shelhamer E,Donahue J,et al.Caffe:Convolutional architecture for fast feature embedding[C]//Proceedings of the 22nd ACM international conference on Multimedia.ACM,2014:675-678.
[16]Montavon G,Orr G,Muller K R.Neural networks:Tricks of the trade[M].Berlin Heidelberg Springer,2012.
[17]Mackay D J C.Information theory,inference,and learning algorithms[M].Cambridge:Cambridge University Press,2003.
[18]Xydeas C S,Petrovic V.Objective image fusion performance measure[J].Military Technical Courier,2000,36(4):308-309.
[19]Sheikh H R,Bovik A C.Image information and visual quality[J].IEEE Transactions on image processing,2006,15(2):430-444.DOI:10.1109/TIP.2005.859378.
[20]Nejati M,Samavi S,Shirani S.Multi-focus image fusion using dictionary-based sparse representation[J].Information Fusion,2015,25:72-84.DOI:10.1016/j.inffus.2014.10.004.
[21]Li S,Kang X,Hu J,et al.Image matting for fusion of multi-focus images in dynamic scenes[J].Information Fusion,2013,14(2):147-162.DOI:10.1016/j.inffus.2011.07.001.
[2]Li S,Kwok J T,Wang Y.Combination of images with diverse focuses using the spatial frequency[J].Information Fusion,2001,2(3):169-176.DOI:10.1016/S1566-2535(01)00038-0.
[3]Wang Z,Ma Y,Gu J.Multi-focus image fusion using PCNN[J].Pattern Recognition,2010,43(6):2003-2016.DOI:10.1016/j.patcog.2010.01.011.
[4]Aslantas V,Kurban R.Fusion of multi-focus images using differential evolution algorithm[J].Expert Systems with Applications,2010,37(12):8 861-8 870.DOI:10.1016/j.eswa.2010.06.011.
[5]郭敏,任娜.基于小波变换与块分割的多聚焦图像融合[J].云南大学学报:自然科学版,2008,30(3):251-255.DOI:10.3321/j.issn:0258-7971.2008.03.006.Guo M,Ren N.Multi-focus image fusion based on wavelet transform and block segment[J].Journal of Yunnan University:Natural Sciences Edition,2008,30(3):251-255.
[6]叶智勇,施继红,裴以建.基于小波变换的图像融合算法[J].云南大学学报:自然科学版,2009,29(S1):23-26.Ye Z Y,Shi J H,Pei Y J.The algorithm of image fusion based wavelet transform[J].Journal of Yunnan University:Natural Sciences Edition,2009,29(S1):23-26.
[7]Liu Y,Liu S,Wang Z.A general framework for image fusion based on multi-scale transform and sparse representation[J].Information Fusion,2015,24:147-164.DOI:10.1016/j.inffus.2014.09.004.
[8]Kumar B K S.Multi-focus and multispectral image fusion based on pixel significance using discrete cosine harmonic wavelet transform[J].Signal Image&Video Processing,2013,7(6):1 125-1 143.
[9]Kumar B K S.Image fusion based on pixel significance using cross bilateral filter[J].Signal Image&Video Processing,2015,9(5):1 193-1 204.
[10]Cao Y,Li S,Hu J.Multi-focus image fusion by nonsubsampled shearlet transform[C]//Sixth International Conference on Image and Graphics,IEEE Computer Society,2011:17-21.
[11]Hyv?rinen A,Hurri J,Hoyer P O.Independent component analysis[J].IEEE Transactions on Neural Networks,2014,15(2):529-529.
[12]Zagoruyko S,Komodakis N.Learning to compare image patches via convolutional neural networks[C]//Computer Vision and Pattern Recognition,IEEE,2015:4 353-4 361.
[13]He K,Zhang X,Ren S,et al.Spatial pyramid pooling in deep convolutional networks for visual recognition[J].IEEE Transactions on Pattern Analysis&machine intelligence,2015,37(9):1 904-1 916.
[14]Krizhevsky A,Hinton G.Learning multiple layers of features from tiny images[R].University of Toronto,2009.
[15]Jia Y,Shelhamer E,Donahue J,et al.Caffe:Convolutional architecture for fast feature embedding[C]//Proceedings of the 22nd ACM international conference on Multimedia.ACM,2014:675-678.
[16]Montavon G,Orr G,Muller K R.Neural networks:Tricks of the trade[M].Berlin Heidelberg Springer,2012.
[17]Mackay D J C.Information theory,inference,and learning algorithms[M].Cambridge:Cambridge University Press,2003.
[18]Xydeas C S,Petrovic V.Objective image fusion performance measure[J].Military Technical Courier,2000,36(4):308-309.
[19]Sheikh H R,Bovik A C.Image information and visual quality[J].IEEE Transactions on image processing,2006,15(2):430-444.DOI:10.1109/TIP.2005.859378.
[20]Nejati M,Samavi S,Shirani S.Multi-focus image fusion using dictionary-based sparse representation[J].Information Fusion,2015,25:72-84.DOI:10.1016/j.inffus.2014.10.004.
[21]Li S,Kang X,Hu J,et al.Image matting for fusion of multi-focus images in dynamic scenes[J].Information Fusion,2013,14(2):147-162.DOI:10.1016/j.inffus.2011.07.001.
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |