递归式多阶特征融合图像超分辨率算法
|
摘要
目的近年来,卷积神经网络在解决图像超分辨率的问题上取得了巨大成功,不同结构的网络模型相继被提出。通过学习,这些网络模型对输入图像的特征进行抽象、组合,进而建立了从低分辨率的输入图像到高分辨率的目标图像的有效非线性映射。在该过程中,无论是图像的低阶像素级特征,还是高阶各层抽象特征,都对像素间相关性的挖掘起了重要作用,影响着目标高分辨图像的性能。而目前典型的超分辨率网络模型,如SRCNN (super-resolution convolutional neural network)、VDSR (very deep convolutional networks for super-resolution)、Lap SRN (Lapla-cian pyramid super-resolution networks)等,都未充分利用这些多层次的特征。方法提出一种充分融合网络多阶特征的图像超分辨率算法:该模型基于递归神经网络,由相同的单元串联构成,单元间参数共享;在每个单元内部,从低阶到高阶的逐级特征被级联、融合,以获得更丰富的信息来强化网络的学习能力;在训练中,采用基于残差的策略,单元内使用局部残差学习,整体网络使用全局残差学习,以加快训练速度。结果所提出的网络模型在通用4个测试集上,针对分辨率放大2倍、3倍、4倍的情况,与深层超分辨率网络VDSR相比,平均分别能够获得0. 24 d B、0. 23 d B、0. 19 d B的增益。结论实验结果表明,所提出的递归式多阶特征融合图像超分辨率算法,有效提升了性能,特别是在细节非常丰富的Urban100数据集上,该算法对细节的处理效果尤为明显,图像的客观质量与主观质量都得到显著改善。
Objective The recovery of a high-resolution( HR) image or video from its low-resolution( LR) counterpart,which is referred to as super resolution( SR),has attracted considerable attention in computer vision community. The SRproblem is inherently ill-posed because the HR image or video actually does not exist. Several methods have been proposedto address this issue. Several typical methods,such as bilinear or bicubic interpolation,Lanczos resampling,and internalpatch recurrence,have been used. Recently,learning-based methods,such as sparse coding,random forest,and convolu-tional neural networks( CNNs),have been utilized to create a mapping between LR and HR images. Particularly,theCNN-based scheme has achieved remarkable performance improvement. Different network models,such as SRCNN,VD-SR,LapS RN,and DRRN,have been proposed. These models abstract and combine the features of LR image to establish an effective nonlinear mapping from LR input images to HR target images. In this process,low-and high-level features play an important role in determining the correlation between pixels and in improving the performance of restored HR images.However,the features of previous layer in the aforementioned typical SR network models are directly fed in the next layer,where multi-level features are incompletely utilized. Inspired by the recent DenseN et,we concatenate and fuse multi-level features from multilayers. Although multi-level features are utilized in this manner,the number of parameters is large,which costs long training time and large storage. Therefore,we employ a recursive network architecture for parameter sharing. The overall model develops an efficient CNN model that can utilize the multi-level features of CNN to improve the SR performance and can control the number of model parameters within an acceptable range. Method We propose an image SR model that utilizes multi-level features. The proposed multi-feature fusion recursive network( MFRN) is based on recursive neural network with the same units in series. The information of features is passed along the basic unit of MFRN,named as the multi-feature fusion unit( MFU). The parameters are shared among these basic units,and the required number of parameters is effectively reduced. The input status within each MFU is obtained from the previous unit with continuous memory mechanism. Then,the low-level to high-level features are concatenated and fused to obtain abundant features in describing the image. Valuable features are extracted and enhanced,which can accurately describe the mapping relationship between LR and HR. With regard to the training process,a residual learning strategy,which involves local residual learning inside each unit and global residual learning through the entire network,is adopted to accelerate the training speed. Specifically,a global residual learning strategy is employed in the training of the overall MFRN,and a local residual learning strategy is applied for MFU. The training difficulty is efficiently reduced,and typical phenomena,such as network degradation and vanishing gradient,can be avoided by combining the aforementioned strategies. In terms of the cost function,the averaged mean square error over the training set is minimized. We train a single model for multiple scales based on the proposed cost function and training methods. Result We use 291 pictures from public databases as the training set. In addition,data augmentation( rotation or flip) is applied. Images with different scales( × 2,× 3,and × 4) are included in the training set. Therefore,only a single model is trained for all different scales. During the training process,we adopt an adaptive learning rate and an adjustable gradient clipping to accelerate the convergence rate while suppressing exploding gradients. We evaluate four network models with different numbers of MFUs,which correspond to 29,37,53,and 81 layers. The network with nine MFUs achieves the best performance by comparing the convergence rate and performance.Hence,we adopt nine MFUs in the final CNN model. Although the proposed network has 37 layers,it elegantly converges at 230 epochs and obtains remarkable gains. The dominant evaluation criteria of image quality,such as PSNR,SSIM,and IFC,are employed for the performance assessment of restored images. Experimental results show that the proposed model achieves average PSNR gains of 0. 24,0. 23,and 0. 19 dB compared with the very deep convolutional networks for superresoluton( VDSR) with the general four test sets for × 2, × 3,and × 4 resolutions. Specifically,the proposed MFRN considerably improves the quality of restored images in the dataset Urban100 that contains rich details. In addition,the subjective quality of restored images is illustrated. The MFRN can produce relatively sharper edges than that of other methods. Conclusion A multilevel feature fusion image SR algorithm based on recursive neural network,referred to as MFRN,is proposed in this study. The MFRN consists of multiple MFUs. Several recursive units are stacked to learn the residual image between the HR and LR images. The parameters with the recursive learning scheme are shared among the units,thereby effectively reducing the number of network parameters. The features of different levels within each unit are concatenated and fused to provide intensive description of the images. In this way,the proposed MFRN can extract and adaptively enhance valuable features,which leads to accurate mapping between LR and HR images. During the training procedure,we adopt a local residual learning inside each unit and a global residual learning through the entire network. Thus,a single model is trained for different scales. Experimental results show that the proposed MFRN considerably improves the performance. Specifically,in the Urban100 dataset,MFRN achieves 0. 4 dB PSNR gains compared with the classical VDSR model.In comparison with the basic recursive network DRRN,0. 14 dB PNSR improvement is obtained. With regard to the subjective quality,MFRN is focused on handling the details of images. The visual perception of images is remarkably improved.
Objective The recovery of a high-resolution( HR) image or video from its low-resolution( LR) counterpart,which is referred to as super resolution( SR),has attracted considerable attention in computer vision community. The SRproblem is inherently ill-posed because the HR image or video actually does not exist. Several methods have been proposedto address this issue. Several typical methods,such as bilinear or bicubic interpolation,Lanczos resampling,and internalpatch recurrence,have been used. Recently,learning-based methods,such as sparse coding,random forest,and convolu-tional neural networks( CNNs),have been utilized to create a mapping between LR and HR images. Particularly,theCNN-based scheme has achieved remarkable performance improvement. Different network models,such as SRCNN,VD-SR,LapS RN,and DRRN,have been proposed. These models abstract and combine the features of LR image to establish an effective nonlinear mapping from LR input images to HR target images. In this process,low-and high-level features play an important role in determining the correlation between pixels and in improving the performance of restored HR images.However,the features of previous layer in the aforementioned typical SR network models are directly fed in the next layer,where multi-level features are incompletely utilized. Inspired by the recent DenseN et,we concatenate and fuse multi-level features from multilayers. Although multi-level features are utilized in this manner,the number of parameters is large,which costs long training time and large storage. Therefore,we employ a recursive network architecture for parameter sharing. The overall model develops an efficient CNN model that can utilize the multi-level features of CNN to improve the SR performance and can control the number of model parameters within an acceptable range. Method We propose an image SR model that utilizes multi-level features. The proposed multi-feature fusion recursive network( MFRN) is based on recursive neural network with the same units in series. The information of features is passed along the basic unit of MFRN,named as the multi-feature fusion unit( MFU). The parameters are shared among these basic units,and the required number of parameters is effectively reduced. The input status within each MFU is obtained from the previous unit with continuous memory mechanism. Then,the low-level to high-level features are concatenated and fused to obtain abundant features in describing the image. Valuable features are extracted and enhanced,which can accurately describe the mapping relationship between LR and HR. With regard to the training process,a residual learning strategy,which involves local residual learning inside each unit and global residual learning through the entire network,is adopted to accelerate the training speed. Specifically,a global residual learning strategy is employed in the training of the overall MFRN,and a local residual learning strategy is applied for MFU. The training difficulty is efficiently reduced,and typical phenomena,such as network degradation and vanishing gradient,can be avoided by combining the aforementioned strategies. In terms of the cost function,the averaged mean square error over the training set is minimized. We train a single model for multiple scales based on the proposed cost function and training methods. Result We use 291 pictures from public databases as the training set. In addition,data augmentation( rotation or flip) is applied. Images with different scales( × 2,× 3,and × 4) are included in the training set. Therefore,only a single model is trained for all different scales. During the training process,we adopt an adaptive learning rate and an adjustable gradient clipping to accelerate the convergence rate while suppressing exploding gradients. We evaluate four network models with different numbers of MFUs,which correspond to 29,37,53,and 81 layers. The network with nine MFUs achieves the best performance by comparing the convergence rate and performance.Hence,we adopt nine MFUs in the final CNN model. Although the proposed network has 37 layers,it elegantly converges at 230 epochs and obtains remarkable gains. The dominant evaluation criteria of image quality,such as PSNR,SSIM,and IFC,are employed for the performance assessment of restored images. Experimental results show that the proposed model achieves average PSNR gains of 0. 24,0. 23,and 0. 19 dB compared with the very deep convolutional networks for superresoluton( VDSR) with the general four test sets for × 2, × 3,and × 4 resolutions. Specifically,the proposed MFRN considerably improves the quality of restored images in the dataset Urban100 that contains rich details. In addition,the subjective quality of restored images is illustrated. The MFRN can produce relatively sharper edges than that of other methods. Conclusion A multilevel feature fusion image SR algorithm based on recursive neural network,referred to as MFRN,is proposed in this study. The MFRN consists of multiple MFUs. Several recursive units are stacked to learn the residual image between the HR and LR images. The parameters with the recursive learning scheme are shared among the units,thereby effectively reducing the number of network parameters. The features of different levels within each unit are concatenated and fused to provide intensive description of the images. In this way,the proposed MFRN can extract and adaptively enhance valuable features,which leads to accurate mapping between LR and HR images. During the training procedure,we adopt a local residual learning inside each unit and a global residual learning through the entire network. Thus,a single model is trained for different scales. Experimental results show that the proposed MFRN considerably improves the performance. Specifically,in the Urban100 dataset,MFRN achieves 0. 4 dB PSNR gains compared with the classical VDSR model.In comparison with the basic recursive network DRRN,0. 14 dB PNSR improvement is obtained. With regard to the subjective quality,MFRN is focused on handling the details of images. The visual perception of images is remarkably improved.
引文
[1]Shi W Z,Caballero J,Ledig C,et al.Cardiac image super-resolution with global correspondence using multi-atlas patchmatch[C]//Proceedings of the 16th International Conference.Berlin Heidelberg:Springer,2013:9-16.[DOI:10.1007/978-3-642-40760-4_2]
[2]Thornton M W,Atkinson P M,Holland D A.Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping[J].International Journal of Remote Sensing,2006,27(3):473-491.[DOI:10.1080/01431160500207088]
[3]Wilman W W Z,Yuen P C.Very low resolution face recognition problem[C]//Proceedings of 2010 Fourth IEEE International Conference on Biometrics:Theory,Applications and Systems.Washington,DC,USA:IEEE,2010:1-6.[DOI:10.1109/BTAS.2010.5634490]
[4]Huang J B,Singh A,Ahuja N.Single image super-resolution from transformed self-exemplars[C]//Proceedings of 2015 IEEEConference on Computer Vision and Pattern Recognition.Boston,MA,USA:IEEE,2015:5197-5206.[DOI:10.1109/CVPR.2015.7299156]
[5]Schulter S,Leistner C,Bischof H.Fast and accurate image upscaling with super-resolution forests[C]//Proceedings of 2015IEEE Conference on Computer Vision and Pattern Recognition.Boston,MA,USA:IEEE,2015:3791-3799.[DOI:10.1109/CVPR.2015.7299003]
[6]Dong C,Loy C C,Tang X O.Accelerating the super-resolution convolutional neural network[C]//Proceedings of 2015 European Conference on Computer Vision.Cham:Springer,2016:391-407.[DOI:10.1007/978-3-319-46475-6_25]
[7]He K M,Zhang X Y,Ren S Q,et al.Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Conference on Computer Vision and Pattern Recognition.Las Vegas,NV,USA:IEEE,2015:770-778.[DOI:10.1109/CVPR.2016.90]
[8]Kim J,Lee J K,Lee K M.Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of 2016IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,NV,USA:IEEE,2016:1646-1654.[DOI:10.1109/CVPR.2016.182]
[9]Martin D,Fowlkes C,Tal D,et al.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings of the 8th IEEE International Conference on Computer Vision.Vancouver,BC,Canada:IEEE,2002:416-423.[DOI:10.1109/ICCV.2001.937655]
[10]Shi W Z,Caballero J,Huszr F,et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,NV,USA:IEEE,2016:1874-1883.[DOI:10.1109/CVPR.2016.207]
[11]Lai W S,Huang J B,Ahuja N,et al.Deep laplacian pyramid networks for fast and accurate super-resolution[C]//Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,USA:IEEE,2017.
[12]Kim J,Lee J K,Lee K M.Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of 2016 IEEEConference on Computer Vision and Pattern Recognition.Las Vegas,NV,USA:IEEE,2016:1637-1645.[DOI:10.1109/CVPR.2016.181]
[13]Bevilacqua M,Roumy A,Guillemot C,et al.Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceedings of 2012 British Machine Vision Conference.Surrey:BMVC,2012.
[14]Zeyde R,Elad M,Protter M.On single image scale-up using sparse-representations[C]//Proceedings of the 7th International Conference.Berlin,Heidelberg:Springer-Verlag,2012:711-730.[DOI:10.1007/978-3-642-27413-8_47]
[15]Tai Y,Yang J,Liu X M.Image super-resolution via deep recursive residual network[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,Hawaii,USA:IEEE,2017:2790-2798.[DOI:10.1109/CVPR.2017.298]
[16]Zhang Y L,Tian Y P,Kong,et al.Residual dense network for image super-resolution[C]//Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake:IEEE,2018:2472-2481.
[17]Huang G,Liu Z,van der Maaten L,et al.Densely connected convolutional networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,Hawaii,USA:IEEE,2017.[DOI:10.1109/CVPR.2017.243]
[18]Yang J C,Wright J,Huang T S,et al.Image super-resolution via sparse representation.[J].IEEE Transactions on Image Processing,2010,19(11):2861-2873.[DOI:10.1109/TIP.2010.2050625]
[19]He K M,Zhang X Y,Ren S Q,et al.Delving deep into rectifiers:surpassing human-level performance on image Net classification[C]//Proceedings of 2015 IEEE International Conference on Computer Vision.Santiago,Chile:IEEE,2015:1026-1034.[DOI:10.1109/ICCV.2015.123]
[20]Sheikh H R,Bovik A C,de Veciana G.An information fidelity criterion for image quality assessment using natural scene statistics[J].IEEE Transactions on Image Processing,2005,14(12):2117-2128.[DOI:10.1109/TIP.2005.859389]
[21]Yang C Y,Ma C,Yang M H.Single-image super-resolution:a benchmark[C]//Proceedings of 2014 European Conference on Computer Vision.Cham:Springer,2014:372-386.[DOI:10.1007/978-3-319-10593-2_25]
[22]Pérez-Pellitero E,Salvador J,Ruiz-Hidalgo J,et al.Psy Co:manifold span reduction for super resolution[C]//Proceedings of2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,NV,USA:IEEE,2016:1837-1845.[DOI:10.1109/CVPR.2016.203]
[23]Tong T,Li G,Liu X J,et al.Image super-resolution using dense skip connections[C]//Proceedings of 2017 IEEE International Conference on Computer Vision.Venice,Italy:IEEE,2017:4809-4817.[DOI:10.1109/ICCV.2017.514]
[2]Thornton M W,Atkinson P M,Holland D A.Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping[J].International Journal of Remote Sensing,2006,27(3):473-491.[DOI:10.1080/01431160500207088]
[3]Wilman W W Z,Yuen P C.Very low resolution face recognition problem[C]//Proceedings of 2010 Fourth IEEE International Conference on Biometrics:Theory,Applications and Systems.Washington,DC,USA:IEEE,2010:1-6.[DOI:10.1109/BTAS.2010.5634490]
[4]Huang J B,Singh A,Ahuja N.Single image super-resolution from transformed self-exemplars[C]//Proceedings of 2015 IEEEConference on Computer Vision and Pattern Recognition.Boston,MA,USA:IEEE,2015:5197-5206.[DOI:10.1109/CVPR.2015.7299156]
[5]Schulter S,Leistner C,Bischof H.Fast and accurate image upscaling with super-resolution forests[C]//Proceedings of 2015IEEE Conference on Computer Vision and Pattern Recognition.Boston,MA,USA:IEEE,2015:3791-3799.[DOI:10.1109/CVPR.2015.7299003]
[6]Dong C,Loy C C,Tang X O.Accelerating the super-resolution convolutional neural network[C]//Proceedings of 2015 European Conference on Computer Vision.Cham:Springer,2016:391-407.[DOI:10.1007/978-3-319-46475-6_25]
[7]He K M,Zhang X Y,Ren S Q,et al.Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Conference on Computer Vision and Pattern Recognition.Las Vegas,NV,USA:IEEE,2015:770-778.[DOI:10.1109/CVPR.2016.90]
[8]Kim J,Lee J K,Lee K M.Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of 2016IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,NV,USA:IEEE,2016:1646-1654.[DOI:10.1109/CVPR.2016.182]
[9]Martin D,Fowlkes C,Tal D,et al.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings of the 8th IEEE International Conference on Computer Vision.Vancouver,BC,Canada:IEEE,2002:416-423.[DOI:10.1109/ICCV.2001.937655]
[10]Shi W Z,Caballero J,Huszr F,et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,NV,USA:IEEE,2016:1874-1883.[DOI:10.1109/CVPR.2016.207]
[11]Lai W S,Huang J B,Ahuja N,et al.Deep laplacian pyramid networks for fast and accurate super-resolution[C]//Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,USA:IEEE,2017.
[12]Kim J,Lee J K,Lee K M.Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of 2016 IEEEConference on Computer Vision and Pattern Recognition.Las Vegas,NV,USA:IEEE,2016:1637-1645.[DOI:10.1109/CVPR.2016.181]
[13]Bevilacqua M,Roumy A,Guillemot C,et al.Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceedings of 2012 British Machine Vision Conference.Surrey:BMVC,2012.
[14]Zeyde R,Elad M,Protter M.On single image scale-up using sparse-representations[C]//Proceedings of the 7th International Conference.Berlin,Heidelberg:Springer-Verlag,2012:711-730.[DOI:10.1007/978-3-642-27413-8_47]
[15]Tai Y,Yang J,Liu X M.Image super-resolution via deep recursive residual network[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,Hawaii,USA:IEEE,2017:2790-2798.[DOI:10.1109/CVPR.2017.298]
[16]Zhang Y L,Tian Y P,Kong,et al.Residual dense network for image super-resolution[C]//Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake:IEEE,2018:2472-2481.
[17]Huang G,Liu Z,van der Maaten L,et al.Densely connected convolutional networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,Hawaii,USA:IEEE,2017.[DOI:10.1109/CVPR.2017.243]
[18]Yang J C,Wright J,Huang T S,et al.Image super-resolution via sparse representation.[J].IEEE Transactions on Image Processing,2010,19(11):2861-2873.[DOI:10.1109/TIP.2010.2050625]
[19]He K M,Zhang X Y,Ren S Q,et al.Delving deep into rectifiers:surpassing human-level performance on image Net classification[C]//Proceedings of 2015 IEEE International Conference on Computer Vision.Santiago,Chile:IEEE,2015:1026-1034.[DOI:10.1109/ICCV.2015.123]
[20]Sheikh H R,Bovik A C,de Veciana G.An information fidelity criterion for image quality assessment using natural scene statistics[J].IEEE Transactions on Image Processing,2005,14(12):2117-2128.[DOI:10.1109/TIP.2005.859389]
[21]Yang C Y,Ma C,Yang M H.Single-image super-resolution:a benchmark[C]//Proceedings of 2014 European Conference on Computer Vision.Cham:Springer,2014:372-386.[DOI:10.1007/978-3-319-10593-2_25]
[22]Pérez-Pellitero E,Salvador J,Ruiz-Hidalgo J,et al.Psy Co:manifold span reduction for super resolution[C]//Proceedings of2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,NV,USA:IEEE,2016:1837-1845.[DOI:10.1109/CVPR.2016.203]
[23]Tong T,Li G,Liu X J,et al.Image super-resolution using dense skip connections[C]//Proceedings of 2017 IEEE International Conference on Computer Vision.Venice,Italy:IEEE,2017:4809-4817.[DOI:10.1109/ICCV.2017.514]