基于改进Faster R-CNN识别深度视频图像哺乳母猪姿态
|
摘要
猪舍场景下,昼夜交替光线变化、热灯光照影响,及仔猪与母猪的粘连等因素,给全天候哺乳母猪姿态自动识别带来很大困难。该文以深度视频图像为数据源,提出基于改进Faster R-CNN的哺乳母猪姿态识别算法。将残差结构引入ZF网络,设计ZF-D2R网络,以提高识别精度并保持实时性;将Center Loss监督信号引入Faster R-CNN训练中,以增强类内特征的内聚性,提升识别精度。对28栏猪的视频图像抽取站立、坐立、俯卧、腹卧和侧卧5类姿态共计7 541张图像作为训练集,另取5类姿态的5 000张图像作为测试集。该文提出的改进模型在测试集上对哺乳母猪的站立、坐立、俯卧、腹卧和侧卧5类姿态的识别平均准确率分别达到96.73%、94.62%、86.28%、89.57%和99.04%,5类姿态的平均准确率均值达到93.25%。在识别精度上,比ZF网络和层数更深的VGG16网络的平均准确率均值分别提高了3.86和1.24个百分点。识别速度为0.058 s/帧,比VGG16网络速度提高了0.034 s。该文方法在提高识别精度的同时保证了实时性,可为全天候母猪行为识别提供技术参考。
The maternal behaviors reflect the health and welfare of the sows, which directly affect the economic benefit of the pig farm. Computer vision provides an effective, low-cost and non-contact method for monitoring the behavior of animal for precision farming. Under the scene of piggery, it is a challenge for 24-hour automatic recognition of lactating sow postures due to the daily illumination variations, influence of heat lamp, and adhesion between piglets and sows. This paper proposed an automatic recognition algorithm of lactating sow postures based on improved Faster R-CNN(convolutional neural network) using depth video images. To improve the recognition accuracy and satisfy the real-time need, we designed a ZF-D2 R(ZF with deeper layers and 2 residual learning frameworks) network by introducing residual learning frameworks into ZF network. First, 3 convolutional layers were added in the ZF network to design ZF-D(ZF with deeper layers). Then, in ZF-D network, shortcut connections were used to form 2 residual learning frameworks. The whole network made up the ZF-D2 R network. Moreover, the Center Loss was introduced to Fast R-CNN detector to construct a joint classification loss function. With the joint supervision signals of F-Softmax Loss and Center Loss in Fast R-CNN detector, a robust model was trained to obtain the deep feature representations with the 2 key learning objectives, which led to intra-class compactness and inter-class dispersion as much as possible. So, the joint supervision of F-Softmax Loss and Center Loss could reduce recognition errors caused by the similar features between different postures. By taking ZF-D2 R as basic net and adding the Center Loss to Fast R-CNN detector, the improved Faster R-CNN was built. Experiments to obtain the actual data set of lactating sow posture from the depth video of sows in the 28 pens were performed. The data set included 2 451 standing images, 2 461 sitting images, 2 488 sternal recumbency images, 2 519 ventral recumbency images and 2658 lateral recumbency images. And 5 000 images were randomly chosen as the testing set. The rest of the images were used as training set. To enhance the diversity of training data, dataset augmentation including rotating and mirroring was employed. Based on the Caffe deep learning framework, our improved Faster R-CNN was trained with end-to-end approximate joint methods. By adding 2 residual learning frameworks to ZF-D, the ZF-D2 R model improved the MAP(mean of average precision) by 1.28 percentage points. After introducing the Center Loss supervision signal, the MAP of the optimal model reached 93.25%, obtaining an increase of 1.3 percentage points, and the MAP of the method proposed achieved 93.25%. And APs(average precisions) of the 5 classes of postures i.e. standing, sitting, sternal recumbency, ventral recumbency and lateral recumbency were 96.73%, 94.62%, 86.28%, 89.57% and 99.04%, respectively. The MAP of our approach was 3.86 and 1.24 percentage points higher than that of Faster R-CNN based on ZF basic net and Faster RCNN based on the deeper VGG16 basic net, respectively. Our method processed images at a speed of 0.058 s per frame, 0.034 s faster than Faster R-CNN based on VGG16. Our proposed method could improve the recognition accuracy and simultaneously ensure the real-time performance. Compared with DPM(deformable part model) detector plus CNN posture classifier, the MAP of the end-to-end recognition method proposed in this paper was increased by 37.87 percentage points, and the speed was raised by 0.855 s per frame. Our method can be used for the 24-hour recognition of sow behaviors and lays the foundation for the analysis of sow dynamic behavior by video.
The maternal behaviors reflect the health and welfare of the sows, which directly affect the economic benefit of the pig farm. Computer vision provides an effective, low-cost and non-contact method for monitoring the behavior of animal for precision farming. Under the scene of piggery, it is a challenge for 24-hour automatic recognition of lactating sow postures due to the daily illumination variations, influence of heat lamp, and adhesion between piglets and sows. This paper proposed an automatic recognition algorithm of lactating sow postures based on improved Faster R-CNN(convolutional neural network) using depth video images. To improve the recognition accuracy and satisfy the real-time need, we designed a ZF-D2 R(ZF with deeper layers and 2 residual learning frameworks) network by introducing residual learning frameworks into ZF network. First, 3 convolutional layers were added in the ZF network to design ZF-D(ZF with deeper layers). Then, in ZF-D network, shortcut connections were used to form 2 residual learning frameworks. The whole network made up the ZF-D2 R network. Moreover, the Center Loss was introduced to Fast R-CNN detector to construct a joint classification loss function. With the joint supervision signals of F-Softmax Loss and Center Loss in Fast R-CNN detector, a robust model was trained to obtain the deep feature representations with the 2 key learning objectives, which led to intra-class compactness and inter-class dispersion as much as possible. So, the joint supervision of F-Softmax Loss and Center Loss could reduce recognition errors caused by the similar features between different postures. By taking ZF-D2 R as basic net and adding the Center Loss to Fast R-CNN detector, the improved Faster R-CNN was built. Experiments to obtain the actual data set of lactating sow posture from the depth video of sows in the 28 pens were performed. The data set included 2 451 standing images, 2 461 sitting images, 2 488 sternal recumbency images, 2 519 ventral recumbency images and 2658 lateral recumbency images. And 5 000 images were randomly chosen as the testing set. The rest of the images were used as training set. To enhance the diversity of training data, dataset augmentation including rotating and mirroring was employed. Based on the Caffe deep learning framework, our improved Faster R-CNN was trained with end-to-end approximate joint methods. By adding 2 residual learning frameworks to ZF-D, the ZF-D2 R model improved the MAP(mean of average precision) by 1.28 percentage points. After introducing the Center Loss supervision signal, the MAP of the optimal model reached 93.25%, obtaining an increase of 1.3 percentage points, and the MAP of the method proposed achieved 93.25%. And APs(average precisions) of the 5 classes of postures i.e. standing, sitting, sternal recumbency, ventral recumbency and lateral recumbency were 96.73%, 94.62%, 86.28%, 89.57% and 99.04%, respectively. The MAP of our approach was 3.86 and 1.24 percentage points higher than that of Faster R-CNN based on ZF basic net and Faster RCNN based on the deeper VGG16 basic net, respectively. Our method processed images at a speed of 0.058 s per frame, 0.034 s faster than Faster R-CNN based on VGG16. Our proposed method could improve the recognition accuracy and simultaneously ensure the real-time performance. Compared with DPM(deformable part model) detector plus CNN posture classifier, the MAP of the end-to-end recognition method proposed in this paper was increased by 37.87 percentage points, and the speed was raised by 0.855 s per frame. Our method can be used for the 24-hour recognition of sow behaviors and lays the foundation for the analysis of sow dynamic behavior by video.
引文
[1]闫丽,沈明霞,谢秋菊,等.哺乳母猪高危动作识别方法研究[J].农业机械学报,2016,47(1):266-272.Yan Li,Shen Mingxia,Xie Qiuju,et al.Research on recognition method of lactating sows’dangerous body movement[J].Transactions of the Chinese Society for Agricultural Machinery,2016,47(1):266-272.(in Chinese with English abstract)
[2]Oczak M,Maschat K,Berckmans D,et al.Classification of nest-building behaviour in non-crated farrowing sows on the basis of accelerometer data[J].Biosystems Engineering,2015,140:48-58.
[3]Nasirahmadi A,Edwards S A,Sturm B.Implementation of machine vision for detecting behaviour of cattle and pigs[J].Livestock Science,2017,202:25-38.
[4]Lao F,Brown-Brandl T,Stinn J P,et al.Automatic recognition of lactating sow behaviors through depth image processing[J].Computers&Electronics in Agriculture,2016,125:56-62.
[5]Kashiha M,Bahr C,Haredasht S A,et al.The automatic monitoring of pigs water use by cameras[J].Computers&Electronics in Agriculture,2013(90):164-169.
[6]Nasirahmadi A,Edwards S,Richter U,et al.Automatic detection of changes in group pig lying behaviour using image analysis[C]//Asabe International Meeting,2015,47(1):51-52.
[7]Nasirahmadi A,Richter U,Hensel O,et al.Using machine vision for investigation of changes in pig group lying patterns[J].Computers&Electronics in Agriculture,2015,119:184-190.
[8]Nasirahmadi A,Hensel O,Edwards S A,et al.Automatic detection of mounting behaviours among pigs using image analysis[J].Computers&Electronics in Agriculture,2016,124:295-302.
[9]刘龙申,沈明霞,柏广宇,等.基于机器视觉的母猪分娩检测方法研究[J].农业机械学报,2014,45(3):237-242.Liu Longshen,Shen Mingxia,Bo Guangyu,et al.Sows parturition detection method based on machine vision[J].Transactions of the Chinese Society for Agricultural Machinery,2014,45(3):237-242.(in Chinese with English abstract)
[10]Kashiha M A,Bahr C,Ott S,et al.Automatic monitoring of pig locomotion using image analysis[J].Livestock Science,2014,159(1):141-148.
[11]刘波,朱伟兴,杨建军,等.基于深度图像和生猪骨架端点分析的生猪步频特征提取[J].农业工程学报,2014,30(10):131-137.Liu Bo,Zhu Weixing,Yang Jianjun,et al.Extracting of pig gait frequency feature based on depth image and pig skeleton endpoints analysis[J].Transactions of The Chinese Society of Agricultural Engineering(Transactions of the CSAE),2014,30(10):131-137.(in Chinese with English abstract)
[12]Lecun Y,Bengio Y,Hinton G.Deep learning[J].Nature,2015,521(7553):436-444.
[13]Redmon J,Divvala S,Girshick R,et al.You only look once:Unified,real-time object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:779-788.
[14]Girshick R,Donahue J,Darrell T,et al.Rich feature hierarchies for accurate object detection and semantic segmentation[J].Computer Science,2013:580-587.
[15]Ren S,He K,Girshick R,et al.Faster R-CNN:Towards real-time object detection with region proposal networks[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2015,39(6):1137?1149.
[16]Girshick Ross.Fast R-CNN[C]//IEEE International Conference on Computer Vision.IEEE Computer Society,2015:1440-1448.
[17]Zeiler M D,Fergus R.Visualizing and understanding convolutional networks[J].Computer Science,2014,8689:818-833.
[18]Simonyan K,Zisserman A.Very deep convolutional networks for large-scale image recognition[J].Computer Science,2014:1-14.
[19]Szegedy C,Liu W,Jia Y,et al.Going deeper with convolutions[C]//IEEE Conference on Computer Vision and Pattern Recognition.IEEE Computer Society,2015:1-9.
[20]He Kaiming,Zhang Xiangyu,Ren Shaoqing,et al.Deep residual learning for image recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition.IEEE Computer Society,2016:770-778.
[21]Huang Gao,Liu Zhuang,Weinberger K Q,et al.Densely connected convolutional networks[C]//IEEE Conference on Computer Vision and Pattern Recognition.IEEE Computer Society,2017.
[22]Wen Yandong,Zhang Kaipeng,Li Zhifeng,et al.Adiscriminative feature learning approach for deep face recognition[C]//European Conference on Computer Vision.Springer International Publishing,2016:499-512.
[23]Xiao Jimin,Xie Yanchun,Tillo T,et al.IAN:The individual aggregation network for person search[C]//IEEE Conference on Computer Vision and Pattern Recognition.2017.
[24]Lou Zhensheng,Hurnik J Frank.Peripartum sows in three farrowing crates:Posture patterns and behavioural activities[J].Applied Animal Behaviour Science,1998,58(1/2):77-86.
[25]Zuiderveld K.Contrast limited ldaptive histogram equalization[J].Graphics Gems,1994:474-485.
[26]Jia Yangqing,Shelhamer E,Donahue J,et al.Caffe:Convolutional architecture for fast feature embedding[J].Computer Science,2014:675-678.
[27]Glorot X,Bengio Y.Understanding the difficulty of training deep feedforward neural networks[J].Journal of Machine Learning Research,2010,9:249-256.
[28]Felzenszwalb P F,Girshick R B,Mcallester D,et al.Object detection with discriminatively trained part-based models[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2010,32(9):1627-1645.
[29]Krizhevsky A,Sutskever I,Hinton Geoffrey E.Image Net classification with deep convolutional neural networks[J].Communications of the ACM,2012,60(2):1097-1105.
[30]He Kaiming,Zhang Xiangyu,Ren Shaoqing,et al.Spatial pyramid pooling in deep convolutional networks for visual recognition[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2015,37(9):1904-1916.
[2]Oczak M,Maschat K,Berckmans D,et al.Classification of nest-building behaviour in non-crated farrowing sows on the basis of accelerometer data[J].Biosystems Engineering,2015,140:48-58.
[3]Nasirahmadi A,Edwards S A,Sturm B.Implementation of machine vision for detecting behaviour of cattle and pigs[J].Livestock Science,2017,202:25-38.
[4]Lao F,Brown-Brandl T,Stinn J P,et al.Automatic recognition of lactating sow behaviors through depth image processing[J].Computers&Electronics in Agriculture,2016,125:56-62.
[5]Kashiha M,Bahr C,Haredasht S A,et al.The automatic monitoring of pigs water use by cameras[J].Computers&Electronics in Agriculture,2013(90):164-169.
[6]Nasirahmadi A,Edwards S,Richter U,et al.Automatic detection of changes in group pig lying behaviour using image analysis[C]//Asabe International Meeting,2015,47(1):51-52.
[7]Nasirahmadi A,Richter U,Hensel O,et al.Using machine vision for investigation of changes in pig group lying patterns[J].Computers&Electronics in Agriculture,2015,119:184-190.
[8]Nasirahmadi A,Hensel O,Edwards S A,et al.Automatic detection of mounting behaviours among pigs using image analysis[J].Computers&Electronics in Agriculture,2016,124:295-302.
[9]刘龙申,沈明霞,柏广宇,等.基于机器视觉的母猪分娩检测方法研究[J].农业机械学报,2014,45(3):237-242.Liu Longshen,Shen Mingxia,Bo Guangyu,et al.Sows parturition detection method based on machine vision[J].Transactions of the Chinese Society for Agricultural Machinery,2014,45(3):237-242.(in Chinese with English abstract)
[10]Kashiha M A,Bahr C,Ott S,et al.Automatic monitoring of pig locomotion using image analysis[J].Livestock Science,2014,159(1):141-148.
[11]刘波,朱伟兴,杨建军,等.基于深度图像和生猪骨架端点分析的生猪步频特征提取[J].农业工程学报,2014,30(10):131-137.Liu Bo,Zhu Weixing,Yang Jianjun,et al.Extracting of pig gait frequency feature based on depth image and pig skeleton endpoints analysis[J].Transactions of The Chinese Society of Agricultural Engineering(Transactions of the CSAE),2014,30(10):131-137.(in Chinese with English abstract)
[12]Lecun Y,Bengio Y,Hinton G.Deep learning[J].Nature,2015,521(7553):436-444.
[13]Redmon J,Divvala S,Girshick R,et al.You only look once:Unified,real-time object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:779-788.
[14]Girshick R,Donahue J,Darrell T,et al.Rich feature hierarchies for accurate object detection and semantic segmentation[J].Computer Science,2013:580-587.
[15]Ren S,He K,Girshick R,et al.Faster R-CNN:Towards real-time object detection with region proposal networks[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2015,39(6):1137?1149.
[16]Girshick Ross.Fast R-CNN[C]//IEEE International Conference on Computer Vision.IEEE Computer Society,2015:1440-1448.
[17]Zeiler M D,Fergus R.Visualizing and understanding convolutional networks[J].Computer Science,2014,8689:818-833.
[18]Simonyan K,Zisserman A.Very deep convolutional networks for large-scale image recognition[J].Computer Science,2014:1-14.
[19]Szegedy C,Liu W,Jia Y,et al.Going deeper with convolutions[C]//IEEE Conference on Computer Vision and Pattern Recognition.IEEE Computer Society,2015:1-9.
[20]He Kaiming,Zhang Xiangyu,Ren Shaoqing,et al.Deep residual learning for image recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition.IEEE Computer Society,2016:770-778.
[21]Huang Gao,Liu Zhuang,Weinberger K Q,et al.Densely connected convolutional networks[C]//IEEE Conference on Computer Vision and Pattern Recognition.IEEE Computer Society,2017.
[22]Wen Yandong,Zhang Kaipeng,Li Zhifeng,et al.Adiscriminative feature learning approach for deep face recognition[C]//European Conference on Computer Vision.Springer International Publishing,2016:499-512.
[23]Xiao Jimin,Xie Yanchun,Tillo T,et al.IAN:The individual aggregation network for person search[C]//IEEE Conference on Computer Vision and Pattern Recognition.2017.
[24]Lou Zhensheng,Hurnik J Frank.Peripartum sows in three farrowing crates:Posture patterns and behavioural activities[J].Applied Animal Behaviour Science,1998,58(1/2):77-86.
[25]Zuiderveld K.Contrast limited ldaptive histogram equalization[J].Graphics Gems,1994:474-485.
[26]Jia Yangqing,Shelhamer E,Donahue J,et al.Caffe:Convolutional architecture for fast feature embedding[J].Computer Science,2014:675-678.
[27]Glorot X,Bengio Y.Understanding the difficulty of training deep feedforward neural networks[J].Journal of Machine Learning Research,2010,9:249-256.
[28]Felzenszwalb P F,Girshick R B,Mcallester D,et al.Object detection with discriminatively trained part-based models[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2010,32(9):1627-1645.
[29]Krizhevsky A,Sutskever I,Hinton Geoffrey E.Image Net classification with deep convolutional neural networks[J].Communications of the ACM,2012,60(2):1097-1105.
[30]He Kaiming,Zhang Xiangyu,Ren Shaoqing,et al.Spatial pyramid pooling in deep convolutional networks for visual recognition[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2015,37(9):1904-1916.