视频图像语义信息提取研究
|
摘要
对视频图像进行语义信息的提取，可以满足用户基于语义的检索需求。在现有的一些语义信息提取方法中，存在如下问题：(1)如何构建合理的语义概念层次；(2)如何有效地表征视频图像所涉及的语义概念；(3)如何自动发现语义概念间的关联性并加以利用；(4)如何动态地融合语义信息；(5)如何挖掘视频在时域上的依赖信息并加以利用。针对上述问题，本文提出了三种方法，从不同层面分别进行解决。
首先，本文提出一个自底向上的层次化语义提取框架。这个框架将视频镜头的底层特征、语义概念中的物体和语义概念中的场景划分为由底向上的三个层次。这个层次结构简单，也具有较好的表征能力。视频镜头的底层特征是在对视频镜头关键帧分割后的区域上提取的。针对每一种底层特征和每一个物体概念，训练得到的支持向量机，用本文提出的boosting方法，在不同特征上进行融合，得到了针对显著物体的检测器。本文提出了两种利用这些检测器的置信度输出，对视频镜头进行语义表征的模型向量，并在这两种模型向量的基础上对场景概念进行学习。实验证明本文的语义概念层次的有效性、boosting融合对性能的提升以及所提出的模型向量的优势。
接下来，对于半自动的图像标注，本文将它形式化为一个多标记学习问题，并提出了一个基于辅助标签的半自动图像标注方法。该方法将归一化互信息作为定量地衡量语义标签之间关联度的指标，并采用一种动态混合模型改善标签的分类结果。该方法具有一个框架性的结构，很容易与标签的相关反馈信息结合，加速人机交互过程。实验结果表明该方法可以改善不同学习算法的分类结果，而且能够更有效地利用相关反馈信息，具有比其他方法更快的人机交互速度。
最后，本文挖掘视频镜头在时域和空域上的关联信息，寻找同一镜头和相邻镜头中有助于一个目标物体检测的辅助物体，并确定这些辅助物体能够提供最大辅助信息的位置，这些辅助信息在一个动态混合模型中被整合，提高了原来的视频镜头中的物体检测性能。
The extraction of semantic information from videos and images can implement theretrieval based on the semantics. There are some issues in the existing extraction methods:(1) how to construct a sound semantic hierarchy; (2) how to effectively expressthe semantics; (3) how to automatically discover and utilize the relevance between semantics;(4) how to dynamically fuse the semantics; (5) how to mine and utilize thetemporal dependency of videos. Three methods are proposed to solve these issues.
First, a bottom-up hierarchical semantics extraction framework is proposed, whichdivide the low-level features of shots, objects and scenes concepts into a bottom-upthree-level hierarchy. A support vector machine(SVM) is trained for a given low-levelfeature and a given object concept. The proposed boosting fusion, combines severalSVMs trained on each feature into a salient object detector. The confidence output ofthese detectors are used by two proposed model vectors to express the semantics ofshots. Scene concepts are then learned on these model vectors.
Secondly, a semi-automatic image annotation method based on auxiliary tags isproposed, which measures the relevance between tags by the normalized mutual information,and improve the tag classification with a dynamic mixture model. This methodcan be easily combined with the relevance feedback on tags to speed up the interactionbetween human and the computer.
Finally, to improve the object detection in video shots, spatial and temporal informationare further mined to discover and locate the helpful objects in the same shot orthe adjacent shot. These spatial and temporal auxiliary information are integrated in adynamic mixture model and improve the original detection results effectively.
首先，本文提出一个自底向上的层次化语义提取框架。这个框架将视频镜头的底层特征、语义概念中的物体和语义概念中的场景划分为由底向上的三个层次。这个层次结构简单，也具有较好的表征能力。视频镜头的底层特征是在对视频镜头关键帧分割后的区域上提取的。针对每一种底层特征和每一个物体概念，训练得到的支持向量机，用本文提出的boosting方法，在不同特征上进行融合，得到了针对显著物体的检测器。本文提出了两种利用这些检测器的置信度输出，对视频镜头进行语义表征的模型向量，并在这两种模型向量的基础上对场景概念进行学习。实验证明本文的语义概念层次的有效性、boosting融合对性能的提升以及所提出的模型向量的优势。
接下来，对于半自动的图像标注，本文将它形式化为一个多标记学习问题，并提出了一个基于辅助标签的半自动图像标注方法。该方法将归一化互信息作为定量地衡量语义标签之间关联度的指标，并采用一种动态混合模型改善标签的分类结果。该方法具有一个框架性的结构，很容易与标签的相关反馈信息结合，加速人机交互过程。实验结果表明该方法可以改善不同学习算法的分类结果，而且能够更有效地利用相关反馈信息，具有比其他方法更快的人机交互速度。
最后，本文挖掘视频镜头在时域和空域上的关联信息，寻找同一镜头和相邻镜头中有助于一个目标物体检测的辅助物体，并确定这些辅助物体能够提供最大辅助信息的位置，这些辅助信息在一个动态混合模型中被整合，提高了原来的视频镜头中的物体检测性能。
The extraction of semantic information from videos and images can implement theretrieval based on the semantics. There are some issues in the existing extraction methods:(1) how to construct a sound semantic hierarchy; (2) how to effectively expressthe semantics; (3) how to automatically discover and utilize the relevance between semantics;(4) how to dynamically fuse the semantics; (5) how to mine and utilize thetemporal dependency of videos. Three methods are proposed to solve these issues.
First, a bottom-up hierarchical semantics extraction framework is proposed, whichdivide the low-level features of shots, objects and scenes concepts into a bottom-upthree-level hierarchy. A support vector machine(SVM) is trained for a given low-levelfeature and a given object concept. The proposed boosting fusion, combines severalSVMs trained on each feature into a salient object detector. The confidence output ofthese detectors are used by two proposed model vectors to express the semantics ofshots. Scene concepts are then learned on these model vectors.
Secondly, a semi-automatic image annotation method based on auxiliary tags isproposed, which measures the relevance between tags by the normalized mutual information,and improve the tag classification with a dynamic mixture model. This methodcan be easily combined with the relevance feedback on tags to speed up the interactionbetween human and the computer.
Finally, to improve the object detection in video shots, spatial and temporal informationare further mined to discover and locate the helpful objects in the same shot orthe adjacent shot. These spatial and temporal auxiliary information are integrated in adynamic mixture model and improve the original detection results effectively.
引文
[1] Kato T, Kurita T, Otsu N, et al. A sketch retrieval method for full color image database-query by visual example. Pattern Recognition, 1992. Vol.I. Conference A: Computer Vision and Applications, Proceedings., 11th IAPR International Conference on, 1992. 530-533.
[2] Flickner M, Sawhney H, Niblack W, et al. Query by Image and Video Content: The QBIC System. Computer, 1995,28(9):23-32.
[3] Pentland A, Picard R, Sclaroff S. Photobook: Tools for content-based manipulation of image databases. Proceedings of Proceedings of the Conference on Storage and Retrieval for Image and Video Database Ⅱ, SPIE.
[4] Smith J R, Chang S F. VisualSEEk: a fully automated content-based image query system. Proceedings of MULTIMEDIA '96: Proceedings of the fourth ACM international conference on Multimedia, New York, NY, USA: ACM, 1996. 87-98.
[5] Ma W, Manjunath B. NeTra: a toolbox for navigating large image databases. Image Processing, 1997. Proceedings., International Conference on, 1997, 1:568-571.
[6] Smeulders A W M, Worring M, Santini S, et al. Content-Based Image Retrieval at the End of the Early Years. IEEE Trans. Pattern Anal. Mach. Intell., 2000,22(12): 1349-1380.
[7] Berchtold S, Keim D A, Kriegel H P. The X-tree: An Index Structure for High-Dimensional Data. Proceedings of VLDB'96. Morgan Kaufmann, 1996. 28-39.
[8] Henrich A. The LSDh-tree: an access structure for feature vectors. Data Engineering, 1998. Proceedings., 14th International Conference on, 1998. 362-369.
[9] Weber R, Schek H J, Blott S. A Quantitative Analysis and Performance Study for Similarity-Search Methods in High-Dimensional Spaces. Proceedings of VLDB '98: Proceedings of the 24rd International Conference on Very Large Data Bases, San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1998. 194-205.
[10] Bohm C. A cost model for query processing in high dimensional data spaces. ACM Trans. Database Syst., 2000, 25(2): 129-178.
[11] TRECVTD. http://www-nlpir.nist.gov/projects/trecvid/.
[12] Shi J, Malik J. Normalized cuts and image segmentation. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2000, 22(8):888-905.
[13] Deng Y, Manjunath B. Unsupervised segmentation of color-texture regions in images and video. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2001, 23(8):800- 810.
[14] Comaniciu D, Meer P. Mean shift: a robust approach toward feature space analysis. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2002, 24(5):603-619.
[15] Smith J R, Chang S F. Single color extraction and image query. Proceedings of ICIP '95: Proceedings of the 1995 International Conference on Image Processing (Vol. 3)-Volume 3, Washington, DC, USA: IEEE Computer Society, 1995.
[16] Haralick R, Shanmugan K, Dinstein I. Texture features for image classification. IEEE Transactions on System Management and Cybernetics, 1973, 3(6):610-621.
[17] Tamura H, Mori S, Yamawaki T. Texture features corresponding to visual perception. IEEE Transactions on System Management and Cybernetics, 1978, 8(6):460-473.
[18] Mallat S. A theory for multiresolution signal decomposition: the wavelet representation. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 1989, 11(7):674-693.
[19] Rui Y, She A, Huang T. Modified fourier descriptors for shape representation - a practical approach. Proceedings of Proc of First International Workshop on Image Databases and Multi Media Search., 1996. 456-461.
[20] Hu M K. Visual pattern recognition by moment invariants. Information Theory, IRE Transactions on, 1962, 8(2):179-187.
[21] Jolliffe I. Principal component analysis. New York, NY: Springer-Verlag, 1986.
[22] McLachlan G J. Discriminant Analysis and Statistical Pattern Recognition. Wiley-Interscience, 2004.
[23] Quinlan J R. C4.5: programs for machine learning. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1993.
[24] Hopfield J J. Neural Networks and Physical Systems with Emergent Collective Computational Abilities. PNAS, 1982, 79(8):2554-2558.
[25] Vapnik V N. The nature of statistical learning theory. New York, NY, USA: Springer-Verlag New York, Inc., 1995.
[26] Gray R, Olshen R. Vector quantization and density estimation. 1997. 172-193.
[27] Vailaya A, Figueiredo M, Jain A, et al. Image classification for content-based indexing.Image Processing, IEEE Transactions on, 2001, 10(1): 117-130.
[28] Huang C L, Shih H C, Chao C Y. Semantic analysis of soccer video using dynamic Bayesian network. Multimedia, IEEE Transactions on, 2006, 8(4):749-760.
[29] Vapnik V N. Estimation of Dependences Based on Empirical Data: Springer Series in Statistics (Springer Series in Statistics). Secaucus, NJ, USA: Springer-Verlag New York, Inc., 1982.
[30] Natsev A P, Naphade M R, Smith J R. Semantic representation: search and mining of multimedia content. Proceedings of KDD '04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, New York, NY, USA: ACM, 2004. 641-646.
[31] Sch(?)lkopf B, Platt J C, Shawe-Taylor J C, et al. Estimating the Support of a High-Dimensional Distribution. Neural Comput., 2001, 13(7): 1443-1471.
[32] Veropoulos K, Campbell C, Cristianini N. Controlling the sensitivity of support vector machines. Proceedings of Proceedings of the International Joint Conference on AI, 1999. 55-60.
[33] Wu G, Chang E Y. KBA: Kernel Boundary Alignment Considering Imbalanced Data Distribution. IEEE Transactions on Knowledge and Data Engineering (TKDE), 2005, 17(6):786-795.
[34] Yuan J, Li J, Zhang B. Learning concepts from large scale imbalanced data sets using support cluster machines. Proceedings of MULTIMEDIA '06: Proceedings of the 14th annual ACM international conference on Multimedia, New York, NY, USA: ACM, 2006. 441-450.
[35] Li B, Chi M, Fan J, et al. Support cluster machine. Proceedings of ICML '07: Proceedings of the 24th international conference on Machine learning, New York, NY, USA: ACM, 2007. 505-512.
[36] Carneiro G, Chan A B, Moreno P J. Supervised Learning of Semantic Classes for Image Annotation and Retrieval. IEEE Trans. Pattern Anal. Mach. Intell., 2007, 29(3):394-410. Member-Vasconcelos,, Nuno.
[37] Fan J, Luo H, Hacid M S. Mining images on semantics via statistical learning. Proceedings of KDD '05: Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, New York, NY, USA: ACM, 2005. 22-31.
[38] Assfalg J, Bertini M, Colombo C, et al. Automatic extraction and annotation of soccer video highlights. Image Processing, 2003. ICIP 2003. Proceedings. 2003 International Conference on, 2003, 2:Ⅱ-527-30 vol.3.
[39] Zhang J, Yang J, Hauptmann A G. A Discriminative Learning Framework with Pairwise Constraints for Video Object Classification. IEEE Trans. Pattern Anal. Mach. Intell., 2006, 28(4):578.
[40] Yang J, Hauptmann A G. 3WNews: who, where, and when in news video. Proceedings of MULTIMEDIA '06: Proceedings of the 14th annual ACM international conference on Multimedia, New York, NY, USA: ACM, 2006. 503-504.
[41] Hauptmann A G. Towards a Large Scale Concept Ontology for Broadcast Video. CIVR, 2004..
[42] Naphade M, Smith J R, Tesic J, et al. Large-scale concept ontology for multimedia. IEEE Multimedia, 2006, 13(3):86-91.
[43] Naphade M, Natsev A, Lin C Y, et al. Multi-granular detection of regional semantic concepts [video annotation]. Multimedia and Expo, 2004. ICME '04. 2004 IEEE International Conference on, 2004, 1:109-112.
[44] Naphade M, Smith J. Learning visual models of semantic concepts. Image Processing, 2003. ICIP 2003. Proceedings. 2003 International Conference on, 2003, 2:Ⅱ-531-4.
[45] Zhang R, Zhang Z. Image Database Classification based on Concept Vector Model. Multi media and Expo, 2005. ICME 2005. IEEE International Conference on, 2005. 93-96.
[46] Yao J, Zhang Z. Semi-supervised learning based object detection in aerial imagery. volume 1, 2005. 1011-1016.
[47] Yao J, Zhang Z. Object detection in aerial imagery based on enhanced semi-supervised learning. volume 2, 2005. 1012-1017.
[48] Naphade M, Smith J. A generalized multiple instance learning algorithm for large scale modeling of multimedia semantics. Acoustics, Speech, and Signal Processing, 2005. Proceedings. (ICASSP '05). IEEE International Conference on, 2005, 5:341-344.
[49] Chen Y, Bi J, Wang J. MILES: Multiple-Instance Learning via Embedded Instance Selection. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2006,28(12):1931-1947.
[50] Zhang D Q, Chang S F. A Generative-Discriminative Hybrid Method for Multi-View Ob ject Detection. Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, 2006, 2:2017-2024.
[51] Yan R, Chen M, Hauptmann A. Mining Relationship Between Video Concepts using Probabilistic Graphical Models. Multimedia and Expo, 2006 IEEE International Conference on, 2006. 301-304.
[52] Xie L, Chang S F. Pattern Mining in Visual Concept Streams. Multimedia and Expo, 2006 IEEE International Conference on, 2006. 297-300.
[53] Ebadollahi S, Xie L, Chang S F, et al. Visual Event Detection using Multi-Dimensional Concept Dynamics. Multimedia and Expo, 2006 IEEE International Conference on, 2006.881-884.
[54] Benitez Jimenez A B. Multimedia knowledge: discovery, classification, browsing, and retrieval [Doctor Thesis]. New York, NY, USA, 2005. Sponsor-Chang,, Shih-Fu.
[55] Fellbaum C, (eds.). WordNet: An Electronic Lexical Database. The MIT Press, May, 1998.
[56] Wang J, Li J, Wiederhold G. SIMPLIcity: semantics-sensitive integrated matching for picture libraries. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2001, 23(9):947-963.
[57] Fan J, Luo H, Xiao J, et al. Semantic video classification and feature subset selection under context and concept uncertainty. Digital Libraries, 2004. Proceedings of the 2004 Joint ACM/IEEE Conference on, 2004. 192-201.
[58] Fan J, Luo H, Elmagarmid A. Concept-oriented indexing of video databases: toward semantic sensitive retrieval and browsing. Image Processing, IEEE Transactions on, 2004, 13(7):974-992.
[59] WidrowB, Hoff ME. Adaptive switching circuits. 1988.123-134.
[60] Freund Y, Schapire R E. A decision-theoretic generalization of on-line learning and an application to boosting. Proceedings of European Conference on Computational Learning Theory, 1995. 23-37.
[61] Smith J R, Naphade M, Natsev A. Multimedia semantic indexing using model vectors. Proceedings of ICME '03: Proceedings of the 2003 International Conference on Multimedia and Expo, Washington, DC, USA: IEEE Computer Society, 2003. 445-448.
[62] Datta R, Ge W, Li J, et al. Toward bridging the annotation-retrieval gap in image search by a generative modeling approach. Proceedings of MULTIMEDIA '06: Proceedings of the 14th annual ACM international conference on Multimedia, New York, NY, USA: ACM, 2006. 977-986.
[63] Flickner M, Sawhney H, Niblack W, et al. Query by Image and Video Content: The QBIC System. Computer, 1995, 28(9):23-32.
[64] Gevers T, Smeulders A. PicToSeek: combining color and shape invariant features for image retrieval. Image Processing, IEEE Transactions on, Jan 2000, 9(1): 102—119.
[65] Gupta A, Jain R. Visual Information Retrieval. Communications of the ACM, 1997, 40(5):70-79.
[66] Ma W, Manjunath B. NeTra: a toolbox for navigating large image databases. icip, 1997, 1:568.
[67] Smith J R, Chang S F. VisualSEEk: a fully automated content-based image query system. Proceedings of MULTIMEDIA '96: Proceedings of the fourth ACM international conference on Multimedia, New York, NY, USA: ACM, 1996. 87-98.
[68] Smeulders A W, Worring M, Santini S, et al. Content-Based Image Retrieval at the End of the Early Years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(12):1349-1380.
[69] Li J, Wang J Z. Real-Time Computerized Annotation of Pictures. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(6):985-1002.
[70] Barnard K, Duygulu P, Forsyth D, et al. Matching words and pictures. J. Mach. Learn. Res., 2003,3:1107-1135.
[71] Tieu K, Viola P. Boosting Image Retrieval. Int. J. Comput. Vision, 2004, 56(1-2): 17-36.
[72] Cheng S F, Chen W, Sundaram H. Semantic visual templates: linking visual features to semantics. Image Processing, 1998. ICIP 98. Proceedings. 1998 International Conference on, 4-7 Oct 1998. 531-535 vol.3.
[73] Tong S, Chang E. Support vector machine active learning for image retrieval. Proceedings of MULTIMEDIA '01: Proceedings of the ninth ACM international conference on Multimedia, New York, NY, USA: ACM, 2001. 107-118.
[74] Zhang C, Chen T. An active learning framework for content-based information retrieval. Multimedia, IEEE Transactions on, Jun 2002, 4(2):260-268.
[75] Monay F, Gatica-Perez D. On image auto-annotation with latent space models. Proceedings of MULTIMEDIA '03: Proceedings of the eleventh ACM international conference on Multimedia, New York, NY, USA: ACM, 2003. 275-278.
[76] Singhal A, Luo J, Zhu W. Probabilistic spatial context models for scene content understanding. Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, 18-20 June 2003, 1:Ⅰ-235-Ⅰ-241 vol.1.
[77] He X, Ma W Y, Zhang H J. Learning an image manifold for retrieval. Proceedings of MULTIMEDIA '04: Proceedings of the 12th annual ACM international conference on Multimedia, New York, NY, USA: ACM, 2004. 17-23.
[78] Rui Y, Huang T, Ortega M, et al. Relevance feedback: a power tool for interactive content-based image retrieval. Circuits and Systems for Video Technology, IEEE Transactions on, Sep 1998, 8(5):644-655.
[79] Rui Y, Huang T. Optimizing learning in image retrieval. Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on, 2000, 1:236-243 vol.1.
[80] Kushki A, Androutsos P, Plataniotis K, et al. Query feedback for interactive image retrieval. Circuits and Systems for Video Technology, IEEE Transactions on, May 2004, 14(5):644-655.
[81] Guan J, Qiu G. Learning user intention in relevance feedback using optimization. Proceedings of MIR '07: Proceedings of the international workshop on Workshop on multimedia information retrieval, New York, NY, USA: ACM, 2007. 41-50.
[82] Liu J, Li Z, Li M, et al. Human behaviour consistent relevance feedback model for imageretrieval. Proceedings of MULTIMEDIA '07: Proceedings of the 15th international conference on Multimedia, New York, NY, USA: ACM, 2007. 269-272.
[83] Wenyin L, Dumais S, Sun Y, et al. Semi-Automatic Image Annotation. Proceedings of Proc. of Interact: Conference on HCI. IOS Press, 2001. 326-333.
[84] Dorado A, Izquierdo E. Semi-automatic image annotation using frequent keyword mining. Proceedings. Seventh International Conference on Information Visualization, 2003. IV 2003., 16-18 July 2003. 532-535.
[85] Yang C, Dong M, Fotouhi F. I~2A: an interactive image annotation system. 6-8 July 2005.
[86] Schapire R E, Singer Y. BoosTexter: A Boosting-based System for Text Categorization. Machine Learning, 2000, 39(2/3): 135-168.
[87] Rousu J, Saunders C, Szedmak S, et al. Kernel-Based Learning of Hierarchical Multilabel Classification Models. J. Mach. Learn. Res., 2006, 7:1601-1626.
[88] Elisseeff A, Weston J. A Kernel Method for Multi-Labelled Classification. Proceedings of Neural Information Processing Systems, 2001. 681-687.
[89] Kang F, Jin R, Sukthankar R. Correlated Label Propagation with Application to Multi-label Learning. Proceedings of CVPR '06: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, 2006. 1719-1726.
[90] Zhang M L, Zhou Z H. ML-KNN: A lazy learning approach to multi-label learning. Pattern Recogn., 2007,40(7):2038-2048.
[91] Zhu S, Ji X, Xu W, et al. Multi-labelled classification using maximum entropy method. Proceedings of SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, New York, NY, USA: ACM, 2005. 274-281.
[92] Qi G J, Hua X S, Rui Y, et al. Correlative multi-label video annotation. Proceedings of MULTIMEDIA '07: Proceedings of the 15th international conference on Multimedia, New York, NY, USA: ACM, 2007. 17-26.
[93] Zhang Z H. Multi-Label Learning by Instance Differentiation. Proceedings of Proceedings of the 22nd Conference on Artificial Intelligence, Vancouver, Canada, 2007. 669-674.
[94] Boutell M R, J. Luo X S, Brown C M. Learning multi-label scene classification. Pattern Recognition, 2004, 37(9):1757-1771.
[95] Goncalves T, Quaresma P. A Preliminary Approach to the Multilabel Classification Problem of Portuguese Juridical Documents. Proceedings of EPIA, 2003. 435-444.
[96] Li T, Ogihara M. Detecting emotion in music. Proceedings of ISMIR, 2003.
[97] Diplaris S, Tsoumakas G, Mitkas P A, et al. Protein Classification with Multiple Algorithms. Proceedings of Panhellenic Conference on Informatics, 2005. 448-456.
[98] Milind Ramesh Naphade; Kozintsev T. Factor graph framework for semantic video indexing. Circuits and Systems for Video Technology, IEEE Transactions on, 2002, 12(l):40-52.
[99] Wu B S J. Ontology-based multi-classification learning for video concept detection. IEEE International Conference on Multimedia and Expo, 2004., 2004, 2:1003-1006.
[100] Jacobs R A, Jordan M 1, Nowlan S J, et al. Adaptive mixtures of local experts. Neural Comput., 1991, 3(l):79-87.
[101] Bishop C, Svensen M. Bayesian Hierarchical Mixtures of Experts. San Francisco, CA:Morgan Kaufmann, 2003.
[102] Karmeshu J, (eds.). Entropy Measures, Maximum Entropy Principle and Emerging Applications. Springer, 2003: 115-136.
[103] Schwarz G. Estimating the Dimension of a Model. The Annals of Statistics, 1978, 6(2):461-464.
[104] Rissanen JJ. Modeling by shortest data description. Automatica, 1978, 14:465-471.
[105] Gader P D, Mohamed M A, Keller J M. Fusion of handwritten word classifiers. Pattern Recogn. Lett., 1996, 17(6):577-584.
[106] Ho T K, Hull J, Srihari S. Decision combination in multiple classifier systems. Pattern Analysis and Machine Intelligence, IEEE Transactions on, Jan 1994, 16(l):66-75.
[107] Qiu G. Indexing chromatic and achromatic patterns for content-based colour image retrieval. Pattern Recognition, 2002, 35(8): 1675-1686.
[108] De Comit (?) F, Gilleron R, Tommasi M. Learning Multi-label Alternating Decision Trees from Texts and Data. Machine Learning and Data Mining in Pattern Recognition, 2003. 251-274.
[109] Wang Y, Makedon F, Ford J, et al. Generating fuzzy semantic metadata describing spatial relations from images using the R-histogram. 2004. 202-211.
[110] Boutell M, Luo J. A Generalized Temporal Context Model for Semantic Scene Classification.2004. 104.
[111] Jang C J, Lee J Y, Lee J W, et al. Smart management system for digital photographs using temporal and spatial features with EXIF metadata. volume 1, 2007. 110-115.
[112] Subudhi B, Nanda P. An Evolutionary Based Slow and Fast Moving Video Object Detection Scheme Using Compound Markov Random Field Model. 2008. 398-405.
[113] Sheikh Y, Shah M. Bayesian modeling of dynamic scenes for object detection. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2005, 27(11): 1778-1792.
[114] Liu K H, Weng M F, Tseng C Y, et al. Association and Temporal Rule Mining for Post-Filtering of Semantic Concept Detection in Video. Multimedia, IEEE Transactions on, 2008, 10(2):240-251.
[2] Flickner M, Sawhney H, Niblack W, et al. Query by Image and Video Content: The QBIC System. Computer, 1995,28(9):23-32.
[3] Pentland A, Picard R, Sclaroff S. Photobook: Tools for content-based manipulation of image databases. Proceedings of Proceedings of the Conference on Storage and Retrieval for Image and Video Database Ⅱ, SPIE.
[4] Smith J R, Chang S F. VisualSEEk: a fully automated content-based image query system. Proceedings of MULTIMEDIA '96: Proceedings of the fourth ACM international conference on Multimedia, New York, NY, USA: ACM, 1996. 87-98.
[5] Ma W, Manjunath B. NeTra: a toolbox for navigating large image databases. Image Processing, 1997. Proceedings., International Conference on, 1997, 1:568-571.
[6] Smeulders A W M, Worring M, Santini S, et al. Content-Based Image Retrieval at the End of the Early Years. IEEE Trans. Pattern Anal. Mach. Intell., 2000,22(12): 1349-1380.
[7] Berchtold S, Keim D A, Kriegel H P. The X-tree: An Index Structure for High-Dimensional Data. Proceedings of VLDB'96. Morgan Kaufmann, 1996. 28-39.
[8] Henrich A. The LSDh-tree: an access structure for feature vectors. Data Engineering, 1998. Proceedings., 14th International Conference on, 1998. 362-369.
[9] Weber R, Schek H J, Blott S. A Quantitative Analysis and Performance Study for Similarity-Search Methods in High-Dimensional Spaces. Proceedings of VLDB '98: Proceedings of the 24rd International Conference on Very Large Data Bases, San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1998. 194-205.
[10] Bohm C. A cost model for query processing in high dimensional data spaces. ACM Trans. Database Syst., 2000, 25(2): 129-178.
[11] TRECVTD. http://www-nlpir.nist.gov/projects/trecvid/.
[12] Shi J, Malik J. Normalized cuts and image segmentation. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2000, 22(8):888-905.
[13] Deng Y, Manjunath B. Unsupervised segmentation of color-texture regions in images and video. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2001, 23(8):800- 810.
[14] Comaniciu D, Meer P. Mean shift: a robust approach toward feature space analysis. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2002, 24(5):603-619.
[15] Smith J R, Chang S F. Single color extraction and image query. Proceedings of ICIP '95: Proceedings of the 1995 International Conference on Image Processing (Vol. 3)-Volume 3, Washington, DC, USA: IEEE Computer Society, 1995.
[16] Haralick R, Shanmugan K, Dinstein I. Texture features for image classification. IEEE Transactions on System Management and Cybernetics, 1973, 3(6):610-621.
[17] Tamura H, Mori S, Yamawaki T. Texture features corresponding to visual perception. IEEE Transactions on System Management and Cybernetics, 1978, 8(6):460-473.
[18] Mallat S. A theory for multiresolution signal decomposition: the wavelet representation. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 1989, 11(7):674-693.
[19] Rui Y, She A, Huang T. Modified fourier descriptors for shape representation - a practical approach. Proceedings of Proc of First International Workshop on Image Databases and Multi Media Search., 1996. 456-461.
[20] Hu M K. Visual pattern recognition by moment invariants. Information Theory, IRE Transactions on, 1962, 8(2):179-187.
[21] Jolliffe I. Principal component analysis. New York, NY: Springer-Verlag, 1986.
[22] McLachlan G J. Discriminant Analysis and Statistical Pattern Recognition. Wiley-Interscience, 2004.
[23] Quinlan J R. C4.5: programs for machine learning. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1993.
[24] Hopfield J J. Neural Networks and Physical Systems with Emergent Collective Computational Abilities. PNAS, 1982, 79(8):2554-2558.
[25] Vapnik V N. The nature of statistical learning theory. New York, NY, USA: Springer-Verlag New York, Inc., 1995.
[26] Gray R, Olshen R. Vector quantization and density estimation. 1997. 172-193.
[27] Vailaya A, Figueiredo M, Jain A, et al. Image classification for content-based indexing.Image Processing, IEEE Transactions on, 2001, 10(1): 117-130.
[28] Huang C L, Shih H C, Chao C Y. Semantic analysis of soccer video using dynamic Bayesian network. Multimedia, IEEE Transactions on, 2006, 8(4):749-760.
[29] Vapnik V N. Estimation of Dependences Based on Empirical Data: Springer Series in Statistics (Springer Series in Statistics). Secaucus, NJ, USA: Springer-Verlag New York, Inc., 1982.
[30] Natsev A P, Naphade M R, Smith J R. Semantic representation: search and mining of multimedia content. Proceedings of KDD '04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, New York, NY, USA: ACM, 2004. 641-646.
[31] Sch(?)lkopf B, Platt J C, Shawe-Taylor J C, et al. Estimating the Support of a High-Dimensional Distribution. Neural Comput., 2001, 13(7): 1443-1471.
[32] Veropoulos K, Campbell C, Cristianini N. Controlling the sensitivity of support vector machines. Proceedings of Proceedings of the International Joint Conference on AI, 1999. 55-60.
[33] Wu G, Chang E Y. KBA: Kernel Boundary Alignment Considering Imbalanced Data Distribution. IEEE Transactions on Knowledge and Data Engineering (TKDE), 2005, 17(6):786-795.
[34] Yuan J, Li J, Zhang B. Learning concepts from large scale imbalanced data sets using support cluster machines. Proceedings of MULTIMEDIA '06: Proceedings of the 14th annual ACM international conference on Multimedia, New York, NY, USA: ACM, 2006. 441-450.
[35] Li B, Chi M, Fan J, et al. Support cluster machine. Proceedings of ICML '07: Proceedings of the 24th international conference on Machine learning, New York, NY, USA: ACM, 2007. 505-512.
[36] Carneiro G, Chan A B, Moreno P J. Supervised Learning of Semantic Classes for Image Annotation and Retrieval. IEEE Trans. Pattern Anal. Mach. Intell., 2007, 29(3):394-410. Member-Vasconcelos,, Nuno.
[37] Fan J, Luo H, Hacid M S. Mining images on semantics via statistical learning. Proceedings of KDD '05: Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, New York, NY, USA: ACM, 2005. 22-31.
[38] Assfalg J, Bertini M, Colombo C, et al. Automatic extraction and annotation of soccer video highlights. Image Processing, 2003. ICIP 2003. Proceedings. 2003 International Conference on, 2003, 2:Ⅱ-527-30 vol.3.
[39] Zhang J, Yang J, Hauptmann A G. A Discriminative Learning Framework with Pairwise Constraints for Video Object Classification. IEEE Trans. Pattern Anal. Mach. Intell., 2006, 28(4):578.
[40] Yang J, Hauptmann A G. 3WNews: who, where, and when in news video. Proceedings of MULTIMEDIA '06: Proceedings of the 14th annual ACM international conference on Multimedia, New York, NY, USA: ACM, 2006. 503-504.
[41] Hauptmann A G. Towards a Large Scale Concept Ontology for Broadcast Video. CIVR, 2004..
[42] Naphade M, Smith J R, Tesic J, et al. Large-scale concept ontology for multimedia. IEEE Multimedia, 2006, 13(3):86-91.
[43] Naphade M, Natsev A, Lin C Y, et al. Multi-granular detection of regional semantic concepts [video annotation]. Multimedia and Expo, 2004. ICME '04. 2004 IEEE International Conference on, 2004, 1:109-112.
[44] Naphade M, Smith J. Learning visual models of semantic concepts. Image Processing, 2003. ICIP 2003. Proceedings. 2003 International Conference on, 2003, 2:Ⅱ-531-4.
[45] Zhang R, Zhang Z. Image Database Classification based on Concept Vector Model. Multi media and Expo, 2005. ICME 2005. IEEE International Conference on, 2005. 93-96.
[46] Yao J, Zhang Z. Semi-supervised learning based object detection in aerial imagery. volume 1, 2005. 1011-1016.
[47] Yao J, Zhang Z. Object detection in aerial imagery based on enhanced semi-supervised learning. volume 2, 2005. 1012-1017.
[48] Naphade M, Smith J. A generalized multiple instance learning algorithm for large scale modeling of multimedia semantics. Acoustics, Speech, and Signal Processing, 2005. Proceedings. (ICASSP '05). IEEE International Conference on, 2005, 5:341-344.
[49] Chen Y, Bi J, Wang J. MILES: Multiple-Instance Learning via Embedded Instance Selection. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2006,28(12):1931-1947.
[50] Zhang D Q, Chang S F. A Generative-Discriminative Hybrid Method for Multi-View Ob ject Detection. Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, 2006, 2:2017-2024.
[51] Yan R, Chen M, Hauptmann A. Mining Relationship Between Video Concepts using Probabilistic Graphical Models. Multimedia and Expo, 2006 IEEE International Conference on, 2006. 301-304.
[52] Xie L, Chang S F. Pattern Mining in Visual Concept Streams. Multimedia and Expo, 2006 IEEE International Conference on, 2006. 297-300.
[53] Ebadollahi S, Xie L, Chang S F, et al. Visual Event Detection using Multi-Dimensional Concept Dynamics. Multimedia and Expo, 2006 IEEE International Conference on, 2006.881-884.
[54] Benitez Jimenez A B. Multimedia knowledge: discovery, classification, browsing, and retrieval [Doctor Thesis]. New York, NY, USA, 2005. Sponsor-Chang,, Shih-Fu.
[55] Fellbaum C, (eds.). WordNet: An Electronic Lexical Database. The MIT Press, May, 1998.
[56] Wang J, Li J, Wiederhold G. SIMPLIcity: semantics-sensitive integrated matching for picture libraries. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2001, 23(9):947-963.
[57] Fan J, Luo H, Xiao J, et al. Semantic video classification and feature subset selection under context and concept uncertainty. Digital Libraries, 2004. Proceedings of the 2004 Joint ACM/IEEE Conference on, 2004. 192-201.
[58] Fan J, Luo H, Elmagarmid A. Concept-oriented indexing of video databases: toward semantic sensitive retrieval and browsing. Image Processing, IEEE Transactions on, 2004, 13(7):974-992.
[59] WidrowB, Hoff ME. Adaptive switching circuits. 1988.123-134.
[60] Freund Y, Schapire R E. A decision-theoretic generalization of on-line learning and an application to boosting. Proceedings of European Conference on Computational Learning Theory, 1995. 23-37.
[61] Smith J R, Naphade M, Natsev A. Multimedia semantic indexing using model vectors. Proceedings of ICME '03: Proceedings of the 2003 International Conference on Multimedia and Expo, Washington, DC, USA: IEEE Computer Society, 2003. 445-448.
[62] Datta R, Ge W, Li J, et al. Toward bridging the annotation-retrieval gap in image search by a generative modeling approach. Proceedings of MULTIMEDIA '06: Proceedings of the 14th annual ACM international conference on Multimedia, New York, NY, USA: ACM, 2006. 977-986.
[63] Flickner M, Sawhney H, Niblack W, et al. Query by Image and Video Content: The QBIC System. Computer, 1995, 28(9):23-32.
[64] Gevers T, Smeulders A. PicToSeek: combining color and shape invariant features for image retrieval. Image Processing, IEEE Transactions on, Jan 2000, 9(1): 102—119.
[65] Gupta A, Jain R. Visual Information Retrieval. Communications of the ACM, 1997, 40(5):70-79.
[66] Ma W, Manjunath B. NeTra: a toolbox for navigating large image databases. icip, 1997, 1:568.
[67] Smith J R, Chang S F. VisualSEEk: a fully automated content-based image query system. Proceedings of MULTIMEDIA '96: Proceedings of the fourth ACM international conference on Multimedia, New York, NY, USA: ACM, 1996. 87-98.
[68] Smeulders A W, Worring M, Santini S, et al. Content-Based Image Retrieval at the End of the Early Years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(12):1349-1380.
[69] Li J, Wang J Z. Real-Time Computerized Annotation of Pictures. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(6):985-1002.
[70] Barnard K, Duygulu P, Forsyth D, et al. Matching words and pictures. J. Mach. Learn. Res., 2003,3:1107-1135.
[71] Tieu K, Viola P. Boosting Image Retrieval. Int. J. Comput. Vision, 2004, 56(1-2): 17-36.
[72] Cheng S F, Chen W, Sundaram H. Semantic visual templates: linking visual features to semantics. Image Processing, 1998. ICIP 98. Proceedings. 1998 International Conference on, 4-7 Oct 1998. 531-535 vol.3.
[73] Tong S, Chang E. Support vector machine active learning for image retrieval. Proceedings of MULTIMEDIA '01: Proceedings of the ninth ACM international conference on Multimedia, New York, NY, USA: ACM, 2001. 107-118.
[74] Zhang C, Chen T. An active learning framework for content-based information retrieval. Multimedia, IEEE Transactions on, Jun 2002, 4(2):260-268.
[75] Monay F, Gatica-Perez D. On image auto-annotation with latent space models. Proceedings of MULTIMEDIA '03: Proceedings of the eleventh ACM international conference on Multimedia, New York, NY, USA: ACM, 2003. 275-278.
[76] Singhal A, Luo J, Zhu W. Probabilistic spatial context models for scene content understanding. Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, 18-20 June 2003, 1:Ⅰ-235-Ⅰ-241 vol.1.
[77] He X, Ma W Y, Zhang H J. Learning an image manifold for retrieval. Proceedings of MULTIMEDIA '04: Proceedings of the 12th annual ACM international conference on Multimedia, New York, NY, USA: ACM, 2004. 17-23.
[78] Rui Y, Huang T, Ortega M, et al. Relevance feedback: a power tool for interactive content-based image retrieval. Circuits and Systems for Video Technology, IEEE Transactions on, Sep 1998, 8(5):644-655.
[79] Rui Y, Huang T. Optimizing learning in image retrieval. Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on, 2000, 1:236-243 vol.1.
[80] Kushki A, Androutsos P, Plataniotis K, et al. Query feedback for interactive image retrieval. Circuits and Systems for Video Technology, IEEE Transactions on, May 2004, 14(5):644-655.
[81] Guan J, Qiu G. Learning user intention in relevance feedback using optimization. Proceedings of MIR '07: Proceedings of the international workshop on Workshop on multimedia information retrieval, New York, NY, USA: ACM, 2007. 41-50.
[82] Liu J, Li Z, Li M, et al. Human behaviour consistent relevance feedback model for imageretrieval. Proceedings of MULTIMEDIA '07: Proceedings of the 15th international conference on Multimedia, New York, NY, USA: ACM, 2007. 269-272.
[83] Wenyin L, Dumais S, Sun Y, et al. Semi-Automatic Image Annotation. Proceedings of Proc. of Interact: Conference on HCI. IOS Press, 2001. 326-333.
[84] Dorado A, Izquierdo E. Semi-automatic image annotation using frequent keyword mining. Proceedings. Seventh International Conference on Information Visualization, 2003. IV 2003., 16-18 July 2003. 532-535.
[85] Yang C, Dong M, Fotouhi F. I~2A: an interactive image annotation system. 6-8 July 2005.
[86] Schapire R E, Singer Y. BoosTexter: A Boosting-based System for Text Categorization. Machine Learning, 2000, 39(2/3): 135-168.
[87] Rousu J, Saunders C, Szedmak S, et al. Kernel-Based Learning of Hierarchical Multilabel Classification Models. J. Mach. Learn. Res., 2006, 7:1601-1626.
[88] Elisseeff A, Weston J. A Kernel Method for Multi-Labelled Classification. Proceedings of Neural Information Processing Systems, 2001. 681-687.
[89] Kang F, Jin R, Sukthankar R. Correlated Label Propagation with Application to Multi-label Learning. Proceedings of CVPR '06: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, 2006. 1719-1726.
[90] Zhang M L, Zhou Z H. ML-KNN: A lazy learning approach to multi-label learning. Pattern Recogn., 2007,40(7):2038-2048.
[91] Zhu S, Ji X, Xu W, et al. Multi-labelled classification using maximum entropy method. Proceedings of SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, New York, NY, USA: ACM, 2005. 274-281.
[92] Qi G J, Hua X S, Rui Y, et al. Correlative multi-label video annotation. Proceedings of MULTIMEDIA '07: Proceedings of the 15th international conference on Multimedia, New York, NY, USA: ACM, 2007. 17-26.
[93] Zhang Z H. Multi-Label Learning by Instance Differentiation. Proceedings of Proceedings of the 22nd Conference on Artificial Intelligence, Vancouver, Canada, 2007. 669-674.
[94] Boutell M R, J. Luo X S, Brown C M. Learning multi-label scene classification. Pattern Recognition, 2004, 37(9):1757-1771.
[95] Goncalves T, Quaresma P. A Preliminary Approach to the Multilabel Classification Problem of Portuguese Juridical Documents. Proceedings of EPIA, 2003. 435-444.
[96] Li T, Ogihara M. Detecting emotion in music. Proceedings of ISMIR, 2003.
[97] Diplaris S, Tsoumakas G, Mitkas P A, et al. Protein Classification with Multiple Algorithms. Proceedings of Panhellenic Conference on Informatics, 2005. 448-456.
[98] Milind Ramesh Naphade; Kozintsev T. Factor graph framework for semantic video indexing. Circuits and Systems for Video Technology, IEEE Transactions on, 2002, 12(l):40-52.
[99] Wu B S J. Ontology-based multi-classification learning for video concept detection. IEEE International Conference on Multimedia and Expo, 2004., 2004, 2:1003-1006.
[100] Jacobs R A, Jordan M 1, Nowlan S J, et al. Adaptive mixtures of local experts. Neural Comput., 1991, 3(l):79-87.
[101] Bishop C, Svensen M. Bayesian Hierarchical Mixtures of Experts. San Francisco, CA:Morgan Kaufmann, 2003.
[102] Karmeshu J, (eds.). Entropy Measures, Maximum Entropy Principle and Emerging Applications. Springer, 2003: 115-136.
[103] Schwarz G. Estimating the Dimension of a Model. The Annals of Statistics, 1978, 6(2):461-464.
[104] Rissanen JJ. Modeling by shortest data description. Automatica, 1978, 14:465-471.
[105] Gader P D, Mohamed M A, Keller J M. Fusion of handwritten word classifiers. Pattern Recogn. Lett., 1996, 17(6):577-584.
[106] Ho T K, Hull J, Srihari S. Decision combination in multiple classifier systems. Pattern Analysis and Machine Intelligence, IEEE Transactions on, Jan 1994, 16(l):66-75.
[107] Qiu G. Indexing chromatic and achromatic patterns for content-based colour image retrieval. Pattern Recognition, 2002, 35(8): 1675-1686.
[108] De Comit (?) F, Gilleron R, Tommasi M. Learning Multi-label Alternating Decision Trees from Texts and Data. Machine Learning and Data Mining in Pattern Recognition, 2003. 251-274.
[109] Wang Y, Makedon F, Ford J, et al. Generating fuzzy semantic metadata describing spatial relations from images using the R-histogram. 2004. 202-211.
[110] Boutell M, Luo J. A Generalized Temporal Context Model for Semantic Scene Classification.2004. 104.
[111] Jang C J, Lee J Y, Lee J W, et al. Smart management system for digital photographs using temporal and spatial features with EXIF metadata. volume 1, 2007. 110-115.
[112] Subudhi B, Nanda P. An Evolutionary Based Slow and Fast Moving Video Object Detection Scheme Using Compound Markov Random Field Model. 2008. 398-405.
[113] Sheikh Y, Shah M. Bayesian modeling of dynamic scenes for object detection. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2005, 27(11): 1778-1792.
[114] Liu K H, Weng M F, Tseng C Y, et al. Association and Temporal Rule Mining for Post-Filtering of Semantic Concept Detection in Video. Multimedia, IEEE Transactions on, 2008, 10(2):240-251.