参数字典稀疏表示的完全无监督域适应
|
摘要
无监督域适应(unsupervised domain adaptation,UDA)针对的是源域有标记而目标域无标记的学习问题,其目的是利用从标记大样本源域中所学"知识"来促进无标记小样本目标域的学习性能。但现实中也往往存在样本无标记的源域,形成了所谓的完全无监督域适应。该问题给域适应学习带来了严峻的挑战。受先前提出的软大间隔聚类学习(soft large margin clustering,SLMC)启发,提出了一种参数迁移方法——参数字典稀疏表示的完全无监督的域适应方法(whole UDA,WUDA)。SLMC采用分类学习思想在输出(标记)空间中实现给定数据的聚类,在这种实现原理的启发下,从参数(决策函数的权重矩阵)公共字典的角度,在源域和目标域的权重间进行互适应参数字典学习实现知识迁移,同时引入l_(2,1)范数来约束字典系数矩阵,使得各域权重可从公共字典中自适应地选择,从而实现域适应学习。最后,在相关数据集上的实验显示了WUDA在聚类性能上的显著有效性。
Unsupervised domain adaptation(UDA) concerns on the learning problem in which the source domain contains labeled samples and target domain only contains unlabeled samples. Its goal is to use the"knowledge"learnt from source domain with a large number of labeled samples to promote learning performance in the target domain where all the samples are unlabeled. In reality, however, there often exists the situation in which the samples are also unlabeled in the source domain, leading to the so-called whole unsupervised domain adaptation. This problem brings severe challenge to domain adaptation learning. To address this, inspired by soft large margin clustering(SLMC) which is proposed previously, this paper proposes a parameter-transfer method, i.e., whole unsupervised domain adaptation using sparse representation of parameter dictionary(WUDA). Specifically, borrowing the idea of SLMC that conducts data clustering in the output(label) space, WUDA realizes the knowledge transfer from the perspective of common dictionary of parameter(weight matrix of decision function) and adaptively learns the parameter dictionary between the weights of source domain and target domain. In addition, a l_(2,1) norm regularization term is introduced to constrain the coefficient matrix of dictionary, which makes the domain weight adaptively selectable from the common dictionary. Finally, the experimental results on the related datasets show the significant improvement of WUDA on the clustering performance.
Unsupervised domain adaptation(UDA) concerns on the learning problem in which the source domain contains labeled samples and target domain only contains unlabeled samples. Its goal is to use the"knowledge"learnt from source domain with a large number of labeled samples to promote learning performance in the target domain where all the samples are unlabeled. In reality, however, there often exists the situation in which the samples are also unlabeled in the source domain, leading to the so-called whole unsupervised domain adaptation. This problem brings severe challenge to domain adaptation learning. To address this, inspired by soft large margin clustering(SLMC) which is proposed previously, this paper proposes a parameter-transfer method, i.e., whole unsupervised domain adaptation using sparse representation of parameter dictionary(WUDA). Specifically, borrowing the idea of SLMC that conducts data clustering in the output(label) space, WUDA realizes the knowledge transfer from the perspective of common dictionary of parameter(weight matrix of decision function) and adaptively learns the parameter dictionary between the weights of source domain and target domain. In addition, a l_(2,1) norm regularization term is introduced to constrain the coefficient matrix of dictionary, which makes the domain weight adaptively selectable from the common dictionary. Finally, the experimental results on the related datasets show the significant improvement of WUDA on the clustering performance.
引文
[1]Jiang J.A literature survey on domain adaptation of statistical classifiers[EB/OL].(2008-03).http://sifaka.cs.uiuc.edu/jiang4/domainadaptation/survey.
[2]Pan S J,Yang Q.A survey on transfer learning[J].IEEETransactions on Knowledge and Data Engineering,2010,22(10):1345-1359.
[3]DauméIII H.Frustratingly easy domain adaptation[J].arxiv:0907.1815,2009:256-263.
[4]Blitzer J,McDonald R T,Pereira F.Domain adaptation with structural correspondence learning[C]//Proceedings of the2006 Conference on Empirical Methods in Natural Language Processing,Sydney,Jul 22-23,2006.Stroudsburg:ACL,2006:120-128.
[5]Li Z,Zhang Y,Wei Y,et al.End-to-end adversarial memory network for cross-domain sentiment classification[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence,Melbourne,Aug 19-25,2017.IJCAI,2017:2237-2243.
[6]Xia R,Hu X L,Lu J F,et al.Instance selection and instance weighting for cross-domain sentiment classification via PUlearning[C]//Proceedings of the 23rd International Joint Conference on Artificial Intelligence,Beijing,Aug 3-9,2013.Menlo Park:AAAI,2013:2176-2182.
[7]Sohn K,Liu S F,Zhong G Y,et al.Unsupervised domain adaptation for face recognition in unlabeled videos[J].arXiv:1708.02191,2017.
[8]Gopalan R,Li R N,Chellappa R.Domain adaptation for object recognition:an unsupervised approach[C]//Proceedings of the IEEE International Conference on Computer Vision,Barcelona,Nov 6-13,2011.Washington:IEEE Computer Society,2011:999-1006.
[9]Li Y,Wang L,Wang J,et al.Transfer learning for survival analysis via efficient L2,1-norm regularized cox regression[C]//Proceedings of the 16th International Conference on Data Mining,Barcelona,Dec 12-15,2016.Piscataway:IEEE,2016:231-240.
[10]Kamnitsas K,Baumgartner C F,Ledig C,et al.Unsupervised domain adaptation in brain lesion segmentation with adversarial networks[C]//LNCS 10265:Proceedings of the 25th International Conference on Information Processing in Medical Imaging,Boone,Jun 25-30,2017.Berlin,Heidelberg:Springer,2017:597-609.
[11]Evgeniou T,Pontil M.Regularized multi-task learning[C]//Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,Seattle,Aug 22-25,2004.New York:ACM,2004:109-117.
[12]Xu S H,Mu X D,Chai D,et al.Domain adaptation algorithm with ELM parameter transfer[J].Acta Automatica Sinica,2018,44(2):311-317.
[13]Yosinski J,Clune J,Bengio Y,et al.How transferable are features in deep neural networks?[C]//Proceedings of the Annual Conference on Neural Information Processing Systems,Montreal,Dec 8-13,2014.Cambridge:MIT Press,2014:3320-3328.
[14]Fernando B,Habrard A,Sebban M,et al.Unsupervised visual domain adaptation using subspace alignment[C]//Proceedings of the IEEE International Conference on Computer Vision,Sydney,Dec 1-8,2013.Washington:IEEE Computer Society,2013:2960-2967.
[15]Sun B C,Feng J S,Saenko K.Return of frustratingly easy domain adaptation[C]//Proceedings of the 30th AAAI Conference on Artificial Intelligenc,Phoenix,Feb 12-17,2016.Menlo Park:AAAI,2016:2058-2065.
[16]Sun B C,Saenko K.Deep coral:correlation alignment for deep domain adaptation[C]//LNCS 9915:Proceedings of the14th European Conference on Computer Vision,Amsterdam,Oct 8-16,2016.Berlin,Heidelberg:Springer,2016:443-450.
[17]Morerio P,Murino V.Correlation alignment by riemannian metric for domain adaptation[J].arXiv:1705.08180,2017.
[18]Arsigny V,Fillard P,Pennec X,et al.Geometric means in a novel vector space structure on symmetric positive-definite matrices[J].SIAM Journal on Matrix Analysis and Applications,2007,29(1):328-347.
[19]Pan S J,Tsang I W,Kwok J T,et al.Domain adaptation via transfer component analysis[J].IEEE Transactions on Neural Networks,2011,22(2):199-210.
[20]Zellinger W,Grubinger T,Lughofer E,et al.Central moment discrepancy(CMD)for domain-invariant representation learning[J].arXiv:1702.08811,2017.
[21]Huang J Y,Gretton A J,Gretton A,et al.Correcting sample selection bias by unlabeled data[C]//Proceedings of the20th Annual Conference on Neural Information Processing Systems,Vancouver,Dec 3-6,2007.Cambridge:MIT Press,2007:601-608.
[22]Li S,Song S J,Huang G.Prediction reweighting for domain adaptation[J].IEEE Transactions on Neural Networks and Learning Systems,2017,28(7):1682-1695.
[23]Wang Y Y,Chen S C.Soft large margin clustering[J].Information Sciences,2013,232:116-129.
[24]Garcia-Romero D,Mc Cree A.Supervised domain adaptation for I-vector based speaker recognition[C]//Proceedings of the International Conference on Acoustics,Speech and Signal Processing,Florence,May 4-9,2014.Piscataway:IEEE,2014:4047-4051.
[25]DauméIII H,Kumar A,Saha A.Frustratingly easy semisupervised domain adaptation[C]//Proceedings of the 2010Workshop on Domain Adaptation for Natural Language Processing,Uppsala,Jul 15,2010.Stroudsburg:ACL,2010:53-59.
[26]Kumar A,Saha A,Daume H.Co-regularization based semisupervised domain adaptation[C]//Proceedings of the 24th Annual Conference on Neural Information Processing Systems,Vancouver,Dec 6-11,2010.Red Hook:Curran Associates,2010:478-486.
[27]Donahue J,Hoffman J,Rodner E,et al.Semi-supervised domain adaptation with instance constraints[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Portland,Jun 23-28,2013.Washington:IEEEComputer Society,2013:668-675.
[28]Gong B Q,Shi Y,Sha F,et al.Geodesic flow kernel for unsupervised domain adaptation[C]//Proceedings of the Conference on Computer Vision and Pattern Recognition,Providence,Jun 16-21,2012.Washington:IEEE Computer Society,2012:2066-2073.
[29]Dai W Y,Yang Q,Xue G R,et al.Self-taught clustering[C]//Proceedings of the 25th International Conference on Machine Learning,Helsinki,Jul 5-9,2008.New York:ACM,2008:200-207.
[30]Jiang W H,Chung F L.Transfer spectral clustering[C]//LNCS 7524:Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases,Bristol,Sep 23-27,2012.Berlin,Heidelberg:Springer,2012:789-803.
[31]Deng Z H,Jiang Y Z,Chung F L,et al.Transfer prototypebased fuzzy clustering[J].IEEE Transactions on Fuzzy Systems,2016,24(5):1210-1232.
[32]Schwaighofer A,Tresp V,Yu K.Learning Gaussian process kernels via hierarchical Bayes[C]//Proceedings of the Neural Information Processing Systems,Vancouver,Dec 5-8,2005.Cambridge:MIT Press,2005:1209-1216.
[33]Ni J,Qiu Q,Chellappa R.Subspace interpolation via dictionary learning for unsupervised domain adaptation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Portland,Jun 23-28,2013.Washington:IEEEComputer Society,2013:692-699.
[34]Li S,Li K,Fu Y.Self-taught low-rank coding for visual learning[J].IEEE Transactions on Neural Networks and Learning Systems,2018,29(3):645-656.
[35]Xu L L,Neufeld J,Larson B,et al.Maximum margin clustering[C]//Proceedings of the Neural Information Processing Systems,Vancouver,Dec 5-8,2005.Cambridge:MIT Press,2005:1537-1544.
[36]Bezdek J C,Ehrlich R,Full W.FCM:the fuzzy C-means clustering algorithm[J].Computers&Geosciences,1984,10(2):191-203.
[37]Tseng P.Convergence of a block coordinate descent method for nondifferentiable minimization[J].Journal of Optimization Theory and Applications,2001,109(3):475-494.
[38]Nie F P,Huang H,Cai X,et al.Efficient and robust feature selection via joint?2,1-norms minimization[C]//Proceedings of the 24th Annual Conference on Neural Information Processing System,Vancouver,Dec 6-11,2010.Red Hook:Curran Associates,2010:1813-1821.
[12]许夙晖,慕晓冬,柴栋,等.基于极限学习机参数迁移的域适应算法[J].自动化学报, 2018, 44(2):311-317.
[2]Pan S J,Yang Q.A survey on transfer learning[J].IEEETransactions on Knowledge and Data Engineering,2010,22(10):1345-1359.
[3]DauméIII H.Frustratingly easy domain adaptation[J].arxiv:0907.1815,2009:256-263.
[4]Blitzer J,McDonald R T,Pereira F.Domain adaptation with structural correspondence learning[C]//Proceedings of the2006 Conference on Empirical Methods in Natural Language Processing,Sydney,Jul 22-23,2006.Stroudsburg:ACL,2006:120-128.
[5]Li Z,Zhang Y,Wei Y,et al.End-to-end adversarial memory network for cross-domain sentiment classification[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence,Melbourne,Aug 19-25,2017.IJCAI,2017:2237-2243.
[6]Xia R,Hu X L,Lu J F,et al.Instance selection and instance weighting for cross-domain sentiment classification via PUlearning[C]//Proceedings of the 23rd International Joint Conference on Artificial Intelligence,Beijing,Aug 3-9,2013.Menlo Park:AAAI,2013:2176-2182.
[7]Sohn K,Liu S F,Zhong G Y,et al.Unsupervised domain adaptation for face recognition in unlabeled videos[J].arXiv:1708.02191,2017.
[8]Gopalan R,Li R N,Chellappa R.Domain adaptation for object recognition:an unsupervised approach[C]//Proceedings of the IEEE International Conference on Computer Vision,Barcelona,Nov 6-13,2011.Washington:IEEE Computer Society,2011:999-1006.
[9]Li Y,Wang L,Wang J,et al.Transfer learning for survival analysis via efficient L2,1-norm regularized cox regression[C]//Proceedings of the 16th International Conference on Data Mining,Barcelona,Dec 12-15,2016.Piscataway:IEEE,2016:231-240.
[10]Kamnitsas K,Baumgartner C F,Ledig C,et al.Unsupervised domain adaptation in brain lesion segmentation with adversarial networks[C]//LNCS 10265:Proceedings of the 25th International Conference on Information Processing in Medical Imaging,Boone,Jun 25-30,2017.Berlin,Heidelberg:Springer,2017:597-609.
[11]Evgeniou T,Pontil M.Regularized multi-task learning[C]//Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,Seattle,Aug 22-25,2004.New York:ACM,2004:109-117.
[12]Xu S H,Mu X D,Chai D,et al.Domain adaptation algorithm with ELM parameter transfer[J].Acta Automatica Sinica,2018,44(2):311-317.
[13]Yosinski J,Clune J,Bengio Y,et al.How transferable are features in deep neural networks?[C]//Proceedings of the Annual Conference on Neural Information Processing Systems,Montreal,Dec 8-13,2014.Cambridge:MIT Press,2014:3320-3328.
[14]Fernando B,Habrard A,Sebban M,et al.Unsupervised visual domain adaptation using subspace alignment[C]//Proceedings of the IEEE International Conference on Computer Vision,Sydney,Dec 1-8,2013.Washington:IEEE Computer Society,2013:2960-2967.
[15]Sun B C,Feng J S,Saenko K.Return of frustratingly easy domain adaptation[C]//Proceedings of the 30th AAAI Conference on Artificial Intelligenc,Phoenix,Feb 12-17,2016.Menlo Park:AAAI,2016:2058-2065.
[16]Sun B C,Saenko K.Deep coral:correlation alignment for deep domain adaptation[C]//LNCS 9915:Proceedings of the14th European Conference on Computer Vision,Amsterdam,Oct 8-16,2016.Berlin,Heidelberg:Springer,2016:443-450.
[17]Morerio P,Murino V.Correlation alignment by riemannian metric for domain adaptation[J].arXiv:1705.08180,2017.
[18]Arsigny V,Fillard P,Pennec X,et al.Geometric means in a novel vector space structure on symmetric positive-definite matrices[J].SIAM Journal on Matrix Analysis and Applications,2007,29(1):328-347.
[19]Pan S J,Tsang I W,Kwok J T,et al.Domain adaptation via transfer component analysis[J].IEEE Transactions on Neural Networks,2011,22(2):199-210.
[20]Zellinger W,Grubinger T,Lughofer E,et al.Central moment discrepancy(CMD)for domain-invariant representation learning[J].arXiv:1702.08811,2017.
[21]Huang J Y,Gretton A J,Gretton A,et al.Correcting sample selection bias by unlabeled data[C]//Proceedings of the20th Annual Conference on Neural Information Processing Systems,Vancouver,Dec 3-6,2007.Cambridge:MIT Press,2007:601-608.
[22]Li S,Song S J,Huang G.Prediction reweighting for domain adaptation[J].IEEE Transactions on Neural Networks and Learning Systems,2017,28(7):1682-1695.
[23]Wang Y Y,Chen S C.Soft large margin clustering[J].Information Sciences,2013,232:116-129.
[24]Garcia-Romero D,Mc Cree A.Supervised domain adaptation for I-vector based speaker recognition[C]//Proceedings of the International Conference on Acoustics,Speech and Signal Processing,Florence,May 4-9,2014.Piscataway:IEEE,2014:4047-4051.
[25]DauméIII H,Kumar A,Saha A.Frustratingly easy semisupervised domain adaptation[C]//Proceedings of the 2010Workshop on Domain Adaptation for Natural Language Processing,Uppsala,Jul 15,2010.Stroudsburg:ACL,2010:53-59.
[26]Kumar A,Saha A,Daume H.Co-regularization based semisupervised domain adaptation[C]//Proceedings of the 24th Annual Conference on Neural Information Processing Systems,Vancouver,Dec 6-11,2010.Red Hook:Curran Associates,2010:478-486.
[27]Donahue J,Hoffman J,Rodner E,et al.Semi-supervised domain adaptation with instance constraints[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Portland,Jun 23-28,2013.Washington:IEEEComputer Society,2013:668-675.
[28]Gong B Q,Shi Y,Sha F,et al.Geodesic flow kernel for unsupervised domain adaptation[C]//Proceedings of the Conference on Computer Vision and Pattern Recognition,Providence,Jun 16-21,2012.Washington:IEEE Computer Society,2012:2066-2073.
[29]Dai W Y,Yang Q,Xue G R,et al.Self-taught clustering[C]//Proceedings of the 25th International Conference on Machine Learning,Helsinki,Jul 5-9,2008.New York:ACM,2008:200-207.
[30]Jiang W H,Chung F L.Transfer spectral clustering[C]//LNCS 7524:Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases,Bristol,Sep 23-27,2012.Berlin,Heidelberg:Springer,2012:789-803.
[31]Deng Z H,Jiang Y Z,Chung F L,et al.Transfer prototypebased fuzzy clustering[J].IEEE Transactions on Fuzzy Systems,2016,24(5):1210-1232.
[32]Schwaighofer A,Tresp V,Yu K.Learning Gaussian process kernels via hierarchical Bayes[C]//Proceedings of the Neural Information Processing Systems,Vancouver,Dec 5-8,2005.Cambridge:MIT Press,2005:1209-1216.
[33]Ni J,Qiu Q,Chellappa R.Subspace interpolation via dictionary learning for unsupervised domain adaptation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Portland,Jun 23-28,2013.Washington:IEEEComputer Society,2013:692-699.
[34]Li S,Li K,Fu Y.Self-taught low-rank coding for visual learning[J].IEEE Transactions on Neural Networks and Learning Systems,2018,29(3):645-656.
[35]Xu L L,Neufeld J,Larson B,et al.Maximum margin clustering[C]//Proceedings of the Neural Information Processing Systems,Vancouver,Dec 5-8,2005.Cambridge:MIT Press,2005:1537-1544.
[36]Bezdek J C,Ehrlich R,Full W.FCM:the fuzzy C-means clustering algorithm[J].Computers&Geosciences,1984,10(2):191-203.
[37]Tseng P.Convergence of a block coordinate descent method for nondifferentiable minimization[J].Journal of Optimization Theory and Applications,2001,109(3):475-494.
[38]Nie F P,Huang H,Cai X,et al.Efficient and robust feature selection via joint?2,1-norms minimization[C]//Proceedings of the 24th Annual Conference on Neural Information Processing System,Vancouver,Dec 6-11,2010.Red Hook:Curran Associates,2010:1813-1821.
[12]许夙晖,慕晓冬,柴栋,等.基于极限学习机参数迁移的域适应算法[J].自动化学报, 2018, 44(2):311-317.