基于P2P的校园网搜索引擎的设计与实现
摘要
校园网是学校信息化建设的基础设施,是广大师生完成教学和科研的一个重要平台。基于P2P校园网搜索引擎利用P2P分布式体系结构,将网络中的资源分布在互联网中的各个节点上。并组建了基于“小世界”现象的兴趣网络,把整个资源共享范围缩小到校园网内,有效解决了P2P网络的带宽占用问题,减轻了校园网路由出口的压力,同时也提高了节点间彼此传输文件的速度。
基于P2P校园网搜索引擎采用了现今流行的两种开源技术——JXTA和Nutch。利用JXTA建立一个通用的、统一的P2P网络平台,采用IP多播技术的动态发现机制和汇聚对等体的静态发现机制,减少了资源浪费。同时采用XML方式,对其资源信息进行描述及传递,并采用了XML信息的缓存技术,通告在其经过的所有中间对等体处都会被缓存,从而使系统性能随着系统“变老”而不断提升。利用Nutch技术构建了基于Lucene的Web搜索引擎,比目前普遍采用的搜索引擎(如Baidu、google等)更具有针对性,将搜索范围限制在校园主页及其相关链接中。在网页爬行过程中采用了广度优先遍历方式,并允许服务器管理者自由指定搜索的深度,使该系统更具有实效性。
University network is the basic facility on the construction of school informationlize. It is also an important platform, with which the teachers and students will teach and research. The university network based on P2P save its resource in the various peers with P2P distributed architecture. Besides, the program builds the interest network based on "small word", which means that the scope of the files shared is limited within the university network. So we can make full use of the resource in our university. It also provides an effective solution to the bandwidth occupied in order to reduce the big pressure on our network export. At the same time, it speeds up file transfers between peers.
The implementation of university search engine based on P2P using two popular open-source technology - JXTA and Nutch. On one hand, the system builds a common and unified P2P network with JXTA. JXTA provides the dynamic discovery mechanism with ip multicast and the static discovery mechanism with rendezvous peers. It can reduce the waste of the bandwidth. At the same time, it describes and transfers information with XML. The advertisement is saved in the middle peers, which just transfer information. On the other hand, the system builds web search engine based on Lucene technology with Nutch. The aim of designing it is more specific than the most popular search engines (such as Baidu, google, etc.). It just search the university home page and the related links. Web crawling use the breadth-first traversal and allow the server administrator to decide the depth of search, so the system is more effective.
基于P2P校园网搜索引擎采用了现今流行的两种开源技术——JXTA和Nutch。利用JXTA建立一个通用的、统一的P2P网络平台,采用IP多播技术的动态发现机制和汇聚对等体的静态发现机制,减少了资源浪费。同时采用XML方式,对其资源信息进行描述及传递,并采用了XML信息的缓存技术,通告在其经过的所有中间对等体处都会被缓存,从而使系统性能随着系统“变老”而不断提升。利用Nutch技术构建了基于Lucene的Web搜索引擎,比目前普遍采用的搜索引擎(如Baidu、google等)更具有针对性,将搜索范围限制在校园主页及其相关链接中。在网页爬行过程中采用了广度优先遍历方式,并允许服务器管理者自由指定搜索的深度,使该系统更具有实效性。
University network is the basic facility on the construction of school informationlize. It is also an important platform, with which the teachers and students will teach and research. The university network based on P2P save its resource in the various peers with P2P distributed architecture. Besides, the program builds the interest network based on "small word", which means that the scope of the files shared is limited within the university network. So we can make full use of the resource in our university. It also provides an effective solution to the bandwidth occupied in order to reduce the big pressure on our network export. At the same time, it speeds up file transfers between peers.
The implementation of university search engine based on P2P using two popular open-source technology - JXTA and Nutch. On one hand, the system builds a common and unified P2P network with JXTA. JXTA provides the dynamic discovery mechanism with ip multicast and the static discovery mechanism with rendezvous peers. It can reduce the waste of the bandwidth. At the same time, it describes and transfers information with XML. The advertisement is saved in the middle peers, which just transfer information. On the other hand, the system builds web search engine based on Lucene technology with Nutch. The aim of designing it is more specific than the most popular search engines (such as Baidu, google, etc.). It just search the university home page and the related links. Web crawling use the breadth-first traversal and allow the server administrator to decide the depth of search, so the system is more effective.
引文
[1]何丰如.P2P技术及其在信息检索中的应用[J].广东广播电视大学学报,2005,14(2):20-25
[2]胡红霞,王振兴,李申堂.搜索引擎技术的现状及发展趋势[J].信息工程大学学报,2001,2(4):66-69
[3]丁宬杰.搜索引擎技术的研究与实现[D].上海:上海交通大学,2006
[4]吴海明.基于Lucene的搜索引擎技术的研究与改进[D].广州:暨南大学,2006
[5]马蓉平.网站搜索引擎的设计与实现[D].沈阳:沈阳大学,2006
[6]赵恒,陈杰.P2P技术的应用及其研究现状[N].电信快报,2004(9)
[7]董小国,文玉锋.P2P技术在电子商务中的应用[J].中国科技信息,2006(2):45-47
[8]门涛.P2P网络信誉评价体系的设计[D].成都:电子科技大学,2005
[9]吴名宇.基于信任模型的资源共享技术的研究与实现[D].南京:东南大学,2004
[10]刘颖.基于P2P之上移动代理的电子商务模型及其安全性的研究[D].山东:山东大学,2005
[11]杜呈伟.基于P2P的应用共享系统的研究与实现[D].南京:东南大学,2006
[12]张晓刚.基于JXTA技术的P2P应用系统的研究与开发[J].微型电脑应用,2006,22(1):25-27
[13]宋雪昌.基于JXTA对等网络的即时通信系统的研究和实现[D].兰州:兰州大学,2005
[14]杨雪.P2P平台JXTA的协议分析与应用[D]。成都:西南交通大学,2007
[15]周华,武兵.JXTA技术与应用发展[J].福建电脑,2006(11):32-33
[16]姜超.JXTA分布式计算技术[J].现代计算机,2005(1):81-84
[17]王娜.基于JXTA的P2P技术的研究和实现[D].沈阳:沈阳工业大学,2006
[18]王瑞.基于JXTA的局域网文件共享系统研究[D].大连:大连交通大学,2004
[19]徐越.具有信任评价机制的CMS系统的设计与研究[D].兰州:兰州大学,2007
[20]谢艳春.基于 JXTA的P2P服务的研究[J].电脑开发与应用,2006,19(5):44-46
[21]Rohit Khare,Ph.D.,Doug Cutting,Kragen Sitaker,Adam Rifkin.Nutch:A Flexible and Scalable Open-Source Web Search Engine[R].CommerceNet Labs Technical Report,2004.
[22]曹羽中,曹勇刚,金茂忠,刘超.支持智能中文分词的互联网搜索引擎的构建[J].计算机工程与设计,2006,27(23):4395-4398
[23]Tom White.Introduction to Nutch.http://today.java.net/pub/a/today/2006/01/10/introductionto-nutch-1.html,2006.
[24]Scott Oaks,Bernard Traversat,Li Gong.JXTA in a Nutshell[M].北京:清华大学出版社,2004.
[25]胡放明,李俊兵,贺贵明,丁健.对P2P网中发现机制的研究[J].计算机应用,2004,24(6):46-47
[26]许加兴,汪厚新,周艳,金炜东.P2P网中基于IP组播的发现机制研究[J].计算机技术与发展,2006,16(6):32-37
[27]张智,李瑞轩.基于JXTA架构的对等网络关键技术研究[J].计算机应用研究,2007,24(3):253-259
[28]许云松,王泽兵.基于P2P的分布式工作流平台[J].计算机工程与设计,2005,26(8):2187-2192.
[29]Guan Boo Chyan,Wang Ping,Chen Sheng-Jyhetal.An extended objectoriented seeurity model for high secure offiee environment.Security Technology[R],IEEE 37th Annual 2003 Intemational CamahanConferene,Taipei,Taiwan.2003
[30]William Tolone,Gail-Joon Ahn,Tanusree Pai,Seng_Phil Hong.Access control in collaborative systems[J].ACM Computing Surveys,2005,37(1):29-41
[31]Joseph Gradecki,Joe Gradecki.Mastering JXTA:Building Java Peer-to-Peer Applications[M].Indianapolis Indiana:Wiley Publishing,Inc,2002.
[32]崔泽永,常晓燕.搜索引擎的Web Robot技术与优化[J].微机发展,2004,14[4]:99-112
[33]王亮.搜速引擎及其相关性排序研究[D].湖北:武汉大学,2004
[34]吴小竹.基于JAVA的多线程Spider的设计与实现[J].福建电脑,2004(6):62-63
[35]李春明.基于全文检索的自动答疑系统的研究和设计[D].苏州:苏州大学,2005
[36]姜华.基于Lucene面向主题搜索引擎的研究和设计[D].华东师范大学,2007
[37]李进松,江春华.使用JXTA实现IP地址穿越[J].信息安全,2006,22(3):7-9
[38]Sun Microsystems Inc,The Internet Society.JXTA Protocols.http://spec.jxta.org/nonav/v1.0/docbook/JXTAProtocols.html
[2]胡红霞,王振兴,李申堂.搜索引擎技术的现状及发展趋势[J].信息工程大学学报,2001,2(4):66-69
[3]丁宬杰.搜索引擎技术的研究与实现[D].上海:上海交通大学,2006
[4]吴海明.基于Lucene的搜索引擎技术的研究与改进[D].广州:暨南大学,2006
[5]马蓉平.网站搜索引擎的设计与实现[D].沈阳:沈阳大学,2006
[6]赵恒,陈杰.P2P技术的应用及其研究现状[N].电信快报,2004(9)
[7]董小国,文玉锋.P2P技术在电子商务中的应用[J].中国科技信息,2006(2):45-47
[8]门涛.P2P网络信誉评价体系的设计[D].成都:电子科技大学,2005
[9]吴名宇.基于信任模型的资源共享技术的研究与实现[D].南京:东南大学,2004
[10]刘颖.基于P2P之上移动代理的电子商务模型及其安全性的研究[D].山东:山东大学,2005
[11]杜呈伟.基于P2P的应用共享系统的研究与实现[D].南京:东南大学,2006
[12]张晓刚.基于JXTA技术的P2P应用系统的研究与开发[J].微型电脑应用,2006,22(1):25-27
[13]宋雪昌.基于JXTA对等网络的即时通信系统的研究和实现[D].兰州:兰州大学,2005
[14]杨雪.P2P平台JXTA的协议分析与应用[D]。成都:西南交通大学,2007
[15]周华,武兵.JXTA技术与应用发展[J].福建电脑,2006(11):32-33
[16]姜超.JXTA分布式计算技术[J].现代计算机,2005(1):81-84
[17]王娜.基于JXTA的P2P技术的研究和实现[D].沈阳:沈阳工业大学,2006
[18]王瑞.基于JXTA的局域网文件共享系统研究[D].大连:大连交通大学,2004
[19]徐越.具有信任评价机制的CMS系统的设计与研究[D].兰州:兰州大学,2007
[20]谢艳春.基于 JXTA的P2P服务的研究[J].电脑开发与应用,2006,19(5):44-46
[21]Rohit Khare,Ph.D.,Doug Cutting,Kragen Sitaker,Adam Rifkin.Nutch:A Flexible and Scalable Open-Source Web Search Engine[R].CommerceNet Labs Technical Report,2004.
[22]曹羽中,曹勇刚,金茂忠,刘超.支持智能中文分词的互联网搜索引擎的构建[J].计算机工程与设计,2006,27(23):4395-4398
[23]Tom White.Introduction to Nutch.http://today.java.net/pub/a/today/2006/01/10/introductionto-nutch-1.html,2006.
[24]Scott Oaks,Bernard Traversat,Li Gong.JXTA in a Nutshell[M].北京:清华大学出版社,2004.
[25]胡放明,李俊兵,贺贵明,丁健.对P2P网中发现机制的研究[J].计算机应用,2004,24(6):46-47
[26]许加兴,汪厚新,周艳,金炜东.P2P网中基于IP组播的发现机制研究[J].计算机技术与发展,2006,16(6):32-37
[27]张智,李瑞轩.基于JXTA架构的对等网络关键技术研究[J].计算机应用研究,2007,24(3):253-259
[28]许云松,王泽兵.基于P2P的分布式工作流平台[J].计算机工程与设计,2005,26(8):2187-2192.
[29]Guan Boo Chyan,Wang Ping,Chen Sheng-Jyhetal.An extended objectoriented seeurity model for high secure offiee environment.Security Technology[R],IEEE 37th Annual 2003 Intemational CamahanConferene,Taipei,Taiwan.2003
[30]William Tolone,Gail-Joon Ahn,Tanusree Pai,Seng_Phil Hong.Access control in collaborative systems[J].ACM Computing Surveys,2005,37(1):29-41
[31]Joseph Gradecki,Joe Gradecki.Mastering JXTA:Building Java Peer-to-Peer Applications[M].Indianapolis Indiana:Wiley Publishing,Inc,2002.
[32]崔泽永,常晓燕.搜索引擎的Web Robot技术与优化[J].微机发展,2004,14[4]:99-112
[33]王亮.搜速引擎及其相关性排序研究[D].湖北:武汉大学,2004
[34]吴小竹.基于JAVA的多线程Spider的设计与实现[J].福建电脑,2004(6):62-63
[35]李春明.基于全文检索的自动答疑系统的研究和设计[D].苏州:苏州大学,2005
[36]姜华.基于Lucene面向主题搜索引擎的研究和设计[D].华东师范大学,2007
[37]李进松,江春华.使用JXTA实现IP地址穿越[J].信息安全,2006,22(3):7-9
[38]Sun Microsystems Inc,The Internet Society.JXTA Protocols.http://spec.jxta.org/nonav/v1.0/docbook/JXTAProtocols.html
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |