[1]徐济惠. 基于Simhash算法的海量文档反作弊技术研究[J].计算机技术与发展,2014,24(09):103-107.
　XU Ji-hui. Research on Huge Amounts of Documents Anti-spamming Technique Based on Simhash Algorithm[J].,2014,24(09):103-107.
点击复制

 基于Simhash算法的海量文档反作弊技术研究()
分享到：

《计算机技术与发展》[ISSN:1006-6977/CN:61-1281/TN]

卷:
24
期数:
2014年09期
页码:
103-107
栏目:
智能、算法、系统工程
出版日期:
2014-09-10

文章信息/Info

Title:
 Research on Huge Amounts of Documents Anti-spamming Technique Based on Simhash Algorithm
文章编号:
1673-629X（2014）09-0103-05
作者:
徐济惠
 宁波城市学院
Author(s):
 XU Ji-hui
关键词:
 ; 重复文本检测; Simhash; 反作弊; 签名计算
Keywords:
 ; duplicate document detection; Simhash; anti-spamming; signature calculation
分类号:
TP311
文献标志码:
A
摘要:
 以互联网重复文档反作弊需求为背景，研究了基于Simhash的海量文档反作弊技术。以Simhash算法为文档判重的核心算法作基础对该算法获取文档特征的过程进行改进，将单词意义作为衡量单词权重的一个考量因素。针对64位文档Simhash签名，提供用户维度、全文维度和黑库维度的文档判重服务，并可基于全文和段落两种粒度进行文档相似性比较。通过测试数据和分析，该技术能保证运行稳定，每个实例可存储1亿文档，平均请求耗时稳定在20 ms左右，高峰期请求耗时会增长，但一般不会超过100 ms。
Abstract:
 On the background of the anti-spamming needs of repeated documents in Internet,research the anti-spamming technique based on the Simhash on huge amounts of documents. On the basis of taking the Simhash algorithm as core algorithm in duplicate document de-tection,improve the procedure of achieving document features of this algorithm. It takes the meaning of words as a consideration factor in measuring the weight of words. Aiming at the Simhash signature of a 64-bit,provide the document service of user dimension,the full di-mension and black dimension,and make a similarity comparison based on the full text and paragraphs. Through test data and analysis,this technique can guarantee the stable operation,100 million documents can be memorized in each example. The average request response time is about 20 ms. The response time will increase during the peak hour,but,in general,will not go over 100 ms.

相似文献/References:

[1]张志宏,吴庆波,邵立松,等.基于飞腾平台TOE协议栈的设计与实现[J].计算机技术与发展,2014,24(07):1.
　ZHANG Zhi-hong,WU Qing-bo,SHAO Li-song,et al. Design and Implementation of TCP/IP Offload Engine Protocol Stack Based on FT Platform[J].,2014,24(09):1.
[2]梁文快,李毅. 改进的基因表达算法对航班优化排序问题研究[J].计算机技术与发展,2014,24(07):5.
　LIANG Wen-kuai,LI Yi. Research on Optimization of Flight Scheduling Problem Based on Improved Gene Expression Algorithm[J].,2014,24(09):5.
[3]黄静,王枫,谢志新,等. EAST文档管理系统的设计与实现[J].计算机技术与发展,2014,24(07):13.
　HUANG Jing,WANG Feng,XIE Zhi-xin,et al. Design and Implementation of EAST Document Management System[J].,2014,24(09):13.
[4]侯善江[],张代远[][][]. 基于样条权函数神经网络P2P流量识别方法[J].计算机技术与发展,2014,24(07):21.
　HOU Shan-jiang[],ZHANG Dai-yuan[][][]. P2P Traffic Identification Based on Spline Weight Function Neural Network[J].,2014,24(09):21.
[5]李璨,耿国华,李康,等. 一种基于三维模型的文物碎片线图生成方法[J].计算机技术与发展,2014,24(07):25.
　LI Can,GENG Guo-hua,LI Kang,et al. A Method of Obtaining Cultural Debris’ s Line Chart Based on Three-dimensional Model[J].,2014,24(09):25.
[6]翁鹤,皮德常. 混沌RBF神经网络异常检测算法[J].计算机技术与发展,2014,24(07):29.
　WENG He,PI De-chang. Chaotic RBF Neural Network Anomaly Detection Algorithm[J].,2014,24(09):29.
[7]刘茜[],荆晓远[],李文倩[],等. 基于流形学习的正交稀疏保留投影[J].计算机技术与发展,2014,24(07):34.
　LIU Qian[],JING Xiao-yuan[,LI Wen-qian[],et al. Orthogonal Sparsity Preserving Projections Based on Manifold Learning[J].,2014,24(09):34.
[8]尚福华,李想,巩淼. 基于模糊框架-产生式知识表示及推理研究[J].计算机技术与发展,2014,24(07):38.
　SHANG Fu-hua,LI Xiang,GONG Miao. Research on Knowledge Representation and Inference Based on Fuzzy Framework-production[J].,2014,24(09):38.
[9]叶偲,李良福,肖樟树. 一种去除运动目标重影的图像镶嵌方法研究[J].计算机技术与发展,2014,24(07):43.
　YE Si,LI Liang-fu,XIAO Zhang-shu. Research of an Image Mosaic Method for Removing Ghost of Moving Targets[J].,2014,24(09):43.
[10]余松平[][],蔡志平[],吴建进[],等. GSM-R信令监测选择录音系统设计与实现[J].计算机技术与发展,2014,24(07):47.
　YU Song-ping[][],CAI Zhi-ping[] WU Jian-jin[],GU Feng-zhi[]. Design and Implementation of an Optional Voice Recording System Based on GSM-R Signaling Monitoring[J].,2014,24(09):47.

更新日期/Last Update: 2015-04-01