[1]王诚,王宇成.基于Simhash 的大规模文档去重改进算法研究[J].计算机技术与发展,2019,29(02):115-119.[doi:10．3969/j．issn．1673－629X．2019．02．024]
　WANG Cheng,WANG Yucheng.Research on Improved Large－scale Documents Deduplication Algorithm Based on Simhash[J].,2019,29(02):115-119.[doi:10．3969/j．issn．1673－629X．2019．02．024]
点击复制

基于Simhash 的大规模文档去重改进算法研究()
分享到：

《计算机技术与发展》[ISSN:1006-6977/CN:61-1281/TN]

卷:
29
期数:
2019年02期
页码:
115-119
栏目:
智能、算法、系统工程
出版日期:
2019-02-10

文章信息/Info

Title:
Research on Improved Large－scale Documents Deduplication Algorithm Based on Simhash
文章编号:
1673－629X(2019)02－0115－05
作者:
王诚; 王宇成
南京邮电大学 通信与信息工程学院，江苏 南京 210003
Author(s):
WANG Cheng; WANG Yu－cheng
School of Telecommunications ＆ Information Engineering，Nanjing University of Posts and Telecommunications，Nanjing 210003，China
关键词:
Web 大数据; Simhash; 近似文本检测; 多维度; 二次哈希
Keywords:
Web big data; Simhash; approximate text detection; multi－dimension; secondary hash
分类号:
TP311
DOI:
10．3969/j．issn．1673－629X．2019．02．024
摘要:
针对大规模文档去重算法 Simhash 存在的缺点和不足，提出一种改进的 Simhash 算法。首先从多个维度综合计算文档的相似度，包括文档内容、文档关键字、文档的标签、文档的引用文献等方面，定义一个新的公式用于计算文档相似度。其次改进 Simhash 算法计算文档特征的方法，通过 TF－IDF 技术和单词的主题相关性综合计算单词的权重，TF－IDF 技术用于计算一个关键词在一个文档集中的一篇文档的重要性，将专业术语词汇的长度统计函数作为判断单词主题相关性的依据。最后在检索步骤中采用哈希到桶的思想，此时出现分布不均匀的情况，为此设定一个阈值，当超过阈值时，对桶内的元素进行二次哈希，可以减少候选对的数量并且使分布更加均匀。实验结果表明，改进后的算法可以明显提高原Simhash 算法的效率和准确率。
Abstract:
:Aiming at the shortcomings and deficiencies of Simhash，we present an improved Simhash algorithm． Firstly，the similarity of documents from multiple dimensions is calculated，including document content，document keywords，document labels and references，and a new formula is defined to calculate document similarity． Secondly，the process of Simhash algorithm calculating document features is improved，and the weight of words is calculated synthetically by TF－IDF technique and the topic relevance of words． TF－IDF technology is used to calculate the importance of a document with a keyword in a document set． The term statistical function of term length is used as the basis for determining the relevance of a word subject． Finally，the idea of hashing to buckets is adopted in the retrieval． At this time，there is an uneven distribution，so a threshold is set． When the threshold is exceeded，the elements in the bucket are hashed twice，which can reduce the number of candidate pairs and make the distribution more evenly． Experiment shows that the improved algorithm can significantly improve the efficiency and accuracy of the traditional algorithm．

相似文献/References:

[1]徐济惠. 基于Simhash算法的海量文档反作弊技术研究[J].计算机技术与发展,2014,24(09):103.
　XU Ji-hui. Research on Huge Amounts of Documents Anti-spamming Technique Based on Simhash Algorithm[J].,2014,24(02):103.
[2]彭双和,图尔贡·麦提萨比尔,周巧凤. 基于Simhash的中文文本去重技术研究[J].计算机技术与发展,2017,27(11):137.
　PENG Shuang-he,Tuergong MAITISABIER,ZHOU Qiao-feng. Research on Deduplication Technique of Chinese Text with Simhash[J].,2017,27(02):137.
[3]张亚男,陈卫卫,付印金,等.基于 Simhash 改进的文本去重算法[J].计算机技术与发展,2022,32(08):26.[doi:10. 3969 / j. issn. 1673-629X. 2022. 08. 005]
　ZHANG Ya-nan,CHEN Wei-wei,FU Yin-jin,et al.Improved Text Deduplication Algorithm Based on Simhash[J].,2022,32(02):26.[doi:10. 3969 / j. issn. 1673-629X. 2022. 08. 005]

更新日期/Last Update: 2019-02-10