基于新词发现的古典文学作品分词方法研究-《计算机技术与发展》

文章信息/Info

Title:: Study on Word Segmentation Method of Classical Literature Based on New Word Discovery

作者:: 高嘉琦¹ ; 赵庆聪¹; 2; 1. 北京信息科技大学信息管理学院,北京 100192;
2. 绿色发展大数据决策北京市重点实验室,北京 100192

Author(s):: GAO Jia-qi1 ; ZHAO Qing-cong1; 2; 1. School of Information Management,Beijing Information Science and Technology University,Beijing 100192, China;
2. Beijing Key Laboratory of Big Data Decision for Green Development,Beijing 100192,China

Keywords:: classical literature; new word discovery; word segmentation; mutual information; left-right entropy

摘要:: 对于中文文本的分词研究来说,现有的分词方法和技术较多都是针对现代汉语,现代汉语的分词方法和体系已经很成熟,但对古代汉语的研究较少。由于古文的特殊性,将现代汉语的分词方法技术直接用于古汉语时,无法得到分词准确的理想效果,目前对古汉语分词方法的研究还未形成成熟的体系。文中提出一种基于新词发现的古典文学作品分词方法,即从大量古典文学作品语料中发现新词,构建古汉语分词词典,在此基础上再对古文文本进行分词。以《三国演义》古文文本处理为例,验证了基于新词发现的古典文学作品分词方法能有效提高古文分词的准确率。

Abstract:: For the research on word segmentation of Chinese text,most of the existing word segmentation methods? ? and technologies are aimed at modern Chinese. The word segmentation methods and systems of modern Chinese have been quite mature,but there are few studies on ancient Chinese. Due to the particularity of ancient Chinese, when the modern Chinese word segmentation method and technology are directly used in ancient Chinese,the ideal effect of accurate word segmentation cannot be obtained. At present,the word segmentation of ancient Chinese has not yet formed a general method and mature system. We propose a method of word segmentation in classical literature based on neologism discovery, that is, discovering new words from a large number of classical literary works,constructing an ancient Chinese word segmentation dictionary,and then segmenting the ancient text on this basis. Taking the ancient text processing of "The Romance of the Three Kingdoms" as an example,it is verified that the word segmentation method of classical literary works based on the discovery of new words can effectively improve the accuracy of ancient text segmentation.