基于深度学习的连续手语语句识别算法-《计算机技术与发展》

文章信息/Info

Title:: Recognition Algorithm for Continuous Sign Language Sentence Based on Deep Learning

作者:: 李摇晨¹; 黄元元¹; 胡作进²; 1. 南京航空航天大学计算机科学与技术学院,江苏南京 211106; 2. 南京特殊教育师范学院数学与信息科学学院,江苏南京 210038

Author(s):: LI Chen¹; HUANG Yuan-yuan¹ ; HU Zuo-jin²; 1. School of Computer Science and Technology,Nanjing University of Aeronautics and Astronautics,Nanjing 211106,China;?
2. School of Math and Information Science,Nanjing Normal University of Special Education,Nanjing 210038,China

Keywords:: continuous sign language sentence recognition; transition action; convolutional neural network; long short-term memory network; transition probability between words

摘要:: 目前,关于连续手语语句识别的研究相对较少,原因在于难以有效地分割出手语词。该文利用卷积神经网络提取手语词的手型特征,同时利用轨迹归一化算法提取手语词的轨迹特征,并在此基础上完成长短期记忆网络的构建,从而为手语语句识别准备好手语词分类器。对于一个待识别的手语语句,采用基于右手心轨迹信息的分割算法来检测过渡动作。由过渡动作可以将语句分割为多个片段,考虑到某些过渡动作可能是手语词内部的动作,所以将若干个片段拼接成一个复合段,并按照层次遍历的次序对所有复合段运用手语词分类器进行识别。最后,采用跨段搜索的动态规划算法寻找最大后验概率的词汇序列,从而完成手语语句的识别。实验结果表明,该算法可以对 47 个常用手语词组成的语句做出识别,且具有较高的准确性和实时性。

Abstract:: At present,the researches on sign language sentences recognition are relatively few,because it is difficult to effectively split out sign language words. For sign language words,a convolutional neural network is used to extract hand-shape feature,and the trajectory normalization algorithm is used to extract trajectory feature. Based on that,a long short-term memory network is completed so as to prepare a sign-words classifier for sentence recognition. For a sign language sentence to be recognized, it can be split into several fragments by transition actions which are detected by the segmentation algorithm based on the right-hand trajectory. Since some transition actions may be actions inside words, we stitch several fragments into a composite segment and all composite segments are classified by the sign-words classifier in the order of hierarchical traverse. Finally,a dynamic programming algorithm with cross-segment search is used to find the word sequence with the greatest posterior probability,which realizes the recognition of sign language sente-nce. The experiment shows that the proposed algorithm can recognize sentences composed of 47 commonly used sign language words with high accuracy and real-time performance.