sentencepiece分词效果测试

首先安装依赖包:

sudo apt-get install autoconf automake libtool libprotobuf9v5 protobuf-compiler libprotobuf-dev

下载sentencepiece : git clone https://github.com/google/sentencepiece

编译与安装sentencepiece:

cd /path/to/sentencepiece./autogen.sh./configuremakemake checksudo make installsudo ldconfig

训练sentencepiece 模型:

spm_train --input=<input> --model_prefix=<model_name> --vocab_size=8000 --model_type=<type>

sentencepiece包括unigram (default), bpe, char, or word等4种模型,训练的时候可以设置对应的model_type.

训练之后,会保存模型文件model_name.model,和词典文件model_name.vocab.

测试分词效果:

输入为一句话:

echo “在北京天安门广场” | spm_encode –model=model_name.model

输入文件:

spm_encode –model=model_name.model –output_format=piece input.txt

原理可参考:http://blog.csdn.net/u011961856/article/details/77165539