I train with translate_enzh_wmt32k, and bleu is only 1.25.What's the reason?I appreciate you telling me

Description

I trained model with translate_enzh_wmt32k, but bleu is only 1.25. What’s the reason?I appreciate you can tell me, thank you!

Environment information

OS: linux

Steps to reproduce:

just flow the doc command

something just like this:
$ PROBLEM=translate_enzh_wmt32k
$ MODEL=transformer
$ HPARAMS=transformer_base_single_gpu
$ DATA_DIR=$HOME/t2t_data
$ TMP_DIR=/tmp/t2t_datagen
$ TRAIN_DIR=$HOME/t2t_train/$PROBLEM/$MODEL-$HPARAMS
$ T2T_USR_DIR=$HOME/t2t_usr_dir

$ t2t-trainer \
  --data_dir=$DATA_DIR \
  --problem=$PROBLEM \
  --model=$MODEL \
  --hparams_set=$HPARAMS \
  --output_dir=$TRAIN_DIR

[yynil]

The solution is pretty simple and straight forward.

1. Extract all unique characters in the corpus including symbols.
2. Use the characters as dictionary and just use the text token encoder.
3. Train the model using transformer_base configuration.
4. It took us 1.5 millions steps to get bleu score 22~23.

All we believe is that for unigram language like Chinese and Japanese, it’s unnecessary to do word cut any more because the deep neural network will learn how to connect characters as words better than any word cut library. The translation model proved my guess might be right. 发自我的 iPhone

在 2018年11月3日，20:34，ConnectDotz [e-mail] 写道：

@yynil thx, did you just use a simple one-hot encoding then? is there an example (parameter?) on how to plug in our own tokenizer? would you care to share more detail? Actually, if the current implementation could not reliably produce a reasonable result(?), will you consider contributing your solution?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[hpulfc]

finally , you should have word segmentation for Chinese, then it will be suitable from SubTokenEncoder