Support Removing BPE when encoding with Sentencepiece

I’m using SentencePiece instead of subword-nmt to tokenize the data. Problem: During evaluation the flag --remove-bpe is useless:

In SentencePiece

1. the bpe-token changes from @@ to ▁.
2. first whitespace needs to be removed and second the bpe-token needs to be replaced with whitespace.

Currently you can fulfill (1) by passing “▁” as an argument for the --remove-bpe flag, but that does not eliminate the additional whitespace from (2).

[myleott]

Just following up on this, you can now detokenize sentencepiece with --remove-bpe=sentencepiece

[bkj]

Seeing the same problem here in the Romanian example – if you use --remove-bpe=sentencepiece in fairseq-generate it removes all of the spaces from the S T and D lines.