About Programmatically usage

I’m trying to use the package programatically. I’m doing

    from subword_nmt.apply_bpe import BPE, read_vocabulary
     # read/write files as UTF-8
    bpe_codes_fin = codecs.open(bpe_codes, encoding='utf-8')
    bpe_vocab_fin = codecs.open(bpe_vocab, encoding='utf-8')
    vocabulary = read_vocabulary(bpe_vocab_fin, vocabulary_threshold)

    bpe = BPE(bpe_codes_fin, merges=-1, separator='@@', vocab=vocabulary, glossaries=None)
    codes = bpe.process_line(line)

Is that correct? Also, I’m not sure of the vocabulary_threshold, since I do not see any default value. Is there any one?

Thank you.

[rsennrich]

try adding this as the first line to the BPE file:

#version: 0.2

the reason for this is explained in the README. It looks like fastBPE implements the new variant (v 0.2) as well.

[rsennrich]

As to your first question, have a look at your vocabulary file - whether you set the threshold to 5 or 500 won’t make a big difference for you, since most rare tokens are single (non-Latin) characters that won’t be affected by this.

FAIR LASER uses a different BPE implementation ( https://github.com/glample/fastBPE ), which seems to store the BPE file in a different format. It might work if you simply remove the third item in each entry (the frequency), but I can’t guarantee there’s no other inconsistency, e.g. in how UTF-8 whitespace is handled.