How do I share the vocab between the source and target language for machine translation

Hi I do I create combined vocabulary from the source and target fields from the multi30k dataset . I am interested in having a shared encoder which can represent source as well the target words.

SRC = Field(tokenize=tokenize_de,
            init_token='<sos>',
            eos_token='<eos>',
            lower=True,
            batch_first=True)

TRG = Field(tokenize=tokenize_en,
            init_token='<sos>',
            eos_token='<eos>',
            lower=True,
            batch_first=True)

train_data, valid_data, test_data = Multi30k.splits(exts=('.de', '.en'),
                                                    fields=(SRC, TRG))

SRC.build_vocab(train_data, min_freq=2)
TRG.build_vocab(train_data, min_freq=2)

[bentrevett]

@thak123

SRC.build_vocab(train_data.src, train_data.trg, min_freq=2)
TRG.vocab = SRC.vocab

This will cause your SRC and TRG to share a single vocab. You can then use one embedding layer for both languages instead of one per language.

[thak123]

omg… @bentrevett thanks for the code.