combining TEXT.build_vocab with flair embeddings

❓ Questions and Help

Description

Hi, we can use glove embedding when building vocab, using something like:

MIN_FREQ = 2

TEXT.build_vocab(train_data, 
                 min_freq = MIN_FREQ,
                 vectors = "glove.6B.300d",
                 unk_init = torch.Tensor.normal_)

We also can create embeddings using flair library, using for example:

embedding_types: List[TokenEmbeddings] = [
 
    WordEmbeddings('glove'),
 
    # comment in this line to use character embeddings
    #CharacterEmbeddings(),
 
    # comment in these lines to use flair embeddings
    FlairEmbeddings('news-forward'),
    FlairEmbeddings('news-backward'),
    ELMoEmbeddings(),
    BertEmbeddings('bert-base-uncased'),
]
 
embeddings: StackedEmbeddings = StackedEmbeddings(embeddings=embedding_types)

Could I use the above embeddings instead of glove in the above code? Is anything similar to this supported?

[mttk]

There are two options, both equally good: Option 1: Copy data from a tensor

vectors = # initialize the vector tensor or leave at None
embedding = nn.Embedding(num_tokens, embedding_size, padding_idx=0)
if vectors is not None: embedding.weight.data.copy_(vectors)
if freeze_vectors: embedding.weight.requires_grad = False

Option 2: use from_pretrained embedding = nn.Embedding.from_pretrained(vectors)

[mttk]

@Edresson the example code would work only for the non-contextualized embeddings, in the case you want to obtain the embedding matrix vectors. This would of course fail when applied on a concrete instance.

If you want to use flair contextual embeddings, I believe you need to load and use the dataset with their pipeline. There might be a workaround, but I’m not familiar with one at the moment as I don’t actively use flair.