Duplicate aliases

Hi @DeNeutoy and team!

This is just a general question, but I was wondering about the duplicate aliases in the UMLS linker. For example, this is the entry for C3657270 (Nivolumab):

CUI: C3657270, Name: nivolumab
Definition: A fully human immunoglobulin (Ig) G4 monoclonal antibody directed against the negative immunoregulatory human cell surface receptor programmed death-1 (PD-1, PCD-1) with immune checkpoint inhibitory and antineoplastic activities. Upon administration, nivolumab binds to and blocks the activation of PD-1, an immunoglobulin superfamily (IgSF) transmembrane protein, by its ligands programmed cell death ligand 1 (PD-L1), which is overexpressed on certain cancer cells, and programmed cell death ligand 2 (PD-L2), which is primarily expressed on antigen-presenting cells (APCs). This results in the activation of T-cells and cell-mediated immune responses against tumor cells. Activated PD-1 negatively regulates T-cell activation and plays a key role in tumor evasion from host immunity.
TUI(s): T116, T121, T129
Aliases (abbreviated, total: 19): 
         nivolumab, nivolumab, nivolumab, nivolumab, nivolumab, Nivolumab, Nivolumab, Nivolumab, Nivolumab, Nivolumab

The 19 aliases listed for Nivolumab are the following, which contain quite some duplicates:

aliases = linker.umls.cui_to_entity["C3657270"].aliases
print(aliases)
> ['nivolumab', 'nivolumab', 'nivolumab', 'nivolumab', 'nivolumab', 'Nivolumab', 'Nivolumab', 'Nivolumab', 'Nivolumab', 'Nivolumab', 'Nivolumab', 'NIVOLUMAB', 'NIVOLUMAB', 'NIVOLUMAB', 'Nivolumab (substance)', 'NIVO', 'NIVO', 'Product containing nivolumab (medicinal product)', 'Nivolumab-containing product']
print(set(aliases))
> {'NIVOLUMAB', 'nivolumab', 'NIVO', 'Nivolumab', 'Nivolumab (substance)', 'Nivolumab-containing product', 'Product containing nivolumab (medicinal product)'}

Why are there duplicates in these lists? Do these maybe originate from the different vocabularies in UMLS (corresponding to the atoms)? And related to this question: could it make the entity linker more efficient if these aliases were de-duplicated?

[kaushikacharya]

By default strings are converted into lowercase by TfidfVectorizer

https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html

lowercase : bool, default=True
Convert all characters to lowercase before tokenizing.

So isn’t it that when removing duplicate aliases, we should ignore the case? In that case, in the example mentioned by @ChantalvanSon 'NIVOLUMAB', 'nivolumab', 'Nivolumab' becomes same? So this can lead to further reduction in size of concept_aliases.json

TfIdf vectorizer is called at https://github.com/allenai/scispacy/blob/master/scispacy/candidate_generation.py#L410

    tfidf_vectorizer = TfidfVectorizer(
        analyzer="char_wb", ngram_range=(3, 3), min_df=10, dtype=numpy.float32
    )

which means we are using the default value for the parameter lowercase

A question:

@DeNeutoy @danielkingai2 As we change the list of concept aliases, it would also change the vector representation of these concept aliases since the document frequency of the char trigram vocabulary also changes. Isn’t that going to impact the similarity score of entity candidate with the concept aliases?

[kaushikacharya]

@DeNeutoy @danielkingai2 In case you have missed it, I have made a suggestion on creating unique aliases in the conversation of https://github.com/allenai/scispacy/pull/274#issuecomment-726096229