Same model & data, similarity scores changed

Hi there, I’ve just upgraded to version 0.3.8 from 0.3.2. After the update, I noticed that running the same notebook I was working on, which finds the most similar texts to predefined queries, returned different rankings than before (e.g. one sentence that had similarity score about 0.9 wrt the query - and was therefore ranked as the most similar text in the corpus - is now ranked second with a score of about 0.6). I am loading the same corpus, using the same text queries and the same model (distiluse-base-multilingual-cased) as before. Any idea what may have caused this change? I realize that several releases occurred in between so it may be close to impossible to answer, but if you have any clue I’d be curious to know. Thanks in advance!

[nreimers]

Hi, I uploaded the old version as distiluse-base-multilingual-case-v1, which is a version that supports (only) 15 languages.

The version that supports 50+ languages is uploaded as distiluse-base-multilingual-case-v2.

[freeIsa]

If you are curious and want to look into this, this is a few lines of code demonstrating that embeddings do change when switching model versions:

MODEL_NAME = '../../../.cache/torch/sentence_transformers/public.ukp.informatik.tu-darmstadt.de_reimers_sentence-transformers_v0.2_distiluse-base-multilingual-cased.zip'
model = SentenceTransformer(MODEL_NAME)
old_model_emb = model.encode("Love is love", convert_to_tensor=True)

MODEL_NAME = 'distiluse-base-multilingual-cased'
model = SentenceTransformer(MODEL_NAME)
new_model_emb = model.encode("Love is love", convert_to_tensor=True)

print(new_model_emb[:5])
print(old_model_emb[:5])

resulting in:

tensor([ 0.0848, -0.0112, -0.0094, -0.0278, -0.0173], device='cuda:0')
tensor([ 0.0863, -0.0242, -0.0092, -0.0280,  0.0026], device='cuda:0')

🤔