What did you change for 'xlm-r-distilroberta-base-paraphrase-v1' training?

Hi, what did you change for xlm-r-distilroberta-base-paraphrase-v1 training compared to xlm-r-bert-base-nli-stsb-mean-tokens? My tests show that it performs better on the german translated stsb:

model = SentenceTransformer('xlm-r-bert-base-nli-stsb-mean-tokens')  #Spearman: 0.8181
model = SentenceTransformer('xlm-r-distilroberta-base-paraphrase-v1')  # Spearman: 0.8201

[nreimers]

Hi @PhilipMay A paper is upcoming for the paraphrase models.

These models were trained on various datasets with Millions of examples for paraphrases, mainly derived from Wikipedia edit logs, paraphrases mined from Wikipedia and SimpleWiki, paraphrases from news reports, AllNLI-entailment pairs with in-batch-negative loss etc.

In internal tests, they perform much better than the NLI+STSb models as they have see more and broader type of training data. NLI+STSb has the issue that they are rather narrow in their domain and do not contain any domain specific words / sentences (like from chemistry, computer science, math etc.). The paraphrase models has seen plenty of sentences from various domains.

More details with the setup, all the datasets, and a wider evaluation will follow soon.

[nreimers]

@PhilipMay Sadly not yet, it is rather far down in the paper pipeline 😕