Cross-Encoder outputs values greater than 1.0

According to https://sbert.net/examples/applications/retrieve_rerank/README.html#re-ranker-cross-encoder the cross encoder “outputs a single score between 0 and 1”. I do get these results with some underlying models: (transformers==4.6.1, sentence-transformers==1.2.0)

cross_encoder_model = CrossEncoder('cross-encoder/stsb-TinyBERT-L-4')  # 0.95060706

But for:

cross_encoder_model = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-4-v2')
query_anchor_pairs = [
    ['i got the client VERY drunk', 'i got the client drunk'],
]
ce_scores = cross_encoder_model.predict(query_anchor_pairs)
print(ce_scores)  # 8.14593

I get scores much greater than 1. What am I doing wrong? Btw, I am getting very good results with either one, especially the latter (Retrieve & Re-Rank). Thank you, Nils.

[nreimers]

Hi @niebb Thanks for pointing this out. The documentation is there outdated.

Previously, a sigmoid was applied on top of the logits score, i.e. the output was sigmoid(logits). This gives scores between 0 and 1.

The new cross-encoders for msmarco output the logits directly, hence, they can be below 0 or above 1. For re-ranking, this does make any difference. If you like, you can call sigmoid() on top of these values to get back a score 0…1

But as mentioned, for re-ranking it does not make any difference.

[nreimers]

You can check the MSMARCO examples for the cross-encoder.

In principle you need some pairs (query, passage) with labels 0: not relevant and 1: relevant.