Ms_marco document / Bi-encoder evaluate

Hi! Thanks for your work in this repo! I was trying to reproduce bi/cross encoder in ms marco dataset. However, there are some questions that confuse me：

1. I saw that you mentioned in the description that the MRR@10 of the passage retrieval task of the msmarco dataset is around 30-36, but when I use distillroberta-base for training, I can easily reach 37-39, even if the negative of each query is 1000. So where is the problem? Or should I use eval_msmarco.py from the example?
2. Have you experimented with bi/cross encoder in document-level tasks? If so, can you share the results of the experiment? I want to verify that my changes are correct. In addition, do you have a recommended method for document-level tasks?
3. When I tried to train the bi-encoder, I observed that the provided example MRR can easily reach 42+ or even 70+ (passage retrieval), but when I changed to document retrieval, it quickly reached 20+, and then MRR was It oscillates around 20-22, so are these two results correct? If I want to evaluate the performance of bi-encoder separately, what should I do?

All the code I use comes from sentence-transformers/examples/training/ms_marco

[nreimers]

Have a look here https://www.sbert.net/docs/pretrained-models/msmarco-v3.html

Will soon add the v3 training scripts https://www.sbert.net/examples/training/ms_marco/README.html#bi-encoder

[nreimers]

Hi,

1. Yes, you should eval_msmarco.py. During training, scores are computed on a small subset with a smaller corpus (so that the scores can be computed more quickly). For comparison, you must run it on all the 8 million passages.

2. I have not done experiments yet. What works well is to split your documents into paragraphs, either by e.g. identifying paragraphs (like 1 / 2 blank lines) or by splitting your doc into e.g. 100 word chunks. Then encode them individually and do standard passage retrieval. Later you can map it back to the document, i.e., from which doc is this passage => show the doc to the user

3. I have no experiences with doc retrieval. For MRR of 70+: Yes, this is normal as during training a tiny corpus is used for quicker evaluation.