Tiny difference in MRR@10 for MS MARCO passage using pyserini.search

hey @qguo96 - here’s what I’m getting:

$ python -m pyserini.search --topics msmarco_passage_dev_subset --index ms-marco-passage --output runs/run.msmarco-passage.2.txt --msmarco --bm25
$ python tools/scripts/msmarco/msmarco_eval.py collections/msmarco-passage/qrels.dev.small.tsv runs/run.msmarco-passage.2.txt
#####################
MRR @10: 0.18741227770955543
QueriesRanked: 6980
#####################

Although note that the MRR@10 here (https://github.com/castorini/anserini/blob/master/docs/experiments-msmarco-passage.md) is:

#####################
MRR @10: 0.18741227770955546
QueriesRanked: 6980
#####################

Note the tiny difference in the final digit. Would you mind looking into this? Diff the actual output from both cases and try to see what’s going on? Just wanted to make sure this wasn’t a bug…

[lintool]

(Sorry to interject!) Since the difference is 3e-17, isn’t the problem rather about the unrounded precision of the eval script, much more than anything else? Differences due to order seem expected (if only, in part, with hindsight!) and they’re in fact smaller than I’d have guessed personally.

Agreed, but for replicability purposes we’d ideally want results to be exactly the same, and if they’re not, we should at least understand why.

[okhat]

(Sorry to interject!) Since the difference is 3e-17, isn’t the problem rather about the unrounded precision of the eval script, much more than anything else? Differences due to order seem expected (if only, in part, with hindsight!) and they’re in fact smaller than I’d have guessed personally.