Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

MS MARCO passage regression errors: BM25prf gives non-deterministic results

See original GitHub issue

Hi @emmileaf I’m getting these MS MARCO passage regression errors:

This is on tuna:

2019-08-11 03:55:02,107 - regression_test - ERROR - !!!!!{"actual": 0.1518, "collection": "msmarco-passage", "expected": 0.152, "metric": "map", "model": "bm25-default+prf", "topic": "[MS MARCO Passage Ranking: Dev Queries](https://github.com/microsoft/MSMARCO-Passage-Ranking)"}!!!!!
...
2019-08-11 03:56:41,141 - regression_test - ERROR - !!!!!{"actual": 0.1579, "collection": "msmarco-passage", "expected": 0.1582, "metric": "map", "model": "bm25-tuned+prf", "topic": "[MS MARCO Passage Ranking: Dev Queries](https://github.com/microsoft/MSMARCO-Passage-Ranking)"}!!!!!

This is on another machine:

2019-08-11 03:38:45,575 - regression_test - ERROR - !!!!!{"actual": 0.1519, "collection": "msmarco-passage", "expected": 0.152, "metric": "map", "model": "bm25-default+prf", "topic": "[MS MARCO Passage Ranking: Dev Queries](https://github.com/microsoft/MSMARCO-Passage-Ranking)"}!!!!!
...
2019-08-11 03:39:51,630 - regression_test - ERROR - !!!!!{"actual": 0.158, "collection": "msmarco-passage", "expected": 0.1582, "metric": "map", "model": "bm25-tuned+prf", "topic": "[MS MARCO Passage Ranking: Dev Queries](https://github.com/microsoft/MSMARCO-Passage-Ranking)"}!!!!!

It seems like BM25prf gives non-deterministic results?

@matthew-z any ideas?

Issue Analytics

State:
Created 4 years ago
Comments:6 (6 by maintainers)

Top GitHub Comments

1reaction

emmileafcommented, Aug 11, 2019

From a quick look at the runs, this might be a score tie handling issue?

https://github.com/castorini/anserini/blob/5b29d1654abc5e8a014c2230da990ab2f91fb340/src/main/java/io/anserini/rerank/lib/BM25PrfReranker.java#L113

https://github.com/castorini/anserini/blob/5b29d1654abc5e8a014c2230da990ab2f91fb340/src/main/java/io/anserini/rerank/lib/Rm3Reranker.java#L111-L118

I’ll add in tiebreak handling similar to the other rerankers, and update regression numbers for PRF.

0reactions

lintoolcommented, Sep 5, 2019

Two trials on tuna (Java 8) give the same result. Two trials on my iMac Pro (Java 8) gives the same result.

I think it’s just the case that we forgot to update the regression values.

See PR #788

Top Results From Across the Web

anserini/regressions-msmarco-passage-doc2query.md at ...

This page documents regression experiments on the MS MARCO passage ranking task with BM25 on (vanilla) doc2query (also called doc2query-base) expansions, ...

MS MARCO - Microsoft Open Source

date type MRR@100 (Dev) MRR@100 (Eval) 2022/02/08 🏆 full ranking 0.512 0.446 2021/07/14 🏆 full ranking 0.500 0.440 2021/06/24 🏆 full ranking 0.496 0.436

MS MARCO: Benchmarking Ranking Modelsin the Large-Data ...

We first describe an older improvement, where signif- icantly better rankers such as BM25 were developed using TREC data in the 1990s. We...

Document Expansions and Learned Sparse Lexical ...

The MS MARCO passage ranking corpus comprises 8.8m passages drawn from texts shown as “answers” at the top of Bing's search results page....

MS MARCO — Sentence-Transformers documentation

MS MARCO Passage Ranking is a large dataset to train models for ... By default, we set a threshold of 3: If the...