Not able to replicate results on FlyingThings++ noted in the paper

Hi,

Great work and thank you for releasing your code and documenting it.

I was trying to replicate numbers on FlyingThings++ shown in this table.

I had setup the conda environment and FlyingThings++ exactly as told in the README. I am using the model checkpoint provided in get_reference_model.sh.

I run the command python test_on_flt.py --modeltype pips and python test_on_flt.py --modeltype raft to evaluate on FlyingThings++.

Here are the results that I get.

PIPS:

1_8_16_pips_flt_17:07:18; step 002542/2542; rtime 0.00; itime 0.09, ate_vis = 10.42, ate_occ = 27.04

RAFT:

1_8_16_raft_flt_18:46:56; step 002542/2542; rtime 0.00; itime 1.03, ate_vis = 15.04, ate_occ = 39 

Do you have any idea why they might be different from the results provided in the paper?

[aharley]

Maybe more time actually. The upload time is reading 55h right now, and that’s assuming no interruptions. Anyway, at the end of this we should have a very good gdrive link. I’ll let you know @ayshrv!

[ayshrv]

Thank you for your response. Yes, that will be great if you could provide the data that you generated. I agree that there could be some randomness in the process.