Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Parameters for reproducing results from paper

See original GitHub issue

Can you provide the parameters for reproducing the results from the paper on FB15k and FB15K-237? I ran the command from the README:

 CUDA_VISIBLE_DEVICES=0 python main.py --dataset FB15k-237 --num_iterations 500 --batch_size 128
                                       --lr 0.0005 --dr 1.0 --edim 200 --rdim 200 --input_dropout 0.3 
                                       --hidden_dropout1 0.4 --hidden_dropout2 0.5 --label_smoothing 0.1

which gave final performance of

Number of data points: 35070
Hits @10: 0.4009124607927003
Hits @3: 0.2555460507556316
Hits @1: 0.1760193897918449
Mean rank: 291.46401482748786
Mean reciprocal rank: 0.24741750020439274
Test:
Number of data points: 40932
Hits @10: 0.3974396560148539
Hits @3: 0.2546662757744552
Hits @1: 0.17094205022964917
Mean rank: 304.61949086289457
Mean reciprocal rank: 0.24344486414937788

Any ideas?

UPDATE: I noticed in the paper that you mention the best learning rate for FB15k-237 is 0.005 instead of 0.0005 and best the learning rate decay is 0.995 instead of 1.0 – might that be the issue?

Issue Analytics

State:
Created 5 years ago
Comments:7 (2 by maintainers)

Top GitHub Comments

1reaction

bkjcommented, Feb 15, 2019

I figured out what was going on – I had to make some small changes to get torch==1.0.0 to stop complaining, and I made a small mistake. Thanks for your help!

0reactions

l6270789commented, May 14, 2019

I used torch==1.0.0 and cannot reproduce the results from the paper. What changes do you make from torch==0.4.0 to torch==1.0.0? Thanks for your help!

Top Results From Across the Web

How to Make Sure Your Research Paper is Reproducible ...

Here, we share commons reasons papers fail, based on reproducibility checks our team has done for 55 papers.

Learn To Reproduce Papers: Beginner's Guide

Step-by-step instructions on how to understand Deep Learning papers and implement the described approaches. With an example: today we are ...

Understanding Reproducibility and Replicability - NCBI - NIH

Reproducibility depends only on whether the methods of the computational analysis were transparently and accurately reported and whether that data, code, or ...

Reproducibility - Wikipedia

This article is about the reproducibility of scientific research results. For reproductive capacity of organisms, see fertility and fecundity.

How to get the data to reproduce a published result?

Consider offering co-authorship on any papers that will result from your analysis, as incentive. Give them enough context about you & your own ......