Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
GLUE results not reproducible
See original GitHub issueHello,
I understand the results mentioned in paper for GLUE are for test set but we are not able to reproduce them. Our pretrained model had a loss of 1.72 and after sweeping through the hyper-parameters mentioned in Table 7 of paper, the best score that we got on CoLA is 43% (in validation set) and as per table 4 your test result is 57.1%. We did finetuning on 1 GPU.
Are we missing something ?
Issue Analytics
- State:
- Created 2 years ago
- Comments:11 (1 by maintainers)
Top Results From Across the Web
Fixing the random seed in the Trainer does not produce the ...
Hi folks, I've noticed that fixing the seed in the Trainer does not produce the same results across multiple training runs.
Read more >Seed for reproducible results is not working (Tensorflow)
I'm having a problem that concern the reproducibility of my results using Tensorflow (v1.15.3). I set all the seeds (os, random, numpy and ......
Read more >How to Solve Reproducibility in ML - neptune.ai
I'm here to tell you that it's completely okay, and it is not your fault! ... Another challenge to reproducibility is different results...
Read more >Furious AI Researcher Creates Site Shaming Non ... - Slashdot
There's no excuse for a ML result to be unreproducible. Really none. If you have the same software versions, the same code to...
Read more >Most computational hydrology is not ... - AGU Journals
Articles that rely on computational work do not provide sufficient information to allow published scientific findings to be reproduced; We argue ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Looks like your model isn’t fully trained. I ditched time based training and trained for 23k steps and achieved val loss ~1.6. Test result on CoLA is 56.5.
all here https://wandb.ai/ctl/budget-bert-pretraining/runs/1pno2mq2/overview?workspace=user-ctl