Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

unable to reproduce results from notebook

See original GitHub issue

I am unable to reproduce results from TimeGrad Notebook. I am getting diverging loss into NaN loss.

predictor = estimator.train(dataset_train, num_workers=8)

99it [00:22, 4.39it/s, avg_epoch_loss=0.945, epoch=0] 99it [00:22, 4.40it/s, avg_epoch_loss=0.495, epoch=1] 99it [00:22, 4.39it/s, avg_epoch_loss=0.466, epoch=2] 99it [00:22, 4.35it/s, avg_epoch_loss=0.795, epoch=3] 99it [00:22, 4.33it/s, avg_epoch_loss=0.852, epoch=4] 99it [00:22, 4.32it/s, avg_epoch_loss=nan, epoch=5]
99it [00:22, 4.33it/s, avg_epoch_loss=nan, epoch=6] 99it [00:22, 4.30it/s, avg_epoch_loss=nan, epoch=7] 99it [00:23, 4.30it/s, avg_epoch_loss=nan, epoch=8] 99it [00:22, 4.34it/s, avg_epoch_loss=nan, epoch=9] 99it [00:23, 4.29it/s, avg_epoch_loss=nan, epoch=10] 99it [00:23, 4.28it/s, avg_epoch_loss=nan, epoch=11] 99it [00:22, 4.33it/s, avg_epoch_loss=nan, epoch=12] 99it [00:23, 4.21it/s, avg_epoch_loss=nan, epoch=13] 99it [00:23, 4.30it/s, avg_epoch_loss=nan, epoch=14] 99it [00:23, 4.30it/s, avg_epoch_loss=nan, epoch=15] 99it [00:22, 4.34it/s, avg_epoch_loss=nan, epoch=16] 99it [00:22, 4.34it/s, avg_epoch_loss=nan, epoch=17] 99it [00:22, 4.34it/s, avg_epoch_loss=nan, epoch=18] 99it [00:23, 4.20it/s, avg_epoch_loss=nan, epoch=19]

Issue Analytics

State:
Created 3 years ago
Comments:15

Top GitHub Comments

1reaction

kashifcommented, Feb 18, 2021

sorry to hear that… i will try to reproduce on a clean env as well!

1reaction

turmeric-blendcommented, Feb 17, 2021

pytorch 1.7.1+cu110

Top Results From Across the Web

python - Not able to reproduce results with Tensorflow even ...

I can see this using model.get_weights() after creating the model (this is the case even when I restart the notebook and re-run the...

Why can't I reproduce my results in keras using random seed?

I want to make my results reproducible. So I strictly followed this post: ... Reproduce Keras training results in Jupyter Notebook ...

Help! I can't reproduce a machine learning project! - Medium

Not being able to reproduce someone else's results is super frustrating. Not being able to reproduce your own results is frustrating and ...

Why it's hard to reproduce results in software - Bjørn Fjukstad

Reproducing an experiment results in equivalent results ... 70% responded that they have tried and failed to reproduce another scientist's experiment.

Understanding and improving the quality and reproducibility of ...

Having a hidden state may make it impossible to reproduce the same results upon the re-execution of the notebook. In fact, the re-execution ......