question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Bug] events.out.tfevents is not generated when using --restore_path flag in python TTS/TTS/bin/train_tacotron.py

See original GitHub issue

Describe the bug Each training run is supposed to generate events.out.tfevents for tensorboard but when I am using --restore_path it doesn’t generate events.out.tfeventsxxxx file

To Reproduce Steps to reproduce the behavior:

  1. Run the following command
 CUDA_VISIBLE_DEVICES="0" python TTS/TTS/bin/train_tacotron.py --restore_path ./Results/ljspeech-ddc-June-09-2021_08+36PM-0000000/best_model.pth.tar \
                                                         --config_path ./tacotron2-DDC.json \
                                                          --coqpit.output_path ./Results  \
                                                          --coqpit.datasets.0.path ./Italian_dataset/it_IT/by_book/male/riccardo_fasol/il_ritratto_del_diavolo   \
                                                          --coqpit.audio.stats_path ./scale_stats.npy \
  1. No Runtime or compile Error

Expected behavior It should generate events.out.tfeventsxxxx file in the training RUN directory by default.

Environment (please complete the following information):

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

  • Ubuntu 18.04.5 LTS

  • PyTorch or TensorFlow version (use command below):

  • PyTorch 1.4.0, TensorFlow 2.1.0

  • Exact command to reproduce:

 CUDA_VISIBLE_DEVICES="0" python TTS/TTS/bin/train_tacotron.py --restore_path ./Results/ljspeech-ddc-June-09-2021_08+36PM-0000000/best_model.pth.tar \
                                                         --config_path ./tacotron2-DDC.json \
                                                          --coqpit.output_path ./Results  \
                                                          --coqpit.datasets.0.path ./Italian_dataset/it_IT/by_book/male/riccardo_fasol/il_ritratto_del_diavolo   \
                                                          --coqpit.audio.stats_path ./scale_stats.npy \

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:10 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
Ca-ressemble-a-du-fakecommented, Feb 15, 2022

Ah ok, that explained what I saw then! Actually the events file was created when the training stopped while I was writing the comment! Thank you for your answer!

0reactions
Sadam1195commented, May 26, 2022

Hi @Ca-ressemble-a-du-fake ,

  1. For me , .pth checkpoints are not being generated when --restore-path arg is used. Do you know why? I’m also using google colab. Rest of the files are being updated though as shown in below pic.
  2. And how to make use of events.out.tfevents file?

image

checkpoint got written now at 280000 step. How can we make to write checkpoints more frequently?

Edit save_step in config.json file to your desired step size for saving model. @TejaswiniiB

Read more comments on GitHub >

github_iconTop Results From Across the Web

[bug] events.out.tfevents files do not get closed. #20606 - GitHub
Running a python script that trains and tests MANY models (tf.Estimator) failes with tf.estimator Error: ResourceExhausted: too many open ...
Read more >
How to turn off events.out.tfevents file in tf.contrib.learn Estimator
My understanding is that this file stores the events generated by tensorflow. I went ahead and deleted this manually. Interestingly, it never got...
Read more >
Visualize Model Training with TensorBoard
Complete the tutorial from Train Intent-Slot model on ATIS Dataset if you have not done so. Once that is done, you should be...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found