[Bug] events.out.tfevents is not generated when using --restore_path flag in python TTS/TTS/bin/train_tacotron.py
See original GitHub issueDescribe the bug
Each training run is supposed to generate events.out.tfevents
for tensorboard but when I am using --restore_path
it doesn’t generate events.out.tfeventsxxxx
file
To Reproduce Steps to reproduce the behavior:
- Run the following command
CUDA_VISIBLE_DEVICES="0" python TTS/TTS/bin/train_tacotron.py --restore_path ./Results/ljspeech-ddc-June-09-2021_08+36PM-0000000/best_model.pth.tar \
--config_path ./tacotron2-DDC.json \
--coqpit.output_path ./Results \
--coqpit.datasets.0.path ./Italian_dataset/it_IT/by_book/male/riccardo_fasol/il_ritratto_del_diavolo \
--coqpit.audio.stats_path ./scale_stats.npy \
- No Runtime or compile Error
Expected behavior
It should generate events.out.tfeventsxxxx
file in the training RUN directory by default.
Environment (please complete the following information):
-
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
-
Ubuntu 18.04.5 LTS
-
PyTorch or TensorFlow version (use command below):
-
PyTorch 1.4.0, TensorFlow 2.1.0
-
Exact command to reproduce:
CUDA_VISIBLE_DEVICES="0" python TTS/TTS/bin/train_tacotron.py --restore_path ./Results/ljspeech-ddc-June-09-2021_08+36PM-0000000/best_model.pth.tar \
--config_path ./tacotron2-DDC.json \
--coqpit.output_path ./Results \
--coqpit.datasets.0.path ./Italian_dataset/it_IT/by_book/male/riccardo_fasol/il_ritratto_del_diavolo \
--coqpit.audio.stats_path ./scale_stats.npy \
Issue Analytics
- State:
- Created 2 years ago
- Comments:10 (5 by maintainers)
Top Results From Across the Web
[bug] events.out.tfevents files do not get closed. #20606 - GitHub
Running a python script that trains and tests MANY models (tf.Estimator) failes with tf.estimator Error: ResourceExhausted: too many open ...
Read more >How to turn off events.out.tfevents file in tf.contrib.learn Estimator
My understanding is that this file stores the events generated by tensorflow. I went ahead and deleted this manually. Interestingly, it never got...
Read more >Visualize Model Training with TensorBoard
Complete the tutorial from Train Intent-Slot model on ATIS Dataset if you have not done so. Once that is done, you should be...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Ah ok, that explained what I saw then! Actually the events file was created when the training stopped while I was writing the comment! Thank you for your answer!
Edit
save_step
inconfig.json
file to your desired step size for saving model. @TejaswiniiB