question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Bars of tqdm seems fixed at 0/50000 and fail to continue the Q-labeling

See original GitHub issue

Make sure you have read FAQ before posting. Thanks! Hello, After running all the above programs correctly as you suggested,I have trained the ego-model and successfully collected the nocrash data about 186GB, then what I need to do is label Q. So I run with $python -m rails.data_phase2 --num-workers=4, it shows as follows:

|         | 0/53267 [00:00<?, ?it/s

And I have checked my GPU, and it shows the ray and running now.

+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 2080    Off  | 00000000:86:00.0 Off |                  N/A |
| 50%   71C    P2   153W / 215W |   2480MiB /  7952MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce RTX 2080    Off  | 00000000:AF:00.0 Off |                  N/A |
| 51%   72C    P2   152W / 215W |   2480MiB /  7952MiB |     98%      Default |

|    2     14845      C   ray::RAILSActionLabeler.run()               1229MiB |
|    2     14855      C   ray::RAILSActionLabeler.run()               1229MiB |
|    3     14898      C   ray::RAILSActionLabeler.run()               1229MiB |
|    3     14919      C   ray::RAILSActionLabeler.run()               1229MiB 

When I tried CTRL+C, it shows:

raceback (most recent call last):
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/shy/Desktop/WorldOnRails/rails/data_phase2.py", line 58, in <module>
    main(args)
  File "/home/shy/Desktop/WorldOnRails/rails/data_phase2.py", line 24, in main
    current_frames = ray.get(logger.total_frames.remote())
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/worker.py", line 1372, in get
    object_refs, timeout=timeout)
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/worker.py", line 304, in get_objects
    object_refs, self.current_task_id, timeout_ms)
  File "python/ray/_raylet.pyx", line 869, in ray._raylet.CoreWorker.get_objects
  File "python/ray/_raylet.pyx", line 142, in ray._raylet.check_status
KeyboardInterrupt
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/node.py", line 868, in _kill_process_type
    process.wait(timeout_seconds)
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/subprocess.py", line 1019, in wait
    return self._wait(timeout=timeout)
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/subprocess.py", line 1647, in _wait
    time.sleep(delay)
KeyboardInterrupt
  0%|                              

It seems it just waiting now? (But we donot need to launch carla in this phase) And the data-dir is set to the collected data direction, the config.yaml is set to the no-crash config. ( default=‘/home/shy/Desktop/WorldOnRails/experiments/config_nocrash.yaml’)), just cp config.yaml to experiments file. Thanks a lot!

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:10 (5 by maintainers)

github_iconTop GitHub Comments

2reactions
varunjammulacommented, Jul 19, 2021

Hi, I think I figured out the issue. The above error occurs if the cluster resources are not available. My admin settings probably block me from auto-balancing when using sbatch scripts. I think the issue can be closed.

2reactions
SunHaoOnecommented, Jun 7, 2021

Just wait for longer. or if you have to eyeball the bar moving, set --num-per-log=1

Thanks for your reply. I have tried to add this args, but still nothing happened, which shows the same message as before. And last time when I did CTRL+C it shows: | 0/53267 [1:20:03<?, ?it/s]. Maybe it is too long for one loop? (And I have checked the datafile, it seems no file is modified today) And I will wait longer, thanks! Later I found that maybe some data is lost and I deleted the latest dataset file and finnally it works and it shows 81/52736 [01:41<16:43:51, 1.14s/it

Read more comments on GitHub >

github_iconTop Results From Across the Web

python - Why is tqdm printing to a newline instead of updating ...
Once interrupted, the user can restart by entering 'restart' into the command line prompt. The second time the progress bar is shown, instead...
Read more >
How to Use Progress Bars in Python? | tqdm and tqdm Notebook
tqdm is a library in Python which is used for creating Progress Meters or Progress Bars. In this article learn how to create...
Read more >
Progress bar and status logging in python with tqdm
Fortunately tqdm can be hacked to display text in fixed positions. C) Convert progress bars to status bars. The idea behind this “hack”...
Read more >
tqdm - PyPI
tqdm derives from the Arabic word taqaddum (تقدّم) which can mean “progress,” ... This means a fixed bar with right-justified text may be...
Read more >
A Complete Guide to Using Progress Bars in Python
Indication if the program is stuck in some error and must be stopped. TQDM has the lowest overhead of all the libraries discussed...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found