Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Error when resume from .pth: BlockingIOError: [Errno 11] Resource temporarily unavailable

See original GitHub issue

I’m trying to resume my training from an just saved .pth, and detectron 2 reported such error:

  1. What exact command you run: python train_net.py --resume
  2. Full logs or other relevant observations:
[01/19 00:36:37 d2.data.build]: Removed 3 images with no usable annotations. 40716 images left.
[01/19 00:36:39 d2.data.dataset_mapper]: [DatasetMapper] Augmentations used in training: [ResizeShortestEdge(short_edge_length=(640, 672, 704, 736, 768, 800), max_size=1333, sample_style='choice')]
[01/19 00:36:39 d2.data.build]: Using training sampler TrainingSampler
[01/19 00:36:41 d2.data.common]: Serializing 40716 elements to byte tensors and concatenating them all ...
[01/19 00:36:42 d2.data.common]: Serialized dataset takes 52.53 MiB
[01/19 00:36:42 fvcore.common.checkpoint]: [Checkpointer] Loading from /data1/jim/detectron2/projects/common_det/ai_mark/model_0009999.pth ...
[01/19 00:36:42 d2.engine.train_loop]: Starting training from iteration 10000
OpenBLAS blas_thread_init: pthread_create failed for thread 63 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
    prepare(preparation_data)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/data1/jim/detectron2/projects/common_det/train_net.py", line 22, in <module>
    import torch
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/site-packages/torch/__init__.py", line 196, in <module>
    from torch._C import *
ImportError: numpy.core.multiarray failed to import
OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 3 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 4 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 5 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 6 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 7 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 8 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 9 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 10 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 11 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 12 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 13 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 14 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 15 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 16 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 17 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 18 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 19 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 20 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 21 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 22 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 23 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 24 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 25 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 26 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 27 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 28 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 29 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 30 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 31 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 32 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 33 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 34 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 35 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 36 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 37 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 38 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 39 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 40 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 41 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 42 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 43 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 44 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 45 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 46 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 47 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 48 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 49 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 50 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 51 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 52 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 53 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 54 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 55 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 56 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 57 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 58 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 59 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 60 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 61 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 62 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
OpenBLAS blas_thread_init: pthread_create failed for thread 63 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
    prepare(preparation_data)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/data1/jim/detectron2/projects/common_det/train_net.py", line 22, in <module>
    import torch
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/site-packages/torch/__init__.py", line 196, in <module>
    from torch._C import *
ImportError: numpy.core.multiarray failed to import
OpenBLAS blas_thread_init: pthread_create failed for thread 63 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1287654 current, 1287654 max
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
    prepare(preparation_data)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/data1/jim/detectron2/projects/common_det/train_net.py", line 22, in <module>
    import torch
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/site-packages/torch/__init__.py", line 196, in <module>
    from torch._C import *
ImportError: numpy.core.multiarray failed to import
Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x7f80ee456320>
Traceback (most recent call last):
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1324, in __del__
    self._shutdown_workers()
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1291, in _shutdown_workers
    if self._persistent_workers or self._workers_status[worker_id]:
AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute '_workers_status'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
_pickle.UnpicklingError: pickle data was truncated
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
_pickle.UnpicklingError: pickle data was truncated
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
_pickle.UnpicklingError: pickle data was truncated
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
_pickle.UnpicklingError: pickle data was truncated
Traceback (most recent call last):
  File "train_net.py", line 258, in <module>
    args=(args,),
  File "/data1/jim/detectron2/detectron2/engine/launch.py", line 79, in launch
    daemon=False,
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 150, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 3 terminated with the following error:
Traceback (most recent call last):
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/data1/jim/detectron2/detectron2/engine/launch.py", line 126, in _distributed_worker
    main_func(*args)
  File "/data1/jim/detectron2/projects/common_det/train_net.py", line 247, in main
    return trainer.train()
  File "/data1/jim/detectron2/detectron2/engine/defaults.py", line 509, in train
    super().train(self.start_iter, self.max_iter)
  File "/data1/jim/detectron2/detectron2/engine/train_loop.py", line 149, in train
    self.run_step()
  File "/data1/jim/detectron2/detectron2/engine/defaults.py", line 519, in run_step
    self._trainer.run_step()
  File "/data1/jim/detectron2/detectron2/engine/train_loop.py", line 267, in run_step
    data = next(self._data_loader_iter)
  File "/data1/jim/detectron2/detectron2/data/common.py", line 234, in __iter__
    for d in self.dataset:
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 355, in __iter__
    return self._get_iterator()
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 301, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 914, in __init__
    w.start()
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 59, in _launch
    cmd, self._fds)
  File "/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/util.py", line 455, in spawnv_passfds
    False, False, None)
BlockingIOError: [Errno 11] Resource temporarily unavailable

(detectron2) jim@VM-16-9-ubuntu:/data1/jim/detectron2/projects/common_det$ /data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 53 leaked semaphores to clean up at shutdown
  len(cache))
/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 53 leaked semaphores to clean up at shutdown
  len(cache))
/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 53 leaked semaphores to clean up at shutdown
  len(cache))
/data1/jim/miniconda3/envs/detectron2/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 62 leaked semaphores to clean up at shutdown
  len(cache))

Environment:

cuda 10.1 torch 1.8.0+cu101 torchvision 0.9.0+cu101 detectron2 0.6 numpy 1.21.5 Provide your environment information using the following command:

wget -nc -q https://github.com/facebookresearch/detectron2/raw/main/detectron2/utils/collect_env.py && python collect_env.py

If your issue looks like an installation issue / environment issue, please first try to solve it yourself with the instructions in https://detectron2.readthedocs.io/tutorials/install.html#common-installation-issues

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:6 (2 by maintainers)

github_iconTop GitHub Comments

2reactions
fingertapcommented, Jan 30, 2022

So this is caused by the uncleaned jobs? I have a newly created k8s pod, and I encountered this error after training for several epochs.

1reaction
Johnqczhangcommented, May 8, 2022

[BlockingIOError: [Errno 11] Resource temporarily unavailable] during fork is a system limitation. See https://access.redhat.com/solutions/22105 for some common cause and solutions…

Thanks for your hint! The newly error log shown below provides useful info. The issue has been solved by adding OMP_NUM_THREADS=1 before the running command. Thanks again!

OpenBLAS blas_thread_init: pthread_create failed for thread 55 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1544645 max
OpenBLAS blas_thread_init: pthread_create failed for thread 56 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1544645 max
OpenBLAS blas_thread_init: pthread_create failed for thread 57 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1544645 max
OpenBLAS blas_thread_init: pthread_create failed for thread 58 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1544645 max
Read more comments on GitHub >

github_iconTop Results From Across the Web

BlockingIOError: [Errno 11] Resource temporarily unavailable
Ditto! I have a similar error using selenium: Traceback: File "/usr/local/lib/python2.7/dist-packages/django ...
Read more >
BlockingIOError: [Errno 11] Resource temporarily unavailable ...
After the 115th iteration, sometimes more, an error appears: pid = os.fork() BlockingIOError: [Errno 11] Resource temporarily unavailable.
Read more >
Issue 43743: BlockingIOError: [Errno 11] Resource temporarily ...
BlockingIOError : [Errno 11] Resource temporarily unavailable: on GPFS. Type: behavior, Stage: resolved. Components: IO, Versions: Python 3.10, ...
Read more >
Changelog — documentation Python 3.7.0a0
bpo-30061: Fixed crashes in IOBase methods __next__() and readlines() when readline() or __next__() respectively return non-sizeable object.
Read more >
HA is randomly restarting - #22 by flobidan - Configuration
I did notice some errors in the log related to the Stream component ... frame): BlockingIOError: [Errno 11] Resource temporarily unavailable ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Building from Dockerfile errors on pip install

Next page

How to get FPS for a model?