Problem with multiprocessing context

I am currently experiencing the following issue when running sockeye-train:

[INFO:sockeye.training] Training started.
[INFO:sockeye.utils] Releasing GPU 0.
[ERROR:root] Uncaught exception
Traceback (most recent call last):
  File "/usr/local/bin/sockeye-train", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/sockeye/train.py", line 825, in main
    train(args)
  File "/usr/local/lib/python3.6/dist-packages/sockeye/train.py", line 955, in train
    existing_parameters=args.params)
  File "/usr/local/lib/python3.6/dist-packages/sockeye/training.py", line 627, in fit
    process_manager = DecoderProcessManager(self.model.output_dir, decoder=decoder)
  File "/usr/local/lib/python3.6/dist-packages/sockeye/training.py", line 1282, in __init__
    self.ctx = mp_utils.get_context()  # type: ignore
  File "/usr/local/lib/python3.6/dist-packages/sockeye/multiprocessing_utils.py", line 71, in get_context
    assert __context is not None, ("Multiprocessing context not initialized. Please call "
AssertionError: Multiprocessing context not initialized. Please call sockeye.multiprocessing_utils.initialize() right after interpreter startup.

Any idea what could be causing this?

I am running sockeye using mxnet-cu100mkl inside a singularity container.

[jamesewoo]

Could it be that when running the generated cli script sockeye-train the train.py is not considered __main__?

I think this is the case. If you look at setup.py it points directly to sockeye.train:main.

Since the multiprocessing init method checks if it’s already been called, could you simply remove the if condition?

[fhieber]

#704 was merged.