GPU and multiprocessing leads to ' initialization error'

spacy.prefer_gpu()
nlp = spacy.load('en_core_web_sm', disable=["tagger", "parser"]) 
docs = nlp.pipe(texts, batch_size=128, n_process=2, disable=["tagger", "parser"])

NOTE : if I am using n_process=1 the code runs fine. If I am not using the gpu the code also runs fine. It is the combination of GPU and multiprocessing that leads to this error.

process Process-2:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/ubuntu/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/spacy/language.py", line 1134, in _apply_pipes
    sender.send([doc.to_bytes() for doc in docs])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/spacy/language.py", line 1134, in <listcomp>
    sender.send([doc.to_bytes() for doc in docs])
  File "nn_parser.pyx", line 253, in pipe
  File "nn_parser.pyx", line 273, in spacy.syntax.nn_parser.Parser.predict
  File "nn_parser.pyx", line 286, in spacy.syntax.nn_parser.Parser.greedy_parse
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/thinc/neural/_classes/model.py", line 167, in __call__
    return self.predict(x)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/thinc/neural/_classes/model.py", line 131, in predict
    y, _ = self.begin_update(X, drop=None)
  File "_parser_model.pyx", line 243, in spacy.syntax._parser_model.ParserModel.begin_update
  File "_parser_model.pyx", line 293, in spacy.syntax._parser_model.ParserStepModel.__init__
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/thinc/neural/_classes/feed_forward.py", line 46, in begin_update
    X, inc_layer_grad = layer.begin_update(X, drop=drop)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/spacy/_ml.py", line 791, in flatten
    lengths = ops.asarray([len(seq) for seq in seqs], dtype="i")
  File "ops.pyx", line 1001, in thinc.neural.ops.CupyOps.asarray
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/cupy/creation/from_data.py", line 43, in array
    return core.array(obj, dtype, copy, order, subok, ndmin)
  File "cupy/core/core.pyx", line 1954, in cupy.core.core.array
  File "cupy/core/core.pyx", line 2031, in cupy.core.core.array
  File "cupy/core/core.pyx", line 2106, in cupy.core.core._send_object_to_gpu
  File "cupy/core/core.pyx", line 136, in cupy.core.core.ndarray.__init__
  File "cupy/cuda/memory.pyx", line 569, in cupy.cuda.memory.alloc
  File "cupy/cuda/memory.pyx", line 1241, in cupy.cuda.memory.MemoryPool.malloc
  File "cupy/cuda/memory.pyx", line 1261, in cupy.cuda.memory.MemoryPool.malloc
  File "cupy/cuda/device.pyx", line 24, in cupy.cuda.device.get_device_id
  File "cupy/cuda/runtime.pyx", line 250, in cupy.cuda.runtime.getDevice
  File "cupy/cuda/runtime.pyx", line 222, in cupy.cuda.runtime.check_status
cupy.cuda.runtime.CUDARuntimeError: cudaErrorInitializationError: initialization error
Process Process-1:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/ubuntu/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/spacy/language.py", line 1134, in _apply_pipes
    sender.send([doc.to_bytes() for doc in docs])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/spacy/language.py", line 1134, in <listcomp>
    sender.send([doc.to_bytes() for doc in docs])
  File "nn_parser.pyx", line 253, in pipe
  File "nn_parser.pyx", line 273, in spacy.syntax.nn_parser.Parser.predict
  File "nn_parser.pyx", line 286, in spacy.syntax.nn_parser.Parser.greedy_parse
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/thinc/neural/_classes/model.py", line 167, in __call__
    return self.predict(x)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/thinc/neural/_classes/model.py", line 131, in predict
    y, _ = self.begin_update(X, drop=None)
  File "_parser_model.pyx", line 243, in spacy.syntax._parser_model.ParserModel.begin_update
  File "_parser_model.pyx", line 293, in spacy.syntax._parser_model.ParserStepModel.__init__
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/thinc/neural/_classes/feed_forward.py", line 46, in begin_update
    X, inc_layer_grad = layer.begin_update(X, drop=drop)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/spacy/_ml.py", line 791, in flatten
    lengths = ops.asarray([len(seq) for seq in seqs], dtype="i")
  File "ops.pyx", line 1001, in thinc.neural.ops.CupyOps.asarray
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/cupy/creation/from_data.py", line 43, in array
    return core.array(obj, dtype, copy, order, subok, ndmin)
  File "cupy/core/core.pyx", line 1954, in cupy.core.core.array
  File "cupy/core/core.pyx", line 2031, in cupy.core.core.array
  File "cupy/core/core.pyx", line 2106, in cupy.core.core._send_object_to_gpu
  File "cupy/core/core.pyx", line 136, in cupy.core.core.ndarray.__init__
  File "cupy/cuda/memory.pyx", line 569, in cupy.cuda.memory.alloc
  File "cupy/cuda/memory.pyx", line 1241, in cupy.cuda.memory.MemoryPool.malloc
  File "cupy/cuda/memory.pyx", line 1261, in cupy.cuda.memory.MemoryPool.malloc
  File "cupy/cuda/device.pyx", line 24, in cupy.cuda.device.get_device_id
  File "cupy/cuda/runtime.pyx", line 250, in cupy.cuda.runtime.getDevice
  File "cupy/cuda/runtime.pyx", line 222, in cupy.cuda.runtime.check_status
cupy.cuda.runtime.CUDARuntimeError: cudaErrorInitializationError: initialization error

[floschne]

I was able to resolve it by adding multiprocessing.set_start_method('spawn') in my main method as the first statement. This causes the process to spawn instead of forking apart from the process that initialized CUDA. The processes then all use CUDA quite efficiently

gpustat output: [7] GeForce RTX 2080 Ti | 38'C, 25 % | 9850 / 11019 MB | 7schneid(1037M) 7schneid(1519M) 7schneid(1541M) 7schneid(1491M) 7schneid(1405M) 7schneid(1355M) 7schneid(1485M)

EDIT: The processes eat up GPU Memory… For larger datasets, this is infeasible with limited GPU Memory…

[honnibal]

Thanks for the report. We probably should have foreseen this when we did the multiprocessing for nlp.pipe.

I think it’s very reasonable to want this, but it’s also pretty hard to solve. Processes cannot be forked after CUDA initialization: https://stackoverflow.com/questions/54808148/cupy-get-error-in-multithread-pool-if-gpu-already-used

I think for now we should raise a better error explaining that multiprocessing is not supported with GPU currently, and linking to a workaround if possible. Definitely open to suggestions for how to actually support it.

I’ll label this “bug” for now, we can consider it “fixed” once we at least have a useful error, at which point we’d convert this to a feature request for the actual support.