Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Assess performance of parallel io.ascii reader with 'spawn' multiprocessing method

See original GitHub issue

Based on my understanding of the fast C reader after the work I did in #8853, currently when using the multiprocessing ‘spawn’ method, it might be that the parallel reader is not efficient because it looks like it will pickle the whole contents of the table (via the __reduce__ method). This would mean memory mapping isn’t used. So this issue is a reminder that we should check if this is indeed the case. If so, we might want to consider passing the filename to sub-processes rather than the memory mapped bytes string.

Issue Analytics

State:
Created 4 years ago
Comments:18 (18 by maintainers)

Top GitHub Comments

1reaction

astrofrogcommented, Oct 14, 2020

I think removing support for spawn will also allow us to remove some of the code in the cython code, and removing code is always nice 😄

1reaction

astrofrogcommented, Jun 17, 2019

By default yes, but on Linux or Mac one can easily test both multiprocessing start methods (see #8851 for some example code)

Top Results From Across the Web

Fast reader in io.ascii does not work with multiprocessing ...

The multiprocessing module in Python supports several modes - including fork ... Assess performance of parallel io.ascii reader with 'spawn' ...

Python Multiprocessing Pool: The Complete Guide

This guide provides a detailed and comprehensive review of the Multiprocessing Pool in Python, including how it works, how to use it, ...

multiprocessing — Process-based parallelism — Python 3.11 ...

import multiprocessing as mp def foo(q): q.put('hello') if __name__ == '__main__': mp.set_start_method('spawn') q = mp.Queue() p = mp.Process(target=foo ...

using Python's multiprocessing module - Sebastian Raschka

In this introduction to Python's multiprocessing module, we will see how we can spawn multiple subprocesses to avoid some of the GIL's ...

Run Code in Parallel Using the Multiprocessing Module

In this video, we will be learning how to use multiprocessing in Python.This video is sponsored by Brilliant.