Demultiplexer may cause `OSError: [Errno 24] Too many open files` on large sample runs

On large sample count runs (or possibly smaller if the system itself is using a large amount of files at the time of analysis), any pipeline using the Demultiplixer class may run into an operating system error due to having too many open file descriptors.

This was brought up by a user on the QIIME 2 forum and appears to be identical to the issue that the initial qiime2/q2-demux implementation was facing that was fixed here.

The way it was solved on QIIME 2’s demultiplexer was by randomly closing X% of the sample files to keep it below the system limit. I would gladly lend a hand in adding a patch and opening a pull request into cutadapt if you would like.

This issue pertains to any version of cutadapt that uses these lines.

[peterjc]

Wow - the lazy import itself was failing (because the file handle limit had been reached). Nice job debugging this 👍

[fasterius]

Okay, cool, thanks for fixing it!