build error with outdated python dependencies

Hello,

I get a build error while testing 2.6 that you may possibly be interested in. It all looked pretty good, until there was this problem right next to your “todo” note. My xopen version is 0.7.3 (you demanded 0.8.4), admittedly, and dnaio at 0.3 (you demanded 0.4 - a likely culprit) . I’ll update that and send an update.

Steffen

I: pybuild base:217: cd /home/moeller/git/med-team/python-cutadapt/.pybuild/cpython3_3.7_cutadapt/build;        py.test-3 --pyargs cutadapt tests
============================= test session starts ==============================
platform linux -- Python 3.7.5, pytest-4.6.6, py-1.8.0, pluggy-0.13.0
rootdir: /home/moeller/git/med-team/python-cutadapt
plugins: xonsh-0.9.11, mock-1.10.4, timeout-1.3.3
collected 388 items

tests/test_adapters.py ..........                                        [  2%]
tests/test_align.py ..........................                           [  9%]
tests/test_commandline.py ....F......ssss............................... [ 21%]
.....................................................................s.. [ 39%]
....................                                                     [ 44%]
tests/test_filters.py ...........                                        [ 47%]
tests/test_main.py .                                                     [ 47%]
tests/test_modifiers.py .....                                            [ 49%]
tests/test_paired.py ................................................... [ 62%]
........................................................................ [ 80%]
....................................                                     [ 90%]
tests/test_parser.py ...........................                         [ 97%]
tests/test_qualtrim.py .                                                 [ 97%]
tests/test_testutils.py ..                                               [ 97%]
tests/test_trim.py ...                                                   [ 98%]
tests/test_utils.py .....                                                [100%]

=================================== FAILURES ===================================
________________________________ test_empty[2] _________________________________

cmdlineargs = ['--cores', '2', '-a', 'TTAGACATATCTCCGTCG', '-o', '/tmp/pytest-of-moeller/pytest-1/test_empty_2_0/empty.fastq', ...]
default_outfile = <_io.FileIO name=7 mode='rb+' closefd=True>

    def main(cmdlineargs=None, default_outfile=sys.stdout.buffer):
        """
        Main function that sets up a processing pipeline and runs it.

        default_outfile is the file to which trimmed reads are sent if the ``-o``
        parameter is not used.
        """
        start_time = time.time()
        parser = get_argument_parser()
        if cmdlineargs is None:
            cmdlineargs = sys.argv[1:]
        args = parser.parse_args(args=cmdlineargs)
        # log to stderr if results are to be sent to stdout
        log_to_stdout = args.output is not None and args.output != "-" and args.paired_output != "-"
        # Setup logging only if there are not already any handlers (can happen when
        # this function is being called externally such as from unit tests)
        if not logging.root.handlers:
            setup_logging(logger, stdout=log_to_stdout,
                quiet=args.quiet, minimal=args.report == 'minimal', debug=args.debug)
        if args.profile:
            import cProfile
            profiler = cProfile.Profile()
            profiler.enable()

        if args.quiet and args.report:
            parser.error("Options --quiet and --report cannot be used at the same time")

        if args.colorspace:
            parser.error(
                "These colorspace-specific options are no longer supported: "
                "--colorspace, -c, -d, --double-encode, -t, --trim-primer, "
                "--strip-f3, --maq, --bwa, --no-zero-cap. "
                "Use Cutadapt 1.18 or earlier to work with colorspace data.")

        paired = determine_paired_mode(args)
        assert paired in (False, True)

        # Print the header now because some of the functions below create logging output
        log_header(cmdlineargs)
        try:
            is_interleaved_input, is_interleaved_output = determine_interleaved(args)
            input_filename, input_paired_filename = input_files_from_parsed_args(args.inputs,
                paired, is_interleaved_input)
            pipeline = pipeline_from_parsed_args(args, paired, is_interleaved_output)
            outfiles = open_output_files(args, default_outfile, is_interleaved_output)
        except CommandLineError as e:
            parser.error(str(e))
            return  # avoid IDE warnings below

        if args.cores < 0:
            parser.error('Value for --cores cannot be negative')
        cores = available_cpu_count() if args.cores == 0 else args.cores
        if cores > 1:
            if ParallelPipelineRunner.can_output_to(outfiles):
                runner_class = ParallelPipelineRunner
                runner_kwargs = dict(n_workers=cores, buffer_size=args.buffer_size)
            else:
                parser.error('Running in parallel is currently not supported for '
                    'the given combination of command-line parameters.\nThese '
                    'options are not supported: --info-file, --rest-file, '
                    '--wildcard-file, --format\n'
                    'Also, demultiplexing is not supported.\n'
                    'Omit --cores/-j to continue.')
                return  # avoid IDE warnings below
        else:
            runner_class = SerialPipelineRunner
            runner_kwargs = dict()
        infiles = InputFiles(input_filename, file2=input_paired_filename,
                interleaved=is_interleaved_input)
        if sys.stderr.isatty() and not args.quiet:
            progress = Progress()
        else:
            progress = DummyProgress()
        try:
            runner = runner_class(pipeline, infiles, outfiles, progress, **runner_kwargs)
        except (dnaio.UnknownFileFormat, IOError) as e:
            parser.error(e)
            return  # avoid IDE warnings below

        logger.info("Processing reads on %d core%s in %s mode ...",
            cores, 's' if cores > 1 else '',
            {False: 'single-end', True: 'paired-end'}[pipeline.paired])
        try:
>           stats = runner.run()

cutadapt/__main__.py:827:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <cutadapt.pipeline.ParallelPipelineRunner object at 0x7f3c446e0cd0>

    def run(self):
        workers, connections = self._start_workers()
        writers = []
        for outfile in self._outfiles:
            if outfile is None:
                continue
            writers.append(OrderedChunkWriter(outfile))
        stats = None
        n = 0  # A running total of the number of processed reads (for progress indicator)
        while connections:
            ready_connections = multiprocessing.connection.wait(connections)
            for connection in ready_connections:
                chunk_index = connection.recv()
                if chunk_index == -1:
                    # the worker is done
                    cur_stats = connection.recv()
                    if stats == -2:
                        # An exception has occurred in the worker (see below,
                        # this happens only when there is an exception sending
                        # the statistics)
                        e, tb_str = connection.recv()
                        # TODO traceback should only be printed in development
                        logger.debug('%s', tb_str)
                        raise e
                    if stats is None:
                        stats = cur_stats
                    else:
                        stats += cur_stats
                    connections.remove(connection)
                    continue
                elif chunk_index == -2:
                    # An exception has occurred in the worker
                    e, tb_str = connection.recv()

                    # TODO traceback should only be printed in development
                    # We should use the worker's actual traceback object
                    # here, but traceback objects are not picklable.
                    logger.debug('%s', tb_str)
>                   raise e
E                   dnaio.exceptions.UnknownFileFormat: Input file format unknown

cutadapt/pipeline.py:702: UnknownFileFormat

During handling of the above exception, another exception occurred:

run = <function run.<locals>._run at 0x7f3c44888cb0>, cores = 2

    def test_empty(run, cores):
        """empty input"""
>       run("--cores {} -a TTAGACATATCTCCGTCG".format(cores), "empty.fastq", "empty.fastq")

tests/test_commandline.py:36:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/conftest.py:19: in _run
    assert main(params) is None
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cmdlineargs = ['--cores', '2', '-a', 'TTAGACATATCTCCGTCG', '-o', '/tmp/pytest-of-moeller/pytest-1/test_empty_2_0/empty.fastq', ...]
default_outfile = <_io.FileIO name=7 mode='rb+' closefd=True>

    def main(cmdlineargs=None, default_outfile=sys.stdout.buffer):
        """
        Main function that sets up a processing pipeline and runs it.

        default_outfile is the file to which trimmed reads are sent if the ``-o``
        parameter is not used.
        """
        start_time = time.time()
        parser = get_argument_parser()
        if cmdlineargs is None:
            cmdlineargs = sys.argv[1:]
        args = parser.parse_args(args=cmdlineargs)
        # log to stderr if results are to be sent to stdout
        log_to_stdout = args.output is not None and args.output != "-" and args.paired_output != "-"
        # Setup logging only if there are not already any handlers (can happen when
        # this function is being called externally such as from unit tests)
        if not logging.root.handlers:
            setup_logging(logger, stdout=log_to_stdout,
                quiet=args.quiet, minimal=args.report == 'minimal', debug=args.debug)
        if args.profile:
            import cProfile
            profiler = cProfile.Profile()
            profiler.enable()

        if args.quiet and args.report:
            parser.error("Options --quiet and --report cannot be used at the same time")

        if args.colorspace:
            parser.error(
                "These colorspace-specific options are no longer supported: "
                "--colorspace, -c, -d, --double-encode, -t, --trim-primer, "
                "--strip-f3, --maq, --bwa, --no-zero-cap. "
                "Use Cutadapt 1.18 or earlier to work with colorspace data.")

        paired = determine_paired_mode(args)
        assert paired in (False, True)

        # Print the header now because some of the functions below create logging output
        log_header(cmdlineargs)
        try:
            is_interleaved_input, is_interleaved_output = determine_interleaved(args)
            input_filename, input_paired_filename = input_files_from_parsed_args(args.inputs,
                paired, is_interleaved_input)
            pipeline = pipeline_from_parsed_args(args, paired, is_interleaved_output)
            outfiles = open_output_files(args, default_outfile, is_interleaved_output)
        except CommandLineError as e:
            parser.error(str(e))
            return  # avoid IDE warnings below

        if args.cores < 0:
            parser.error('Value for --cores cannot be negative')
        cores = available_cpu_count() if args.cores == 0 else args.cores
        if cores > 1:
            if ParallelPipelineRunner.can_output_to(outfiles):
                runner_class = ParallelPipelineRunner
                runner_kwargs = dict(n_workers=cores, buffer_size=args.buffer_size)
            else:
                parser.error('Running in parallel is currently not supported for '
                    'the given combination of command-line parameters.\nThese '
                    'options are not supported: --info-file, --rest-file, '
                    '--wildcard-file, --format\n'
                    'Also, demultiplexing is not supported.\n'
                    'Omit --cores/-j to continue.')
                return  # avoid IDE warnings below
        else:
            runner_class = SerialPipelineRunner
            runner_kwargs = dict()
        infiles = InputFiles(input_filename, file2=input_paired_filename,
                interleaved=is_interleaved_input)
        if sys.stderr.isatty() and not args.quiet:
            progress = Progress()
        else:
            progress = DummyProgress()
        try:
            runner = runner_class(pipeline, infiles, outfiles, progress, **runner_kwargs)
        except (dnaio.UnknownFileFormat, IOError) as e:
            parser.error(e)
            return  # avoid IDE warnings below

        logger.info("Processing reads on %d core%s in %s mode ...",
            cores, 's' if cores > 1 else '',
            {False: 'single-end', True: 'paired-end'}[pipeline.paired])
        try:
            stats = runner.run()
            runner.close()
        except KeyboardInterrupt:
            print("Interrupted", file=sys.stderr)
            sys.exit(130)
        except BrokenPipeError:
            sys.exit(1)
        except (dnaio.FileFormatError, dnaio.UnknownFileFormat, EOFError) as e:
>           sys.exit("cutadapt: error: {}".format(e))
E           SystemExit: cutadapt: error: Input file format unknown

cutadapt/__main__.py:835: SystemExit
=============== 1 failed, 382 passed, 5 skipped in 6.08 seconds ================
E: pybuild pybuild:341: test: plugin custom failed with: exit code=1: cd /home/moeller/git/med-team/python-cutadapt/.pybuild/cpython3_3.7_cutadapt/build;       py.test-3 --pyargs cutadapt tests
dh_auto_test: pybuild --test -i python{version} -p 3.7 -s custom "--test-args=cd {build_dir};   py.test-3 --pyargs cutadapt tests" returned exit code 13

[marcelm]

Great you were able to fix it. Thanks for your packaging work! I think I should add a note to the documentation mentioning that apt install python3-cutadapt as an alternative installation method for Debian-based systems.

[smoe]

It was as you said - the update of dnaio fixed it. I just uploaded. Many thanks!