pyximport compilation failure tracebacks are excessively long.

Note: This is a very minor, quality-of-life annoyance.

I’m working on a relatively complex project that uses several cython modules, with additional pyxbld mediated c++ source files.

Right now, when I’m working on the C++ sources, cython compilation failures yield massively long tracebacks. The common failure output is 100+ lines.

This wouldn’t be too much of a problem, if cython did not currently output the C++ compilation failure, and then the multiple python tracebacks it caused. As it is, basically all the information I need to actually fix the issue is inevitably scrolled off the screen.

Here is an example failure:

Error compiling Cython file:
------------------------------------------------------------
...

                # Queue structure:
                # [USB Thread] -> [Parse Thread] -> [Log Thread]

                self.usb_thread.insert_output_queue(self.parse_thread.get_queue_reference())
                self.parse_thread.insert_output_queue(self.log_thread.get_queue_reference())
                  ^
------------------------------------------------------------

Hardware/interface_wrapper.pyx:138:19: Object of type 'BatParserThread' has no attribute 'insert_output_queue'

Error compiling Cython file:
------------------------------------------------------------
...

                # Queue structure:
                # [USB Thread] -> [Parse Thread] -> [Log Thread]

                self.usb_thread.insert_output_queue(self.parse_thread.get_queue_reference())
                self.parse_thread.insert_output_queue(self.log_thread.get_queue_reference())
                                                                          ^
------------------------------------------------------------

Hardware/interface_wrapper.pyx:138:75: Cannot convert 'ReaderWriterQueue[UsbMessageRecord] &' to Python object
building 'Hardware.interface_wrapper' extension
arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/ -I/usr/lib/openmpi/include/ -I/usr/include/hdf5/serial -I/usr/include/python3.4m -c /home/pi/.pyxbld/temp.linux-armv7l-3.4/pyrex/Hardware/interface_wrapper.cpp -o /home/pi/.pyxbld/temp.linux-armv7l-3.4/home/pi/.pyxbld/temp.linux-armv7l-3.4/pyrex/Hardware/interface_wrapper.o -std=c++11 -O3 -Wall -Werror -lrt -lusb-1.0 -DLINUX
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/home/pi/.pyxbld/temp.linux-armv7l-3.4/pyrex/Hardware/interface_wrapper.cpp:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation.
 #error Do not use this file, it is the result of a failed Cython compilation.
  ^
Traceback (most recent call last):
  File "/usr/lib/python3.4/distutils/unixccompiler.py", line 116, in _compile
    extra_postargs)
  File "/usr/lib/python3.4/distutils/ccompiler.py", line 909, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/usr/lib/python3.4/distutils/spawn.py", line 36, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/usr/lib/python3.4/distutils/spawn.py", line 162, in _spawn_posix
    % (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/pyximport/pyximport.py", line 216, in load_module
    inplace=build_inplace, language_level=language_level)
  File "/usr/local/lib/python3.4/dist-packages/pyximport/pyximport.py", line 192, in build_module
    reload_support=pyxargs.reload_support)
  File "/usr/local/lib/python3.4/dist-packages/pyximport/pyxbuild.py", line 102, in pyx_to_dll
    dist.run_commands()
  File "/usr/lib/python3.4/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.4/dist-packages/Cython/Distutils/old_build_ext.py", line 185, in run
    _build_ext.build_ext.run(self)
  File "/usr/lib/python3.4/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/usr/local/lib/python3.4/dist-packages/Cython/Distutils/old_build_ext.py", line 193, in build_extensions
    self.build_extension(ext)
  File "/usr/lib/python3.4/distutils/command/build_ext.py", line 503, in build_extension
    depends=ext.depends)
  File "/usr/lib/python3.4/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/usr/lib/python3.4/distutils/unixccompiler.py", line 118, in _compile
    raise CompileError(msg)
distutils.errors.CompileError: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "acquire_only.py", line 23, in <module>
    import Hardware.acq_hardware
  File "/home/pi/BatLogger/Interface/Hardware/acq_hardware.py", line 11, in <module>
    from . import interface_wrapper
  File "/usr/local/lib/python3.4/dist-packages/pyximport/pyximport.py", line 445, in load_module
    language_level=self.language_level)
  File "/usr/local/lib/python3.4/dist-packages/pyximport/pyximport.py", line 232, in load_module
    raise exc.with_traceback(tb)
  File "/usr/local/lib/python3.4/dist-packages/pyximport/pyximport.py", line 216, in load_module
    inplace=build_inplace, language_level=language_level)
  File "/usr/local/lib/python3.4/dist-packages/pyximport/pyximport.py", line 192, in build_module
    reload_support=pyxargs.reload_support)
  File "/usr/local/lib/python3.4/dist-packages/pyximport/pyxbuild.py", line 102, in pyx_to_dll
    dist.run_commands()
  File "/usr/lib/python3.4/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.4/dist-packages/Cython/Distutils/old_build_ext.py", line 185, in run
    _build_ext.build_ext.run(self)
  File "/usr/lib/python3.4/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/usr/local/lib/python3.4/dist-packages/Cython/Distutils/old_build_ext.py", line 193, in build_extensions
    self.build_extension(ext)
  File "/usr/lib/python3.4/distutils/command/build_ext.py", line 503, in build_extension
    depends=ext.depends)
  File "/usr/lib/python3.4/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/usr/lib/python3.4/distutils/unixccompiler.py", line 118, in _compile
    raise CompileError(msg)
ImportError: Building module Hardware.interface_wrapper failed: ["distutils.errors.CompileError: command 'arm-linux-gnueabihf-gcc' failed with exit status 1\n"]

I don’t see why the later tracebacks couldn’t just re-raise the initial error, preventing the additional clutter. As it is, I tend to do my test-building python3 script_name.py 2>&1 | head -n 50, but shell hacks seem kind of crude.

Alternatively, why not pack the error messages from the compilation failure into the error? It already contains lines from further down the build process:

ImportError: Building module Hardware.interface_wrapper failed: ["distutils.errors.CompileError: command 'arm-linux-gnueabihf-gcc' failed with exit status 1\n"]

It seems like if the g++ stderr was attached to the python error and percolated up through the handler chain, it could at least be output at the end of the traceback block, rather then before (for that matter, if you have a custom error with a overridden __repr__() and __str__(), does the interpreter use those when it’s printed to the console)?

[fake-name]

In this case, the project is some custom hardware, C++ that talks to it, and python that manages the runtime.

Development hasn’t even reached the point where I’d vaguely considered trying to abstract everything up into a library that gets versioned separately from the entire hardware stack.

I’d assume that the great majority of people writing cython code are not writing libraries, and as such pyximport is probably more broadly used/convenient then depending on distutils.

Note that there are probably more people using cython code via distutils then pyximport, but that’s separate from people writing cython code.

---

pyximport is ok for testing and developing. That’s its main use case, IMHO. Pull requests that improve pyximport are welcome. It has not changed in years and is not up to date with the less old improvements in Cython’s build support.

Uh, really? Like what? I use the pyximport stuff pretty heavily as a end-user of cython, typically to do spot-performance-improvements of non-widely-deployed code, and its been quite nice. If there’s improvements I’m missing, I’d love to hear about them.

[fake-name]

While that is an unbacked opinion, I fully appreciate people who extend the applicable range of tools beyond what they where initially designed for.

Hah, well, I did say I assume, so I’d happily admit I have little basis.

That being said, I think this ticket is getting out of scope. I’d like to restrict it the initial intention of improving the error output of pyximport.

Ok.