Expose Parser/AST Internals

Hi all, I’m trying to root around and see if I can re-use the parser and intermediate-representation (perhaps an AST?) that the Cython project uses for a separate project of mine. I’ve been able to find several wiki pages proposing changes to the system (ref1, ref2), but no up-to-date (read: 2016) summary of the system.

It’s been awhile since I’ve used Cython, but I remember really enjoying the commented-out Python code printed above each block of generated C code, it greatly helped with debugging. The Python ast module doesn’t support such a thing, although I believe the lib2to3 module does (although it seems a bit more heavy-weight).

[fabioz]

As a note, after dabbling a bit on the cython code, I’m using

from Cython.Compiler.TreeFragment import parse_from_strings
mod = parse_from_strings(name, source)

to obtain the AST and it seems it’s working properly for me (as a note, I started with TreeFragment but went for parse_from_strings because with TreeFragment changes the source code removing empty lines, which means that lines don’t match with that API).

[MarcoGorelli]

Thanks @scoder ! My issue is that that would still miss some types - for example:

from Cython.Compiler.Visitor import CythonTransform
from Cython.Compiler.TreeFragment import parse_from_strings

class MyVisitor(CythonTransform):
    def __init__(self):
        super().__init__(context=None)
        self.my_names = set()

    def visit_CSimpleBaseTypeNode(self, node):
        self.my_names.add(node.name)
        return node


content = (
    'from foo cimport bar, baz, bat\n'
    '\n'
    'ctypedef fused MyType:\n'
    '    bar\n'
    '    baz\n'
    '\n'
    'cdef void myfunc():\n'
    '    cdef int a = 0\n'
    '    a = 3\n'
    '\n'
)

tree = parse_from_strings('t.pyx', content)

my_visitor = MyVisitor()
my_visitor(tree)
print(my_visitor.my_names)

prints

{'void', 'int'}

, rather than {'void', 'int', 'bar', 'baz'}. This is because FusedTypeNode.childattrs is empty

I have to patch this by adding 'types' to child_attrs - would this (and the other patches in the link) be OK to do here in Cython directly? I asked about this one in the mailing list and was told

The effect of not including it in child_attrs is that types isn’t processed by ParseTreeTransforms but just left as is. As far as I can tell nothing interesting actually gets done with CSimpleBaseTypeNode or CBaseTypeNode by anything in ParseTreeTransforms. Therefore leaving it out practically makes no difference.

It possibly is an omission, but not one that has any consequence for Cython itself.