Error parsing preprocessor statements with a space in front after stubbed macros

I’m noticing a misparse when a #define is preceded by a space.

#define MACRO_STUB
#define MACRO x

yields this parse: (translation_unit (preproc_def name: (identifier)) (preproc_def name: (identifier) value: (preproc_arg)))

but if I insert a space before the second #define like this:

#define MACRO_STUB
 #define MACRO x

yields this (incorrect) parse: (translation_unit (preproc_def name: (identifier) value: (preproc_arg)))

Notably, the parser still produces correct output if the first macro is not stubbed:

#define MACRO_NOSTUB x
 #define MACRO x

(translation_unit (preproc_def name: (identifier) value: (preproc_arg)) (preproc_def name: (identifier) value: (preproc_arg))) I tried like one change so far that didn’t work (since I don’t understand the parser well), changing preproc_arg to: preproc_arg: $ => token(prec(-1, repeat1(/[^\n]|\\\r?\n/))),

[amachronic]

In the extreme case, a preproc directive can occur between any two tokens. I think putting them in extras is the only correct solution.

@pluick – It seems comment tokens are returned by tree-sitter, there is even a test case (extra_non_terminals) in the tree-sitter repo which shows them being returned despite being in extras. Otherwise, every editor using tree-sitter for syntax highlighting must be using language-specific hacks for highlighting comments…

[amachronic]

I’d just like to point out an example:

void f() {
#if A
    if(x) {
#else
    if(y) {
#endif
        blech();
    }
}

There are two versions of the file – both are syntactically valid, but it’s impossible to combine both versions in a way that fits the grammar. I suppose this could be handled by treating the different branches as ambiguities but tree-sitter isn’t designed to parse ambiguous grammars, is it?

If a ‘perfect’ parse is impossible, maybe the preprocessor aspect of the C/C++ grammars should be designed with that in mind.