Special Complexity Handling for Left-Recursion Patterns?

Hello, I have implemented the work here in C++ (https://github.com/schrodingerzhu/pika_cxx) but I found a special cases in the complexity analysis?

Consider

A <- (A ~ 'a') / 'a'

Of course, this should better be written in A <- 'a'+, but the form should be okay for pika parser. However, consider parsing aaaaaaaa from the rightmost position, at the k-th step:

parsed 'a' -> add A to queue -> parsed A with 'a' -> add A to queue -> parsed A with 'aa' -> add A to queue ...

This will repeat for k times, so overall we should obtain a

1 + 2 + 3 + ... + length

steps and this should be quadratic?

[lukehutch]

@SchrodingerZhu I’m so sorry for the very delayed response. I was busy last October, then I forgot to come back to this.

Thanks for the detailed statistics and analysis. This does show that you are correct.

I think this is going to be unavoidable as a result of bottom-up, right-to-left parsing with this grammar, since the beginning of the OneOrMore run cannot be known a priori.

The following grammar should exhibit linear performance, since the rule is right-associative:

A <- ('a' ~ A) / 'a'

The reason this degrades to O(N^2) in the left-associative (left-recursive) case is that the beginning (innermost match) of the tree of left-associative (left-recursive) matches cannot be known a priori when matching right to left. But all matches have to be memoized along the way – so this is wasted work.

I actually don’t see a way to fix this, and I don’t think it’s fixable for this algorithm. This does undermine the strongest claim of this algorithm – that it is linear in the length of the input, albeit with a large constant overhead of wasted work. (Now the overhead is itself linear in the length of the input, making the overall performance O(N^2) in the length of the input.)

I really appreciate you pointing this out! This makes me think I should not publish this paper…

Actually the reason I remembered to come back to this bug report though was because this evening I had an idea about an extremely simple modification of standard left-to-right, top-down packrat parsing to enable it to support left recursion. I am planning to implement this to test it. I can let you know whether it works, if you are interested.

[SchrodingerZhu]