Possible optimization?

https://github.com/lark-parser/lark/blob/7c8d6ee787692f7287d38de92d06174bf956d72a/lark/lexer.py#L193 to

lenstream = len(stream)
while line_ctrl.char_pos < lenstream:
  ...

In [1]: stream = ' ' * int(1e6)                                                                                    

In [2]: %timeit for _ in range(int(1e6)): len(stream)                                                              
71.9 ms ± 3.54 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [3]: lenstream = len(stream)                                                                                    

In [4]: %timeit for _ in range(int(1e6)): lenstream                                                                
30.7 ms ± 853 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

[erezsh]

Yep. I’m not sure it will be noticeable in a real task though.

[erezsh]

Thanks Ludwig! I’ll close this now unless someone’s benchmark shows otherwise 😃

@pwwang Personal computers usually aren’t tuned for benchmarking. There’s random processes popping up, the cpu may change speed according to temperature, and other such pitfalls. I would guess that many of the differences are due to such random circumstances.