Trying to understand non-deterministic behaviour

Hi. Frist, I have to admit that I am new both to Lark and Python (I came from the land of C). I’m trying to write a simple assembler.

I’ve prepared the following Lark grammar:

start: source+
source: [LABEL_DEF+] MNEMONIC [[IMM_OPERAND|OPERAND|SYMBOL][","IMM_OPERAND|","OPERAND|","SYMBOL]] [COMMENT] _NL -> instruction
      | META [META_ARG] [COMMENT] _NL -> meta
      | [COMMENT] _NL

LABEL_DEF: SYMBOL":"
SYMBOL: NAME
META_ARG: DEC_NUMBER|HEX_NUMBER
OPERAND: DEC_NUMBER|HEX_NUMBER
IMM_OPERAND: IMM_DEC_NUMBER|IMM_HEX_NUMBER
MNEMONIC: "nop"|"stp"
META: ".org"
DEC_NUMBER: /[1-9]\d*l?/i
IMM_DEC_NUMBER: /#[1-9]\d*l?/i
HEX_NUMBER: /0x[\da-f]*l?/i
IMM_HEX_NUMBER: /#0x[\da-f]*l?/i
COMMENT: ";"/.*/
WHITE: " "|"\t"
%import common.CNAME -> NAME
%import common.LETTER
%import common.INT -> NUMBER
%import common.NEWLINE -> _NL
%ignore WHITE
%ignore COMMENT
%ignore _NL

(disregard the fact that there are only two mnemonics defined for now)

I’ve also got a test file that looks like:

.org 0xC00

foo:	nop 1
	nop 0x1,1	; fdas
foo2:
foo3:
	nop #0x1,1
	nop lab1
	stp #2

; comment
	nop		; fds
; fds

The parser works consistently on this example. However, if I change the first line to be a comment, to look like:

;.org 0xC00

Then approximately half of the time, I’ll get the expected parsing output, and the other half of the time I get an empty token at the beginning:

[]
[Token(LABEL_DEF, 'foo:'), Token(MNEMONIC, 'nop'), Token(OPERAND, '1')]
[Token(MNEMONIC, 'nop'), Token(OPERAND, '0x1'), Token(OPERAND, '1')]
[Token(LABEL_DEF, 'foo2:'), Token(LABEL_DEF, 'foo3:'), Token(MNEMONIC, 'nop'), Token(IMM_OPERAND, '#0x1'), Token(OPERAND, '1')]
[Token(MNEMONIC, 'nop'), Token(SYMBOL, 'lab1')]
[Token(MNEMONIC, 'stp'), Token(IMM_OPERAND, '#2')]
[Token(MNEMONIC, 'nop')]

Any hints on how to handle this?

[erezsh]

Yes, we’ve corrected this bug recently. I’m glad to hear that it seems to be working for others as well.

[federicotdn]

Glad it helped! If parts of the code are still non-deterministic when using a fixed PYTHONHASHSEED, then the problem may be that a class (not str, bytes or datetime) is missing its __hash__ function, and somewhere the code is iterating over sets/dictionaries of this class.