Parsing of trailing TAB works differently for Python and C

This works properly (note the trailing TAB):

>>> from yaml import CLoader as Loader, CDumper as Dumper
>>> data = load('"bar"\t', Loader=Loader)

This fails:

>>> from yaml import Loader, Dumper
>>> data = load('"bar"\t', Loader=Loader)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/usr/lib/python3/dist-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/usr/lib/python3/dist-packages/yaml/composer.py", line 35, in get_single_node
    if not self.check_event(StreamEndEvent):
  File "/usr/lib/python3/dist-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/lib/python3/dist-packages/yaml/parser.py", line 142, in parse_implicit_document_start
    if not self.check_token(DirectiveToken, DocumentStartToken,
  File "/usr/lib/python3/dist-packages/yaml/scanner.py", line 116, in check_token
    self.fetch_more_tokens()
  File "/usr/lib/python3/dist-packages/yaml/scanner.py", line 258, in fetch_more_tokens
    raise ScannerError("while scanning for the next token", None,
yaml.scanner.ScannerError: while scanning for the next token
found character '\t' that cannot start any token
  in "<unicode string>", line 1, column 6:
    "bar"	
         ^

[ingydotnet]

Ah but they are not the very same parser. They are the 2 distinctly different parsers that PyYAML contains. A pure Python one and libyaml. Note that libyaml was originally a direct port from PyYAML, written by the same person.

There are several known places where PyYAML using pure Python and PyYAML using libyaml differ. These are of course bugs, either in the Python code or libyaml. For the test case you posted, libyaml parses according to the spec, and PyYAML’s python parser has a bug. That’s why I said libyaml is right and PyYAML (the Python code) is wrong.

Note: It was my understanding that you were trying to find out how to interpret the spec, so that you could implement your SnakeYAML Java YAML parser correctly.

[ingydotnet]

The short answer to your query is that in this case libyaml is right and pyyaml is wrong.

https://play.yaml.io/main/parser?input=ImJhciIJ shows the results of 14 YAML parsers, and PyYAML, Ruamel (fork of PyYAML) and SnakeYAML get this one wrong. The New Reference Parser there is literally generated from the spec productions and therefore is almost always correct in its interpretation. That might be a useful resource for you.

The productions involved are:

• https://yaml.org/spec/1.2.2/#rule-c-ns-flow-map-json-key-entry
  • https://yaml.org/spec/1.2.2/#rule-c-flow-json-node
    • https://yaml.org/spec/1.2.2/#rule-c-double-quoted
• https://yaml.org/spec/1.2.2/#rule-s-separate
  • https://yaml.org/spec/1.2.2/#rule-s-separate-in-line
    • https://yaml.org/spec/1.2.2/#rule-s-white

Which is spaces and tabs. Put another way, non-indentation whitespace is usually tabs and spaces.