pyproject.toml 'include' directive parsing bug.

Highlighting here but elaborated a bit in https://github.com/python-poetry/poetry/issues/6094#issuecomment-1326009538

• Poetry version: 1.2.2
• Python version: 3.8.10 (Windows) / 3.10.6 (Ubuntu)
• OS version and name:
  • Cygwin under Windows 11
  • Ubuntu 22.04.1 LTS (GNU/Linux 5.15.74.2-microsoft-standard-WSL2 x86_64)
• pyproject.toml: https://gist.github.com/RichardBruskiewich/7c41ce4ea3b61224dcbf3080df04b9d0

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate (but slightly related to https://github.com/python-poetry/poetry/issues/6094#issuecomment-1326009538).
  • Now recognized to be a possible duplication of an unresolved 3 year old issue https://github.com/uiri/toml/issues/270
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Traceback (most recent call last):
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/__init__.py", line 185, in console_main
    code = main()
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/__init__.py", line 143, in main
    config = _prepareconfig(args, plugins)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/__init__.py", line 318, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/helpconfig.py", line 100, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1003, in pytest_cmdline_parse
    self.parse(args)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1283, in parse
    self._preparse(args, addopts=addopts)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1157, in _preparse
    self._initini(args)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1076, in _initini
    rootpath, inipath, inicfg = determine_setup(
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/findpaths.py", line 184, in determine_setup
    rootdir, inipath, inicfg = locate_config([ancestor])
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/findpaths.py", line 106, in locate_config
    ini_config = load_config_dict_from_file(p)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/_pytest/config/findpaths.py", line 69, in load_config_dict_from_file
    config = toml.load(str(filepath))
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/toml/decoder.py", line 134, in load
    return loads(ffile.read(), _dict, decoder)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/toml/decoder.py", line 511, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/toml/decoder.py", line 778, in load_line
    value, vtype = self.load_value(pair[1], strictly_valid)
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/toml/decoder.py", line 880, in load_value
    return (self.load_array(v), "array")
  File "/mnt/c/Users/richa/PycharmProjects/reasoner-validator/ubuntu/lib/python3.10/site-packages/toml/decoder.py", line 1002, in load_array
    a[b] = a[b] + ',' + a[b + 1]
IndexError: list index out of range

Issue

I added the following directive to a pyproject.toml file:

include = [
    "CHANGELOG.md",
    { path = "tests" },
    { path = "docs" }
]

Using my library, e.g. just running PyTests, triggers a IndexError: list index out of range exception reading in deep in the toml decoder.py module here:

                    while (not self.bounded_string(ab) or
                           (len(ab) > 2 and
                            ab[0] == ab[1] == ab[2] and
                            ab[-2] != ab[0] and
                            ab[-3] != ab[0])):
                        a[b] = a[b] + ',' + a[b + 1]  # <= site of error

but this error is not triggered either by

include = [
    "CHANGELOG.md"
]

alone, nor by

include = [
    { path = "tests" },
    { path = "docs" }
]

alone.

I guess I’ll remove one or the other components of the include, but it would be helpful to know what’s happening (and fix it!) or, at least, document the constraints of include usage accordingly.

[RichardBruskiewich]

@neersighted, I deeply appreciated your frank reply. I sincerely apologize for the consternation my own comments inspired on your part.

As a good colleague of mine remarked when I was once on the other (receiving) side of such comments: “…maybe so…and…so is just having a bad day on their (software engineering) side…”. I hope that you can make such allowances for me today.

I was obviously mistaken in considering the error within the remit of poetry: the pyproject.toml file in my project was only created when I migrated the project to Poetry… in fact, poetry created the initial file and I simply added to it following poetry documented additions. I guess I’d be forgiven for incorrectly assuming it was a poetry bug… I was inadvertently - not so blissfully - unaware that poetry and pytest share the same name for a configuration file. What an eye opener! Talk about (admirable!) standards reuse

I furthermore now also understand the point that there are multiple diverse implementations of TOML parsing, some more compliant than others with the expected constructs… and that Poetry and Pytest each use different libraries to parse their TOML file! Talk about lack of reuse of standards, in terms of libraries!

On the issue of open source, I’ve long been a fan of, and contributor to, many such projects, so I do understand your angst with my harsh comments.

Again, please accept my apologies… and also, my gratitude for your thoughtful reply.

For now, I’ll just avoid the offending construct in the TOML file in question since I don’t actually enough professional or personal degrees of freedom at the moment to jump into the fray of uiri/toml to fix this bug… although all of the information gleaned here about the coding landscape for this was very helpful (I’m truly sorry that I didn’t pick up on any of this earlier…)

[RichardBruskiewich]

@dimbleby thank you for the head’s up on this!