Poetry builds sdist against common packaging practices

• Poetry version: 1.2.2
• Python version: Python: 3.10.7
• OS version and name: Red Hat Enterprise Linux 8.6 (Ootpa)
• pyproject.toml: https://gist.github.com/BrendanJM/3834260523acb0d04d75c8f491a243a4

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Poetry 1.2.2 introduced this change (https://github.com/python-poetry/poetry/pull/6621) with the intention to normalize sdist naming conventions, which appears to put the generated sdist files in muddy waters regarding PEP625 and historical convention for sdist naming. The specification states:

The name of an sdist should be {distribution}-{version}.tar.gz.

Unless I’m missing something, it does not specifically state how to handle underscores in the distribution names themselves, but convention has been to have the package name contain dashes rather than underscores for sdist, and underscores for the wheel. E.g., for the example pyproject.toml, the previous behavior (v1.2.1) was:

# poetry build
Building my-package (0.0.1)
  - Building sdist
  - Built my-package-0.0.1.tar.gz
  - Building wheel
  - Built my_package-0.0.1-py3-none-any.whl

The introduced changes appear to violate this for packages which have an underscore/dash in them. In the latest version, it builds an sdist package with underscores rather than dashes:

# poetry build
Building my-package (0.0.1)
 - Building sdist
 - Built my_package-0.0.1.tar.gz
 - Building wheel
 - Built my_package-0.0.1-py3-none-any.whl

While as far as I can tell this does not technically run awry of the PEP, we are seeing in practice that our internal PyPI infrastructure in place seems to reject sdist packages with an error along the lines of:

Exception: cannot publish unexpected package name my_package-0.0.1.tar.gz (expected my-package-0.0.1.tar.gz)

I cannot pretend to understand all of the configuration or customizations that exist in our internal PyPI infrastucture, as the exact nature of this is obfuscated from me. I also understand that it would be easy to close this problem as WONTFIX, under the idea that “this does not sound like a poetry problem”. But I think the changes we are seeing were not intended to be breaking, especially as a patch fix. But this change appears to have inadvertently injected a fairly opinionated change to how sdist packages are named that breaks implied backwards compatibility, without offering much in the way of clear upside. If we have issues on our system, I imagine others who rely on this historical convention may also see issues.

This thread in pypa/build offers some good insights here: https://github.com/pypa/build/issues/369. Specifically, this helps contextualize this:

For example, the wheel always use _ because using - in the filename would make parsing ambiguous. An sdist can technically use either (the rightmost dash separates the name and version aprts); underscore is preferred for consistency with wheels, but setuptools use dash for backward compatibility reasons (its behaviour outdates wheels).

I think ultimately the intent of name normalization names sense, and yes, it is probably the fault of other systems for relying on convention over explicit specification. But in this case it seems as though there is a high potential this violates assumptions that other systems hold that poetry would ideally integrate nicely with. And in doing so, it’s not clear that any specific value has been added in terms of coercing dashes to underscores as the means of normalization.

In thinking about how to resolve this, my recommendation would be the following:

1. Revert the breaking behavior in a follow-on patch release.
2. Add a flag that allows users to opt in to this normalized naming convention that is by default off
3. Provide warnings that this behavior will be switched to default in a follow-on release (if we actually care - though TBH it seems like it’s not really an issue, so I’m not sure what benefit there is to fixing it)
4. In a subsequent minor release, flip this to default, allow users to opt in to legacy behavior (if the new behavior really is the desired default)

I’d be interested in hearing thoughts on what the intent and preferred way forward would be here.

[neersighted]

This is not the best forum, but hopefully the conversation is short:

What is being referenced here is sdist names, which are now normalized like wheel names. Technically a registry can still host a package using a pretty name (with dots or capitalization, etc) while serving distfiles with normalized names. However, as PyPI derives the project name from the name of the first artifact uploaded, the only way to name a PyPI project with a pretty name is to upload using a renamed sdist with Twine, so that PyPI uses the pretty name of the sdist.

After that point, you can use Poetry to upload and PyPI will correctly match the files to the project. But Poetry will only ever output normalized names as required by the spec, so unless you rename the files yourself you cannot manipulate PyPI this way.

Other indexes that don’t infer a project name from an attempt to upload a non-existent name will be less obnoxious here. There’s also the fact that the naming rules still create ambiguity and PyPI misbehaves here (but won’t fix it under the under-specification is solved) e.g. https://discuss.python.org/t/amending-pep-427-and-pep-625-on-package-normalization-rules/17226.

[Secrus]

@neersighted do you have some links to when Poetry became compliant to this naming ambiguity (PR / issue …)?

Here: #6621