Proposal: Shades of Black

We often get proposals for formatting changes that could be useful, but that change the AST or are otherwise unsafe. Examples include:

• Import sorting (#333)
• Formatting stringified annotations (#3067)
• Dropping useless f-strings (#3081)
• Aspects of docstring formatting (#144, #2150)
• not (x in y) (#212)

Many of these proposals make sense as an extension of Black’s mission to create a single, consistent style for Python code, but users have legitimate concerns about the safety of such changes.

Relatedly, there are some tools that wrap Black together with other formatters:

• https://github.com/Zac-HD/shed (black+isort+pyupgrade+autoflake+blacken-docs)
• https://github.com/omnilib/ufmt (black+usort)

However, these tools have not reached Black’s level of popularity. Creating this logic in Black itself would make it more discoverable for users.

I observed that several of the existing formatting options can be described as “either leave it alone, or let Black make its decision”: --skip-string-normalization, --skip-magic-trailing-comma. That’s a useful principle to follow: formatting options shouldn’t be about using one style or another, only about whether to handle some aspect of formatting at all.

Proposal

So here’s the proposal: There is only one Black code style, but you get a choice how far you commit to it. We’d introduce a system of shades, aspects of formatting that can be individually turned on or off depending on what is safe on your codebase. Initial shades could include:

• String normalization: on by default; if off, string quotes are left alone.
• Docstrings: on by default; if off, docstrings are not changed at all.
• Misc AST unsafe: on by default; if off (and other AST-changing options are off), we guarantee full AST safety. I believe this would currently only affect del parentheses, but we could make a few other things go into this group (#3081, #212). These would be things that change the AST, but in minor ways that we’re confident don’t change the meaning of the program.
• Magic trailing comma: off by default; if on, we ignore pre-existing magic trailing commas.

In the future, we could add additional shades for import sorting, annotation formatting, and other automatic improvements to Python code.

We could add shortcut options for enabling all shades, like black --all-the-way for enabling everything, and black --super-safe for disabling all AST-unsafe options for the paranoid among us. (Names could be improved.)

These new shades would be orthogonal to --preview style and the stability policy: any new shades would be enabled only in --preview mode until the new year starts.

Alternatives

• Do nothing. The proposal adds significant complexity to Black, and we’ve billed ourselves as a simple, no-configuration tool. I have a lot of sympathy for this position. But if we don’t do it, people will build other tools for additional formatting, creating a more fragmented ecosystem that’s harder for users to navigate. Black has a chance to provide a much simpler solution to users.

[Zac-HD]

I’d caution strongly against adding shades:

• The minimal-configuration aspect of black is very valuable for user experience across projects.
• Semantic changes are much more contentious than formatting alone. not x in y -> x not in y would be fine, but I don’t think it’s feasible to build consensus on how to sort imports - and IMO that means it should be left to other tools.
• This seems like a poor fit for the once-a-year stability policy.
• I’m also concerned about the maintainence burden on Black maintainers; supporting shades is likely to mean that the project can never really approach ‘finished’.

I really value Black as a single centralized source of uncontroversial zero-config autoformatting. If the Black maintainers are interested in providing semantic changes as well I’d be delighted, but prefer to see that built and stabilized in a downstream package before considering whether it should be upstreamed.

[felix-hilden]

After discussing with the team internally, we’ve decided to:

• Not introduce new configuration for AST safety or add shades
• Continue formatting docstrings and making improvements there

So this is the hard “do nothing” option. I’m sorry to disappoint, but the advantages of us formatting docstrings outweigh the rare cases where making such modifications is problematic. And consolidating formatting choices is preferable to strict separation of concerns (using another tool). Now all that’s left is to make the processing as good as possible 👍 Thank you for the discussion, to me it’s been a fruitful one since now we’ve clarified what we want to do exactly.