Project coding standards should be documented

Here comes the grumpy guy again. Please understand that this is not at all meant to be an attack on any particular individual. But, I think it should be pointed out, in a top-level ticket, rather than in just after-the-fact comments, that repeated adjustment of a stable, multi-developer, open-source project’s style requirements creates a lot of friction. It’s questionable whether the benefits, in terms of actually encouraging code maintenance, are commensurate

My thesis:

The project’s coding standards should be documented. They should not be changed in an impromptu manner. Changes should be discussed — with the goal being to achieve consensus and buy-in from all contributors. It generally helps achieve buy-in if the standards are no more restrictive than is really necessary.

Python is not Javascript

Python, as a language, is somewhat unique in its expressivity and readability. I’m not totally sure why that is, but because of things like relying on indentation rather than braces, it is hard to write syntactically ugly python code. Of course, one can write bad code in any language, but in the curly-brace languages, it’s easy to make the program control structure hard to read in a way that is nearly impossible to do in Python. Many of those other languages do benefit from using tools to enforce indentation and other style requirements. Imposing overly-restrictive style requirements on python code is detrimental to the expressivity that is one of Python’s strengths.

For example, black won’t let me write:

config_ini.write_text(
    "name = Fred\n"
    "[section]\n"
    "foo = bar\n"
)

It changes that to the inarguably less readable

config_ini.write_text(
    "name = Fred\n" "[section]\n" "foo = bar\n"
)

And, does it really improve “readability” to change all the single-quotes to double quotes? (Sometimes a programmer can put the freedom to choose one or the other to good use. I tend to use single-quotes for “constant” strings: keys, etc., while using double-quotes for more general strings, e.g. messages.)

While it has a place, I consider the use of tools like black generally harmful. I think they discourage the careful writing of well-crafted code. See this blog post for more on that.

Unnecessary Friction

In addition to, arguably, at times decreasing the expressivity and readability of Python code, imposing too much pickiness in the style-checkers just creates all kinds of friction for contributors.

Just overnight, apparently, the CI tests (in the master branch) broke due to the release of a new version (2.7) of pylint. Now everything comes to a halt until someone fixes that. All these restrictions impose quite a bit of friction.

Unnecessary Churn

Every few months, it seems another change is made and a bunch of files are reformatted. See #696 (84 files touched), #823 (79 files), #871 (49 files), #887 (16 files). Reformatting the source code due to style changes on a stable project has many disadvantages.

• Outstanding PRs will no longer apply cleanly (or pass CI tests). In the past couple weeks, I’ve had to fix three outstanding PRs for breakage caused by #871 and #887. This is quite demotivating to contributors.
• It pollutes the commit log, making the git history less useful. (Look at this directory and try to determine when the most recent real change to any of the source files was made. The most recent commit for nearly every file is a “beautification” commit of some sort.)

For the most part, the python code was well-crafted from the beginning.

This is an established, stable code base, currently in use by many, contributed to by dozens over the years. The python code, for the most part, has always been well-crafted. (Admittedly, the tests are another issue — they could use some love.)

Arguments such as “consistent style make future debugging easier” are fine, so far as that goes, but, unless one can point out specific issues in the current code base which are truly improved by the changes, is it worth the churn? If there are one or two places where readability could be significantly improved, just fix those places.

Community Buy-In

Many of these pylint flags, the use of black, etc. are truly matters of personal preference. For most of them, there are arguments to be made both ways as to how useful they are in really improving readability or maintainability. It is rather discouraging to those contributors who happen to feel differently about some of these choices when the PR is made and merged without much discussion.

There is churn in the contributors to this project. The original author is no longer involved in maintenance. Even those who were active last year seem to have moved on. Personally, I feel that it is disrespectful to reformat another author’s code without good reason. (I find it a bit galling that the big blacking PR (#823) appears to have been the last contribution by that particular contributor.)

Getting pickier about style requirements in a public project results in more frustration and more arguments over style requirements. Is there some bit of the code that is hard to read? Fix it and move on. If there really is some pervasive, systematic bit of style that is causing significant problems, please identify it, illustrate that the proposed fix is worth the disruption, and allow for some discussion before pushing it to master.

There are a number of popular and published style guides starting with PEP 8. There are more specific guides like google’s, or django’s too. Lots of thought and experience has gone into these. When starting a new multi-developer project, pick one. Change it judiciously and deliberately if there is good reason. For an established, multi-developer project, be even more judicious and deliberate about changing style requirements.

If it is decided that code reformatting is called for, it should be in one big whack, rather than in bits and pieces, so as to minimize the commit log pollution. And it should be done with lots of warning, to try to avoid horking current PRs, etc.

If one wants to encourage developer participation in the project, it’s important that the potential developers feel heard when these things are discussed, and that unnecessary friction to contributing is removed.

Identify the problem before making the change

It is dismissive to others who care about this code to say, “we should black the code because it makes it more maintainable.” That’s fine as far as it goes, and it may be true, but without evidence, it just sounds like corporate-speak. If there parts of the codebase that would truly be improved by blacking, point them out. In an established, stable project like this, if there’s just one bit that needs fixing, just fix it — there’s no need to munge everything. If the issue is pervasive, point that out.

My Preference

As you can tell, I believe simpler is better. My preference, as you may have guessed by now, especially for a mature, stable, open-source project, is PEP 8 (with its “A Foolish Consistency is the Hobgoblin of Little Minds”.) Along with that, I feel that flake8 is better suited towards open-source, established projects than the pickier pylint. (Here’s more reading on that.)

Here’s a tweet from some random guy that agrees with me.

[xlotlu]

@dairiki I just dropped by to say: ❤️ man. 😃

Yes, I’m that one maintainer who opposed the horrible horribleness: https://github.com/lektor/lektor/issues/693#issuecomment-714203961. It’s also the reason why I left the project - or rather, the way it’s been handled is the reason. Among other things, this one in particular is eyes-rolling-worthy:

Personally, I feel that it is disrespectful to reformat another author’s code without good reason. (I find it a bit galling that the big blacking PR (#823) appears to have been the last contribution by that particular contributor.)

So dear mister, regards for the touch of sanity you brought to the discussion.

[yagebu]

Yes, the timing of some of these last two PRs was a bit unfortunate with relation to your PRs, sorry about that again.

Except for the no-self-use one (where I’d be quite fine with disabling the pylint rule again), these changes were fixing and motivated by actual bugs or issues, so I think it’s a bit harsh to brush most of the changes in the last months off as just “style-adjustment”.

I’d be in favor of keeping pylint, I find it helpful since it points things that I would try to manually watch out for in a review. It is annoyingly noisy though and doesn’t provide many helpful suggestions and the red cross does look quite scary in CI runs, so I understand that that can be off-putting. So I think documentation could be improved to point out that people shouldn’t be scared off by pylint warnings. And maybe we can run it in a less CI-breaking way so that its warnings are displayed as just warnings and not errors?