The line length is not consistent through the whole project

Before adding the flake8 checker, we were allowing any line length, recommending people to follow pep8 (79 characters) but allowing any length “if it improves readability”.

When adding the flake8 checker we needed to add the E501 rule exception to most of the files (https://github.com/scrapy/scrapy/blob/master/pytest.ini), so technically, now we are allowing any line length in those files and a line length of 79 characters in the other files.

As this is bad for the code readability and maintenance (and self-contradictory), I think we should decide a clearer position of the line length for the entire project.

Some possibilities are:

1. Allow any length in all the project (if that’s the case we can just ignore the E501 rule for the whole project).
2. Restrict to 79 characters as PEP8 stands for.
3. Restrict to another agreed length.

In the 2nd and 3rd case it could be possible to allow longer lines individually (and not per file) by adding the # noqa: E501 to the line.

Looking at the project

To get a big picture of the issue I decided to check the current status of the project and I graphed this (please, obviate the y axis scale), that represents the line length frequency (for those lines with length > 79).

When looking at the graph, it is curious to see that when arriving to the 110 characters it decreases a lot and that there is a “natural” end in about the 140 characters.

You can find here the full file list: https://gist.githubusercontent.com/noviluni/0993ac9d2cbdad12e84b8f5ba2a18d17/raw/d84c298645bd5d6b10545726c2292a417432888c/line_too_long_ordered.txt

Solutions

1. Allow any length

I think it’s obvious that this (and the current) option is the worst, as it allows some lines to be excessively long. Take this example: https://github.com/scrapy/scrapy/blob/c841a1f3e7c0d44b277dd42300948ba2f6109e67/tests/test_utils_sitemap.py#L26 that is 237 characters long without improving the readability (in fact, it gets worse).

2. Restrict to 79 characters (PEP8)

Some maintainers have declared in the past that this is the prefered option, but that if the code readability improves we should allow longer lines. If we decide to follow this, we should first decide or at least try to describe what does “improves the readability” mean and allow only those cases to get a longer length by adding a # noqa: E501 to those lines, avoiding the current status, where we allow all the file to have any line length.

3. Restrict to another agreed length.

There are several projects where the maintainers have decided to follow another line length.

• Django: “Don’t limit lines of code to 79 characters if it means the code looks significantly uglier or is harder to read. We allow up to 119 characters as this is the width of GitHub code review; anything longer requires horizontal scrolling which makes review more difficult.”
• Flask: “79 characters with a soft limit for 84 if absolutely necessary”
• Requests: “Line-length can exceed 79 characters, to 100, when convenient. Line-length can exceed 100 characters, when doing otherwise would be terribly inconvenient.”
• Pandas: “We restrict line-length to 80 characters to promote readability”
• Black: “Black defaults to 88 characters per line, which happens to be 10% over 80. This number was found to produce significantly shorter files than sticking with 80 (the most popular), or even 79 (used by the standard library). In general, 90-ish seems like the wise choice.”

On the other hand, some editors/IDEs (like Pycharm) default to 120 characters.

My opinion

I personally like the Django’s argument, as it isn’t subjective, it exposes a good and objective reason to limit the line length to 119 characters. Moreover, after looking the graph it seems that the “natural end” of the current codebase is about that number.

I can understand that PEP8 purists could consider a sacrilege to allow all the lines to be longer than 79 characters by default, but I consider that the modern screens shouldn’t have any problem to show them completely.

Your opinion

What do you think? Is there any of the three proposals that fits the project? Do you have more arguments (pros and cons) or another point of view?

All feedback is welcomed 😃

[noviluni]

After merging this: https://github.com/scrapy/scrapy/pull/4237 I think we can finally close this 😄

Thank you @elacuesta, @Gallaecio, @kmike, @wRAR and @nyov! ❤️

[nyov]

My personal opinion is:

• Any but mobile device screens these days are widescreens (and those usually come with landscape mode switchers), but even 4:3 screens have more screen-width than height.
• Any sensible developer tool allows the use of that screen estate, so even with, say, side-by-side diffs one probably should have plenty screen for long lines.
• A whitespace-indent-level language like python, that also “requires” a 4-space indent according to pep8, shouldn’t at the same time enforce a line limit of 72/79 chars; that’s bad for indent-heavy lines.

So I would prefer if we could stick with “sensible” default of “any reasonable length”, making case-by-case decisions on that.

But 119 chars also feels plenty for use as a hard-limit, to me. I can agree to that. However 79 feels rather draconian, with no reasonable hardware limitation behind it any more, and I don’t agree with that pep8 rule. That only encourages people to use non-descriptive short or one-char variable names, and hurts legibility and quick code comprehension far worse than longer lines of text would, in my opinion.

But “sensible” or “reasonable”, in my opinion also wouldn’t apply to the quoted line of https://github.com/scrapy/scrapy/blob/c841a1f3e7c0d44b277dd42300948ba2f6109e67/tests/test_utils_sitemap.py#L26 since that one could easily be pretty-printed to a much more sensible

self.assertEqual(list(s), [{'changefreq': 'daily',
                            'lastmod': '2009-08-16',
                            'loc': 'http://www.example.com/',
                            'priority': '1'},
                           {'changefreq': 'weekly',
                            'lastmod': '2009-08-16',
                            'loc': 'http://www.example.com/Special-Offers.html',
                            'priority': '0.8'}])