Extremely poor docs.python.org SEO performance.

Describe the bug docs.python.org has atrocious search performance on Google. It’s so bad that I suspect Google is actively downranking it for some reason.

To Reproduce Search Google for virtually any Python documentation topic. The ones that drove me here were searches for [python set] and [python shuffle list].

Expected behavior docs.python.org is the authoritative source for Python documentation on the web. I expect to find relevant results on docs.python.org somewhere on the first page of Google search results.

Instead, I find the opposite. For [python set] I expect to find https://docs.python.org/3/library/stdtypes.html#set somewhere on the first page of results, but instead, the only python.org result I see is for the long-deprecated Python 2 sets module – https://docs.python.org/2/library/sets.html

For [python shuffle list], neither https://docs.python.org/3/library/random.html nor any other python.org result shows up anywhere on the first page.

Screenshots

Additional context These results are egregious enough to make me suspect you’re being actively downranked for some reason. This isn’t a request for general SEO optimization – although that’d be a great project if someone has the interest – but for a domain admin to try to use Google’s search console (https://developers.google.com/search) to investigate if there’s something egregiously wrong with an easy fix.

[ericholscher]

It looks like Python is indexing old versions, and they are disallowed in the robots.txt: https://docs.python.org/robots.txt – at least the link that @di is pointing at. That is probably the first thing I’d try to fix them, but agreed that canonicalization can sometimes help. Google is quite fickle though, and hard to understand how to fix this. We’ve tried a number of different things.

We have lots more tips here: https://docs.readthedocs.io/en/stable/guides/technical-docs-seo-guide.html.

I’m guessing the Python docs history of non-mobile friendly design has probably hurt it a lot over time. I believe that’s fixed now though.

The first step I’d do is probably add canonical links to /3/, since I believe that is the “canonical” version. (It looks like y’all are already doing that though, so yea, the unknowable Google indexing conundrum lives on).

The next step is definitely diving into Google Search Console for what it says there.

[davidfischer]

I may probably be fixed with some proper sed-fu, but is it worth it as previous versions are denied by robots.txt:

I would definitely fix it. This will stop version 3.4 from showing up in Google’s results. You may have to open up the robots.txt for a while after making the change but I’m not sure there.

As to whether to take on a huge docs reformatting/rework projects, it’s a terrifying never-ending project of incremental improvement. I’d fix all the concrete easy things (like 3.4 docs showing up in search engines) first.