Blog/docs: unlisted

🚀 Feature

As @MattiaPrimavera pointed out here: https://github.com/facebook/docusaurus/issues/3417#issuecomment-939320974

Docusaurus v1 had an “unlisted” feature allowing a blog post to exist in the production build, but to not appear in sidebar etc, and only those having the link could know the existence of it.

We should port this feature to v2 and make it consistent across all content plugins.

Things to not forget:

• meta noindex
• filter from the sitemaps plugin
• docs sidebar categories when all doc items are unlisted => filter category? (+ same for whole tree)

We might need to add an unlisted flag to the created route definitions to transmit this info to the sitemap plugin.

[shortcuts]

Is this new? Because I think we discussed that a long time ago with Kevin and he said it wasn’t supported

Yup, since we’ve migrated to the new infra/Algolia Crawler 😄

[shortcuts]

However I’m not sure Algolia has any metadata to exclude a page from their search engine crawler (afaik it doesn’t respect noindex. @shortcuts do you know if it’s possible to add an Algolia meta to exclude from search?

Hey! We respect every restrictions and metadatas (noindex, robots.txt, etc.) by default. You can freely set those on the Docusaurus side and we won’t index those pages.

In case you want to let you user know about it/let them control if it should be indexed, we have options to bypass them:

• https://www.algolia.com/doc/tools/crawler/apis/configuration/ignore-no-index/
• https://www.algolia.com/doc/tools/crawler/apis/configuration/ignore-robots-txt-rules/