RoutablePageMixin URLs which include "." are not routable

I just lost 2 hours of work due to my ignorance of Wagtail’s page routing regex. Which is unsurprising, because it’s completely undocumented.

I was utterly baffled as to why the RoutablePageMixin code that I wrote to create an RSS feed for my NewsIndexPages was not being executed. I eventually had to fire up the debugger and dig deep into the resolver code to figure out why django was throwing a 404 from example.com/news/feed.rss, even though example.com/news worked just fine. I had the feed route very clearly defined with:

@route(r'^feed\.rss$')
rss_feed(self, request):
   ...

Turns out that the culprit is in wagtailcore/utls.py:

if WAGTAIL_APPEND_SLASH:
    serve_pattern = r'^((?:[\w\-]+/)*)$'
else:
    serve_pattern = r'^([\w\-/]*)$'

Because serve_pattern doesn’t include \. in its regex, example.com/news/feed.rss will never even engage the Page serving mechanism. Django cycles through your site’s URL patterns, sees nothing that matches serve_pattern, and gives up before calling any Wagtail code.

Now I’m not sure what the proper remedy for this would be. I’m assuming there’s some sort of reason that . is excluded from serve_pattern, so simply adding it doesn’t sound like a valid solution. I notice also that the javascript on the Page edit form prevents you from adding a . to a Page’s slug.

So that means a documentation update would probably be appropriate. Perhaps the RoutablePageMixin instructions should state which characters are valid for use in @route() regexes? I’d be happy to write a PR for that if you think it’s a good solution.

Note, too, that I had to contend with the bug acknowledged in #2871, since I use WAGTAIL_APPEND_SLASH = False. Fortunately, that wasn’t nearly so frustration-inducing.

[darthmall]

Just ran into this bug trying to create a routable page (for a product) that would resolve URLs for release notes of the product of the form <product>/releases/5.3 and <product>/releases/5.3.1, in case you’re looking for real world use cases to motivate fixing the bug. 😁

For now I’ll just use a valid slug of the version number to avoid the issue, but it’d be great to get this fixed. 🙏 😁

[gasman]

From a quick dig into RFC 3986, I think the full set of punctuation characters we should be accepting in path components is:

- . _ ~ % ! $ & ' ( ) * + , ; = : @

There might be edge cases when it comes to things like Unicode (at the protocol level URLs don’t allow it, but it depends whether Django sees the raw URL or an unescaped version) - but allowing the above set should solve the immediate use-cases (while avoiding the “what happens with slashes and question marks” concerns that we’d get from allowing everything).