Implement PCRE style in-regex comments; e.g. (?#comment)

The Python regex implementation we used does not appear to implement any method of having in-regex-text comments which would work in the watchlist and blacklists.¹ It would be beneficial for us to be able to include comments in at least our watchlist and blacklist entries, and potentially the other regexes that we use in findspam.py. PCRE implements in-regex comments using comments like (?#comment).

It would be relatively easy for us to implement support for PCRE style regex comments. These could be implemented by just removing from the strings we convert to regexes any content which matches the regex \(\?#(?<!(?:[^\\]|^)(?:\\\\)*\\\(\?#)[^)]*\).²

This substitution could be performed at one of the following points (listed in in order of increasing generality):

1. For watchlist and blacklists only: when we read the watchlist and blacklist lines from the files
2. All 'regex' detections: just prior to using regex.compile() on the text provided in all the 'regex' detections, or
3. All regexes: as a wrapper to regex.compile().

---

1. There is the possibility of “Verbose” regexes using the X flag, which I assume is also available in the regex module we’re using. However, using these would not address having comments in the watchlist and blacklists.
2. That regex is untested, as it relies on variable length look-behinds for which I don’t have a simulator/tester. The regex \(\?#(?<!^\\\(\?#)(?<![^\\]\\\(\?#)(?<!\\\\\\\(\?#)[^)]*\) is tested and correctly matches, or not, for up to 3 \ escapes prior to the `(?#comment).

[makyen]

Prior to writing this RFE, I had found non-official documentation which said these are not implemented. In addition, the official documentation I looked in didn’t mention them, while mentioning other types of comments (“verbose” regular expressions).

However, having looked in the source code for regex, it does appear that this style of comment is already implemented as a standard part of both the re and regex implementations. After finding it in the source, I also found it in the re documentation.

So, there’s no need for this RFE, as it’s already natively supported. So, sorry to waste everyone’s time.

[makyen]

@quartata Assuming that python-pcre does implement just PCRE, it’s unlikely that it would be easier to use it. We currently use capabilities (e.g. variable length look-behind) which are not part of PCRE. If it was me, I’d much rather implement a single regex-replace (which is all that implementing this requires) than take on the known and unknown issues of moving to a different regex implementation.