VSCode's tmLanguage support cannot match zero-width `begin` and `end` correctly.

The syntax is:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>fileTypes</key>
    <array>
        <string>testlang</string>
    </array>
    <key>name</key>
    <string>testlang</string>
    <key>patterns</key>
    <array>
        <dict>
            <key>include</key>
            <string>#indentedVerbatimOp</string>
        </dict>
    </array>
    <key>repository</key>
    <dict>
        <key>indentedVerbatimOp</key>
        <dict>
            <key>begin</key>
            <string>^([ \t]*)(?=(.*?)\|$)</string>
            <key>end</key>
            <string>^(?!\1[ \t])(?=[ \t]*\S)</string>
            <key>name</key>
            <string>string.unquoted.verbatim.youki</string>
        </dict>
    </dict>
    <key>scopeName</key>
    <string>text.testlang</string>
    <key>uuid</key>
    <string>159375af-d9b4-448c-a9da-d235eadf3556</string>
</dict>
</plist>

It matches indented block with a leading line ending with a bar (|). I’ve found that Code cannot match the block’s ending, which is zero-width:

Sublime can match it perfectly:

If I add some character to the end tag then Code works again:

            <key>end</key>
            <string>^(?!\1[ \t])(?=[ \t]*\S)xx</string>

[octref]

I still don’t understand why we do this:

• Why does begin & end have to consume anything? If the inner pattern consumes something it should be enough to advance parsing without infinite loop.
• I think there is a step wrong in https://github.com/Microsoft/vscode-textmate/issues/12#issuecomment-220957609. @alexandrudima my understanding is if you have begin that matches the beginning location of an empty location, the end pattern will go forward from that point on instead of starting matching at that point, causing the begin and end to consume the same zero-width part.
• There are many existing grammars that use lookahead or \G for boundaries. What’s a migration path for them? I don’t want to add (?=.) to each of them: https://github.com/Microsoft/vscode/commit/ccd3c1f94e64f7798ddf30d187274bd8905593fd.

/cc @aeschli

[alexdima]

@be5invis You can (re)tokenize the captured text via beginCaptures:

See here an example where it is done for a match captures, but the principle is the same: https://github.com/Microsoft/vscode-textmate/blob/master/test-cases/first-mate/fixtures/makefile.json#L350

Here the match captures the entire line which is then further (re)tokenized to assign more fine-grained tokens. You just need to convert that into equivalent plist.