Problem with natural numbers

I’m at a bit of a loss as to why this is happening so I thought I’d ask here. Following is a small reproduction of what I’m trying to achieve.

I have the following grammar:

{
  "$schema": "https://raw.githubusercontent.com/martinring/tmlanguage/master/tmlanguage.json",
  "name": "Unison",
  "patterns": [{
    "match": "([^\\+\\-\\w\\d]|^)(\\d+)",
    "captures": {
      "2": {
        "name": "constant.numeric.integer.unsigned.unison"
      }
    }
  },{
    "match": "([^\\w\\d]|^)([+-]\\d+)",
    "captures": {
      "2": {
        "name": "constant.numeric.integer.signed.unison"
      }
    }
  }],
  "scopeName": "source.u"
}

if I build a Grammar from this json and try to tokenize +42, I get as tokens

[ { startIndex: 0,
    endIndex: 3,
    scopes: [ 'source.u', 'constant.numeric.integer.signed.unison' ] } ]

which is expected.

However when I tokenize 42, I get:

[ { startIndex: 0, endIndex: 3, scopes: [ 'source.u' ] } ]

without the unsigned scope.

[msftrncs]

Its because your first part of the rule says that you cannot match a digit, and the 4 is a digit. I bet 42 ‘(space)42’ works.

I think you meant the first part to be a look behind. I recommend staying away from look behinds. Always try matching going forward, as looking behind can set you up for things that are not the way they seem. For instance, looking behind for a quote, was the quote escaped? You’d have to look behind further, but wait, was the escape escaped? When you first start out writing a grammar you will want to use look behinds because you haven’t handled much yet, but you should try to refrain from that, save the failures as TODO items . Once everything is scoped, theres almost no reason to look behind.

[msftrncs]

For clarification, I am still not quite right. \b always gives me trouble. \b works like a look behind (hence it should be refrained from being used) as (?<=\w)(?=\W)|(?<=\W)(?=\w). So, ‘(space)+’ doesn’t count as a \b between the space and the ‘+’, so it fails. It would instead require:

"match": "\\b(\\d+)", and "match": "(?<=\W|^)([+-]\\d+)",

This just demonstrates why look behinds should be avoided. Instead, all text should be matched by rules looking in the forward direction only. Only use look-behinds when you know exactly what could have been matched previously.