Add support for all token types with the new semantic highlighting

Summary of the new feature

Powershell has 20 tokentypes and it would be nice if the semantic highlighting supported all of them. See: https://docs.microsoft.com/en-us/dotnet/api/system.management.automation.pstokentype

The current semantic highlighting implementation is currently missing the following types:

• Attribute
• CommandArgument
• GroupEnd/GroupStart
• LineContinuation (backtick)
• LoopLabel
• StatementSeparator (Semicolon)

If/When these gets added I would suggest changing the following mappings:

• Value in Using namespace <Value> should use CommandArgument scope. Currently it uses Function.
• Attributes like [cmdletbinding()] should use Attribute scope. Currently it uses Type.
• Names in functions and configurations should use CommandArgument scope. Currently they use the Function scope except functions with no dashes which has no semantic token type.
• Unquoted parameter values like “C:\SomePath” in Get-ChildItem C:\SomePath should use CommandArgument scope. Currently they use the function scope.

The following screenshot hopefully shows what I mean:

[rjmholt]

If there’s a way to do inheritance or something where you have a new token type called Attribute that falls back to Type if the theme doesn’t support it that might be cool

The problem is that no popular theme will support it, and we’ve just increased complexity without fixing the issue. Most people will continue using their Dark+/Monokai/Solarized theme and see attributes and types the same. So it’ll end up being like in https://github.com/PowerShell/vscode-powershell/issues/2856#issuecomment-669240917, where we correctly categorise the token differently and the colour is the same.

Otherwise I’m not sure it’s a great idea to use a classification that isn’t accurate (when an accurate one exists).

This is the real problem with themes. The themes themselves tend to be hacks designed to achieve nice coloration in the face of bad token categorisation. Which in turns forces the token classifiers to lie. I seem to recall there was a contentious change in the EditorSyntax repo where tokens were categorised more correctly and people complained because highlighting got worse.

I think the ideal here would be for us to define a correct token type, and a hack fallback, like: attribute -> label. That way the ISE theme could colour things with high specificity but users of popular themes will still get the color differentiation they’re looking for.

[MartinGC94]

Then you can do something 😃 Attributes have the “AttributeName” token flag which as far as I can tell is completely unique to attributes. Unquoted parameters have the kind+TokenFlag combination of “Generic” + “None” which would affect the same things I suggested “CommandArgument” would affect in the original post above.

For convenience here’s a table showing the relevant types and their Kind+TokenFlag combinations:

Token	Kind	TokenFlags
Command	Generic	CommandName
Unquoted Parameter	Generic	None
Type	Identifier	TypeName
Attribute	Identifier	TypeName,AttributeName