Document all token types

Motivation Themes depend on Prism producing tokens with specific types (or aliases). Right now, we do not guarantee or document any of those types.

Description Document all standard token types (e.g. keyword, comment, …). It should explain the general concept behind each token type and give at least one example.

The documentation should also include how languages are embedded. Example.

We should also guarantee that these concepts within languages are guaranteed to use these token types. (E.g. we guarantee that keywords always have a keyword class.) It might sound like we already do this but this is not the case right now. (E.g. we have many languages with operator keywords (e.g. NOT in SQL) that do not have a keyword class.)

[RunDevelopment]

I thought about a name again. How about “composite tokens”?

composite a distinct entity formed by the combining of two or more different things

I think the word “composite” it fits pretty well for our combination of non-standard and standard tokens.

[RunDevelopment]

It does seem to tie a bit into #2850 as well?

You’re right, it does. These “extended standard tokens”/token combinations (let’s hope this name doesn’t stick) are pretty much what I wanted and (if implemented) would resolve #2850.

Also, capturing and listing (some of) these non-standard tokens would be useful too, so there would be some consistency/re-use of non-standard tokens across languages in a way.

Good point.

I’m happy to help with the code/docs to whatever extent I can

Thank you so much!

Besides that, I think I’m really hitting the limits of the level I can think at right now. […] I’m sorry I can’t quite contribute to the conversation! 😊

Oh, I think you contributed quite a lot to it already. Thank you!