RFC: Support for custom TSDoc tags

In RFC: Core set of tags for TSDoc, @karol-majewski brought up the topic of custom tags. Extensiblity is a primary goal of TSDoc. The challenge is that JSDoc’s tags have inconsistent tag-specific syntaxes, whereas a custom tag needs to be safely skippable by parsers that are unaware of the tag.

Kinds of tags

Here’s some proposed core tags that we already discussed supporting, grouped by their type of syntax:

Standalone tags

These act as simple on/off switches. In terms of parser ambiguity, it doesn’t matter too much where they appear inside the doc comment:

• @readonly
• @public
• @internal
• @alpha
• @beta
• @ignore

Section-starter tags

These probably will act as delimiters that start blocks of rich text content (which possibly may include inline tags). Probably they should appear at the start of their line. Their captured content will probably terminate with the first TSDoc tag that is not an inline tag:

• @remarks
• @returns
• @deprecated

Tags with unusual syntax

These are like section starter tags, but with special meaning for the “-” delimiter:

• @param variable.member - description
• @template TThing - description

These act like bullet items, so they must appear at the start of the line and must appear as a group. Otherwise they are like section-starter tags:

• @see {@link <https://domain.com>} and other resources

Inline tags

These use curly braces to delimit their content. Within the curly braces, highly specialized parsing rules apply:

• {@inheritdoc @scope/my-package/components:Foo.member}
• {@link @scope/my-package/components:Foo.member | the thing}

{@link http://blarg | the thing}

Custom tag syntaxes

Custom inline tags should be fairly easy to support. Some interesting design questions:

• Should we assume that unrecognized tags are standalone tags by default?

• If we want to support custom section-starter tags, what would the notation look like?

• For an unrecognized tag, should its content be included in the API documentation? Or should it be omitted? Is the answer different for different custom tag kinds?

[octogonz]

@raymondfeng’s PR #1296 for API Extractor got me thinking some more about how a documentation tool should handle unrecognized TSDoc tags.

In the current implementation, the TSDoc parser can interpret the @myExampleTag syntax three ways, according to how it is defined in the TSDoc configuration:

• block tag: (For example, @remarks or @privateRemarks.) Introduces a documentation block. In the AST, it creates a DocBlock object that becomes the parent of the following paragraph objects.
• modifier tag: (For example, @public or @eventProperty.) Its presence acts as a flag to communicate some attribute of the API. A modifier tag does NOT start a new block, and the tag is “invisible” to the surrounding text (although the attribute itself may be indicated in the generated docs somehow). In the AST, the DocBlockTag appears inline without any DocBlock.
• undefined tag: Gets parsed the same as a modifier tag, but has no special meaning, and typically is not shown in the generated docs at all.

Why are modifier tags parsed differently from block tags? Suppose someone meant to write this:

/**
 * Adds two numbers.
 *
 * @remarks
 * Use this API to add two numbers
 * together.
 * @privateRemarks
 * This API is rarely used. It should be replaced by DXP team's feature in Q4.
 * @public
 */
function add(x: number, y: number): number;

…but they actually wrote it like this:

/**
 * @public
 * Adds two numbers.
 *
 * @remarks
 * Use this API to add two numbers
 * together.
 * @privateRemarks
 * This API is rarely used. It should be replaced by DXP team's feature in Q4.
 */
function add(x: number, y: number): number;

In the above example, we don’t want @public (a modifier tag) to consume up the “Adds two numbers” text the way the @remarks (a block tag) would. That would produce an empty summary block, and in that case a documentation tool should probably discard the “Adds two numbers” text entirely (since modifier tags don’t use it).

If you try it in the TSDoc Playground, you will see that our special parsing of modifier tags causes this input to get rendered as intended:

Summary

Adds two numbers.

Remarks

Use this API to add two numbers together.

Modifiers

@public

And if you misspell @public (e.g. as @xublic), it is also parsed like a modifier, so you will see it rendered like this, which is pretty good:

Summary

Adds two numbers.

Remarks

Use this API to add two numbers together.

But this approach has some downsides. Consider an input like this:

/**
 * Adds two numbers.
 *
 * @remarks
 * Use this API to add two numbers
 * together.
 *
 * @xrivateRemarks
 * This API is rarely used. It should be replaced by DXP team's feature in Q4.
 *
 * @svgIcon 
 * UHJldGVuZCB0aGlzIHRleHQgY29udGFpbnMgc29tZSBieXRlcyB0aGF0IGVuY29kZSBhIGNvb2wtb
 * G9va2luZyBTVkcgaW1hZ2UgdGhhdCB3aWxsIGJlIHNob3duIG5leHQgdG8gdGhlIGRvY3VtZW50YX
 * Rpb24uLi4=
 */
function add(x: number, y: number): number;

Here, the person has misspelled the @privateRemarks tag, and they also used a custom block tag @svgIcon that is undefined in our TSDoc configuration (perhaps because it’s intended for some other tool).

The Playground renders it like this:

Summary

Adds two numbers.

Remarks

Use this API to add two numbers together.

This API is rarely used. It should be replaced by DXP team’s feature in Q4.

UHJldGVuZCB0aGlzIHRleHQgY29udGFpbnMgc29tZSBieXRlcyB0aGF0IGVuY29kZSBhIGNvb2wtb G9va2luZyBTVkcgaW1hZ2UgdGhhdCB3aWxsIGJlIHNob3duIG5leHQgdG8gdGhlIGRvY3VtZW50YX Rpb24uLi4=

This is bad:

• We’re accidentally showing internal developer commentary on the public website. Often this is no big deal, but for a commercial product it looks unprofessional and can be embarrassing.
• We’re rendering gibberish, because we ignored the @svgIcon block tag, and instead treated its content as being part of the @remarks tag

I’m thinking these problems may be worse than the original problems we were trying to solve when we introduced the special parsing for modifiers. Also, this special parsing rule is harder to understand, since it requires developers to memorize which names are “modifier tags” versus “block tags”.

What if we changed the parser as follows:

• In “strict mode”, block tags, modifier tags, and undefined tags always get parsed the same way: They always create a DocBlock AST node, and always consume any content up until the next such tag.
• If a modifier tag consumes any (non-whitespace) content, it is reported as an error.
• In “lax mode” parsing, if a tag is defined to be a modifier tag, we can associate its content with the previous block.

Do you see any problems with this change?

[Gerrit0]

The first two bullet points sound like an ideal solution to me. The lax mode change makes sense, but if I wrote a comment after a tag (that I didn’t know/forgot that was a modifier tag), I would be surprised to see the content appear under another tag… Unfortunately I don’t think I have a better solution. Just dropping the text isn’t any better.