meta: Product-agnostic HCL syntax highlighting

Problem Statement

Most editors, plugins, extensions (including this Terraform VS Code extension), or websites wanting displaying files with some flavour of HCL (such as Terraform language) need a way of highlighting these files.

The fundamental problem of highlighting involves answering two (related) questions:

• which files to highlight
  • e.g. highlight *.tf as Terraform configuration
• how to highlight the file that should be highlighted
  • e.g. sequence of bytes between 0 and 10 on line 3 is an attribute name

Often times implementations start by supporting a single flavour of HCL, such as Terraform and many end there, or just decide to treat different HCL flavours as equivalent, e.g. highlight Nomad job spec with the same set of rules as Terraform configuration.

This creates a few challenges as described below.

Differences between Terraform versions

Terraform language contains a number of components which change between versions. For example function names, or block types. Product-specific highlighting often hard-codes such components, which makes it version-specific.

For example alltrue or anytrue functions were added in Terraform 0.14.0, but are highlighted differently (in this VS Code extension) than other functions introduced earlier (such as lower or max):

Other HCL-based languages

Reusing the same implementation which is tailored for one product (Terraform) results in inaccurate and incomplete highlighting.

While the underlying parser is the same and components available within HCL2 are the same, the highlighting differs in small subtle ways. This results in inconsistent and suboptimal experience across HashiCorp products.

For example Packer template highlighted with Terraform VS Code extension:

or Nomad job spec

User Impact

All users of VS Code Terraform extension would benefit from more accurate highlighting of Terraform configurations.

At the time of writing there’s close to 18 000 public repositories on GitHub which are detected as majority HCL (with close to 1M files across them). All of these files in these repositories could be highlighted more accurately.

Visitors of all HashiCorp .io product sites contain snippets of HCL configurations which would benefit from more accurate highlighting.

Expected User Experience

• Users of the VS Code Terraform extension are provided with files highlighted consistently and correctly for any Terraform 0.12+ compatible configuration.
• Users displaying Terraform configuration on GitHub see it highlighted consistently and correctly for any Terraform 0.12+ compatible configuration
• Visitors of terraform.io and other HashiCorp .io websites see snippets of HCL2 code highlighted accurately

Proposal

• create new repository, e.g. hashicorp/hcl-grammar to host the HCL grammar
• implement a generator which generates TextMate JSON grammar (initially) for product-agnostic HCL
• commit generated grammar to the repository
• create tests to cover the generated grammar, e.g. via https://github.com/PanAeon/vscode-tmgrammar-test
• use this grammar in the Terraform extension to highlight *.tf files (while assuming that any richer highlighting experience can be provided via LS)
• switch GitHub’s linguist to this grammar for *.tf, *.hcl and other known extensions

Blocking Questions (Optional)

• Is generating syntax grammars feasible? A prototype would likely need to be created to answer this question.

[jpogran]

I can’t explain more at the moment, but this is something we’re actively working on and will have more information in the coming weeks.

[github-actions[bot]]

I’m going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.