Support for binary templates

Hex editors are usually used for binary file formats. Currently only the file bytes are displayed (in hex), and to see a UInt32 value you have to know where it is in the file and move the cursor to it.

Most hex editors include a feature that allows to define binary templates for different file formats and view the parsed results as JSON or similar.

Update: This is no longer about one specific template format, but on supporting templates at all - see https://github.com/microsoft/vscode-hexeditor/issues/50#issuecomment-663935031

---

Original:

The cross-editor standard for binary templates is Kaitai Struct - https://kaitai.io/

It supports integration with many programming languages, it is free and open source, and it has a lot of community-contributed templates for many popular file formats - https://formats.kaitai.io/

The templates are defined using YAML, with a .ksy file extension.

Example template:

seq:
  - id: length
    type: u32
  - id: contents
    type: str
    size: length
    encoding: UTF-8
  - id: has_crc
    type: u1
  - id: crc32
    type: u4
    if: has_crc32 != 0

This will generate a JS class that you can pass the file data to and it will give you the parsed result.

Kaitai Struct also has a small web editor to try out templates that you can draw inspiration from:

Unfortunately Kaitai Struct doesn’t support editing the parsed results and writing them back to the file, but the parsing is the important part.

What do you think?

[NotWearingPants]

I wonder if there will be a way for other extensions to add features to the hex editor?

This made me realize that probably the best way to go about implementing this feature is to just give an API for extensions to register as parsers or parser adapters, and have the Kaitai part be implemented as an adapter extension.

This means we need to implement:

• Parser selection (with settings based on file extension?)
• Some sort of LSP-like protocol which returns the parsed results, and sends updates when the file is edited
  • The results are JSON with metadata about each field (highlight color, view type). Maybe the metadata can be a constant JSON schema (but without fields which have more than one type)?
  • Parser can report errors and warnings on specific fields or on the entire file
• Parsed results tree view
  • For each element show name, byte range, and type
  • Search names and values
  • When an element is selected, highlight the relevant bytes
  • When a byte/range is selected, highlight the relevant parsed element(s)
• Editing parsed results (if the parsed field supports it)
  • Support multiple formats (like vscode’s settings editor):
    • Decimal numbers
    • Hex numbers
    • Binary numbers
    • Strings
    • Hex byte arrays
    • Booleans
    • Enums
    • Arrays (add/remove/reorder items)
  • Validation (min/max for numbers, length for strings, one field being greater than another, etc.)
• Region highlighting in the file
  • The parser should define the grammar scope for each element, and the colors should come from the editor theme
• Tooltips on byte ranges showing their path in the parsed tree and size

[fudgepop01]

Hello! Well, I’m honored to be mentioned here ahahah

I’ve actually been eyeing this extension for awhile now! Even before I built the hex editor I use in my own, I was wondering if adding support to this extension would be sufficient. The main thing that turned me away was a lack of support for highlighting “regions” - blocks of bytes that represent various structures. In mine, I use SVG elements that are overlaid on top of the editor, but here (I think) that would need to be done using some sort of semantic highlighting mechanism.

As far as the underlying parsing, most things in my extension are done. It’s even able to handle recursive instances! However, the way I do it is technically incorrect and even that needs an upgrade at some point. Still, most of the important stuff is there and it would be SUPER cool to have my code in this thing.

That said, I’m also afraid I wouldn’t be able to dedicate much time to maintaining whatever I merge in here due to going back to college soon. That fact alone makes me think I should wait for awhile until I see how much time I’ll actually have to dedicate to things like this.