Allow publishers to embed metadata, enhancing etcher functionality and UX

Etcher’s mission has been to empower users to flash images with a minimal amount of effort while using intelligent extraction of context from the environment to prevent common failure modes.

There are a few more things we can do to improve on this mission, but those too will soon run out if we continue at the current pace. The next frontier for improving functionality and user experience is to create a channel whereby image publishers can convey additional information about the image being written. Etcher can then utilise that metadata to optimise its current functionality, and eventually provide additional features to users.

Types of extended metadata

Some examples of what kinds of metadata could be bundled:

• Logos, links, descriptions about the publisher, intended device(s), and the image itself.

On the most basic level, additional descriptive metadata can help Etcher customise its appearance, inform the user, and generally provide an experience that fits better with the workflow Etcher is being used for, both stylistically and functionally.

• Image bmap

bmaps are an exciting technology allowing image writers to “skip over” meaningless areas of the image (e.g. what would normally represent empty space for the filesystem being written). Indicatively, a bmap can save anywhere from 15% to 60% of the time used to write, though due to the nature of the technology, that number could theoretically range anywhere from near-0% to near-100%.

• Configuration metadata

Assuming there was a reliable way to map a fairly simple set of key-value pairs into various configuration files in a filesystem, the extended format could include those mappings, allowing Etcher to present a fairly simple ui to the end-user, and then use that data to configure the image. Image configuration is particularly important for images aimed for headless devices that give their users very little opportunity to interact with them once booted.

• Image update checking

A common problem is that an image that is downloaded may be used much later, when a newer version exists. That newer version may have fixed an important bug or simply be better aligned with the documentation. Having a way to notify a user that there is a newer version of the image selected would help avoid many negative experiences.

There are, of course, many more things that can be done with extended metadata, but each of the four types described here is both compelling and feasible in the short term, some of them having been already partially or fully implemented in Etcher.

How do we embed?

Making a new format is always a big decision, and the kind of ecosystem fracturing it entails is painful, often preventing adoption. In the case of OS images however, there is a loophole we can use to convey this metadata without forcing publishers to use between etcher and all other writers.

Since images are often delivered as archives, we can simply require an additional folder, with a name that users would be inclined to ignore, exist in the archive. That folder could contain all sorts of assets and metadata, leaving the root of the archive to the image. A user can still perform their normal workflow of uncompressing and writing the image with any other procedure, but if the archive is opened with Etcher, then all the extended functionality can be activated.

We could still provide an extension for this format in case publishers explicitly want to prevent use of other writers, e.g. .etch, but the method itself should not require this extension be used.

What next?

This issue has been written as a bit of a time-travel to provide context for the work being done in #707. We welcome comments on the overall approach here, and on the specifics of it in the PR itself.

All this work is still in its early stages and can change significantly until its release, though we do hope to release a first version in the not too distant future.

[dlech]

As a vendor of images for multiple embedded systems, I particularly find the “intended device” feature potentially useful. But I would like to point out that many of our users don’t even know what “Raspberry Pi” or “BeagleBone” are, so having a picture of the intended device would be even more useful than just the name. It has been my experience so far that people will try to, for example, install a BeagleBone image on another device even if I write “THIS IMAGE IS FOR BEAGLEBONE ONLY” in all capital letters on the download page.

[jviotti]

Hi @WasabiFan ,

Sorry for not being explicit enough (I think I described things better in a couple of issues but then become lazy about it). The idea is that the image metadata will live in a centralized metadata repository. By decoupling the metadata from the image itself, we solve a lot of streaming related issues, simplify the whole model, and allow Etcher to become some sort of “marketplace”, where we will provide a way to browser our “catalog” of officially supported images (with metadata) and download & flash directly from it.

This is all being heavily discussed at the moment, but I’ll be creating a stream of new GitHub issues as a new GitHub milestone to kickstart development very, very soon.