🎉 🎉 🎉 CornerstoneTools 4.0 🎉 🎉 🎉

I’m very proud to announce the release of cornerstoneTools 4.0!

The overarching goal of 4.0 is to improve segmentation support for cornerstoneTools, including:

• Improvements to the labelmap renderer.
• Scalable and efficient labelmap storage.
• A standardised mechanism for building segmentation tools.
• Easier interoperability with other libraries.

Why the need for a major version bump?

In 3.X labelmaps were comprised of a set of binary maps stored as an annotation on each imageId in the global tool state. This got inordinately expensive to store non-overlapping labelmaps, and made it hard to perform 3D operations on the data, as it was disjointed and spread over multiple locations.

Refactoring this to a more user friendly and scalable solution required breaking the existing brush tools, and thus any platform that builds upon them, requiring a breaking change.

Will it be difficult to migrate?

Migrating from 3.X -> 4.0 is a much smaller leap than 2.X -> 3.0 was. If you aren’t using 3.X’s brush tools, migration should be a minimal effort. Following the window of opportunity that comes with a major version bump, we implemented a few additional minor fixes that touch top level API (outlined below), and these are all you will need to care about.

If you were using the brush tools, the main thing that has changed at the surface level is input/output and where to store metadata, the rest is mostly internal and should function out-of-the-box. To help with IO issues, we’ve also authored dcmjs 0.6, which contains an adapter to read/write DICOM SEG with cornerstoneTools 4.0.

Major Changes

Labelmap Renderer

Previously the labelmap renderer worked by converting each segment to a bitmap using the native createImageBitmap functionality in many browsers. This came with two flaws:

• createImageBitmap is an asynchronous process, so you had to store a cache of the previous bitmap whilst the new one was built. You had to do this for each and every segment. Invalidating these bitmaps when you had multiple viewports showing the same scans became a nightmare.
• Safari has no native createImageBitmap implementation. So we wrote a polyfill… which was insanely slow for any reasonably sized image.

We now create a seperate Canvas object the size of the labelmap, fill in its pixels with the desired segment colors and then draw this canvas to the cornerstoneTools canvas, much in the same way cornerstone itself renders its images. This is much faster on all platforms, especially Safari. And no more awkward caching issues.

Outline Rendering

A new feature is the option to render labelmaps with an outline. This outline has a specific line thickness on the canvas, and doesn’t grow when zooming in or out:

The outline renderer has been tested to work well for images up to 512x512, but has performance problems for much larger images (e.g. large DX images). We could fix this with a GPU implementation, but a lot of value lies in cornerstone being a cpu-only framework. There are specific improvements that can drastically help on the scalability of the CPU algorithm, but they aren’t on my own priority list at this time. If you are interested on working on these improvements, please reach out to me and I’d be very happy to talk to about this.

Labelmap storage

In 3.X individual segments are stored on 2D binary Uint8Arrays. This meant that when you had (for example), a freesurfer brain segmentation with 200 segments, the memory requirements became very heavy, or impossible for some platforms (in fact, as we knew this was a limitation, we actually had a hardcoded maximum of 20 segments).

In 4.0 we moved to non-overlapping labelmap based storage. Switching to Uint16Arrays means that for non-overlapping segmentations, we can store 65535 segments on one labelmap (fun fact: 3276750% more compact than before). Overlapping segments are less common in the wild, but are still possible in 4.0, by simply using multiple labelmaps:

This change increases the memory requirements for single segments or overlapping segments slightly, but in both those cases memory was never an issue, so we feel the trade off is more than worth it.

The labelmaps themselves are allocated when a cornerstone stack is drawn on, and are statically allocated to be the size of the entire stack volume. Previously we allocated 2D labelmaps on each frame only when they were painted on, but some experimental work earlier this year proved the benefits of dynamic allocation to be overshadowed by the interfacing pains caused when interoperating with other libraries. Allocated the whole array at once also makes 3D manipulations of the labelmap much easier to perform.

Labelmap data is now all stored in the segmentation module. Check the module’s new documentation for a practical useability guide and how to interact with its API.

New Tools

We have formalised a way to build tools by composition that take some manner of user input, and use it to apply some strategy to modify a labelmap. By using the cornerstoneTool’s mixin pattern, we can reuse delineation mechanisms for segmentation tools that do different things. A detailed explanation is available in the docs.

That being said, this wouldn’t be a fun update if the changes were purely architectural, so here is a plethora of cool new segmentation tools:

Spherical Brush Tool

Scissor Tools

Correction Scissor Tool

Undo/Redo for labelmaps

History is now saved for each labelmap by computing the diffs of each brush stroke/segmentation tool application:

These actions can be bound to UI/keybinds through a simple API:

const { setters } = cornerstoneTools.getModule('segmentation');

setters.undo(element);
setters.redo(element);

Where element is the cornerstone element you wish to perform the time travelling on. By default the undo/redo will apply to the active labelmap, however you can use a different labelmap by specifying its index as a second parameter: setters.undo(element, 2);.

Undo/redo only applies for labelmaps for now. However, perhaps in the future we will have a global history that includes modifications to annotation tools as well.

Minor Changes

• Tools that modify the labelmap now trigger the new LABELMAP_MODIFIED event, tool authors should trigger this event when their segmentation tool completes it action.

Naming Changes

• cornerstoneTools.import has been renamed cornerstoneTools.importInternal, so that it does not use a reserved word. However, cornerstoneTools.import has been left as an alias.
• The importInternal library now correctly exports convertToVector3, which was previously exposed as convertToVectro3.
• The importInternal library has been updated to expose all the new functionality introduced.
• FreehandMouseTool has been renamed FreehandRoiTool and FreehandSculpterMouseTool has been renamed FreehandRoiSculptorTool, to be more in line with the naming of the other ROI tools. We also realised putting the interaction type (mouse) into the toolname (which is top level API) was a terrible mistake, as we couldn’t change it once we introduced touch controls for these tools. We will not put the supported interaction types in toolnames anymore.
• The brush module has been renamed the segmentation module, as it concerns more segmentation tooling than just brush tools.

getModule API

Previously modules could only be fetched by e.g. cornerstoneTools.store.modules.brush, which made them feel like an escape hatch that shouldn’t be touched. Modules are completely intended to be interacted with by consumers, and as such are now exposed like so:

const {
  getters,
  setters,
  configuration,
  state
} = cornerstoneTools.getModule('segmentation);

doneMovingCallback is explicitly tool API

In response to PR #915, doneMovingCallback is no explicitly tool API. Previously the doneMovingCallback of tools was stored on the tools options, which can be changed by the consumer. As such doneMovingCallback looked like consumer-level API, however changing its functionality would break some tools. It is now explicitly tool API and passed in separately to helpers that require it. options.doneMovingCallback is deprecated but still exists for now, and will be executed along with a deprecation warning.

[swederik]

Hoooray! Now integration with https://github.com/Kitware/vtk-js will also be easier @floryst 😃 Nice work @JamesAPetts!

[kevinleedrum]

👏👏👏 Excellent work!