Request for High level details to implement brush&erase functionality

My actions before raising this issue

• Read/searched the docs
• Searched past issues

Expected Behaviour

Current Behaviour

Possible Solution

Steps to Reproduce (for bugs)

Context

I am looking for details to implement Brush and erase functionality to create segmentation masks (At this moment I am not looking into watershed, superpixel etc implementation). This would ease creation of segmentation masks for very irregular objects appearing in aerial view images, microscope images, medical image segmentation. Some other functionality might require it as well, #606 #772. It seems to be also part of the project Integration of OpenCV.js tools here.

I am not a web developer (i.e. have no javascript experience, but python,c,c++), but are interested in looking into this as my time allows. My request is project admins’ thoughts, specific details and advice on implementing brush&erase functionality.

Your Environment

• Git hash commit (git log -1):
• Docker version docker version (e.g. Docker 17.0.05):
• Are you using Docker Swarm or Kubernetes?
• Operating System and version (e.g. Linux, Windows, MacOS):
• Code example or link to GitHub repo or gist to reproduce problem:
• Other diagnostic information / logs:
  Logs from `cvat` container

Next steps

You may join our Gitter channel for community support.

[bsekachev]

@juanmed @nmanovic

Hi, Implementing only brush and erase in JS client is easy and it doesn’t require using OpenCV. Simple example of brush can be found here. As you can see, only 20 lines of code. Of course it’s very simplified version, but I do not see any difficulties to implement a full version. It’s a question of time only.

The main problem I see is actually converting client masks to polygons and back. I expect to came across “underwater rocks” here. And I am not sure, that implementing the feature on client only is a right way.

Now we have buttons “add rectangle”, “add polygon”, etc. I would suggest to add button “add mask” and consider this mask as an object, just drawn with a brush. So, a user presses the button, selects a label and starts drawing, being able to switch between brush/eraser and to choose a brush/eraser size.

One more important task here is to be able assignee mapping between labels and colors. It is important not only for masks actually. I don’t expect it is difficult. Just to add a field to database and change label constructor (add a color picker)

Later we can add advanced features like watershead from OpenCV.

I am not sure if you need to be acquainted with technical details of implementation right now. Let me know if you need.

[nmanovic]

@bsekachev , could you please share your ideas?

From my perspective I will recommend to prototype the feature in UI only. Let’s add these tools and convert to polygons internally. You need to convert masks to polygons only when somebody saves annotations.

Read https://github.com/opencv/cvat/blob/develop/CONTRIBUTING.md guide. Unfortunately in this case you need to implement the whole feature in js/ts/css.