Fork implementing multi-region spatial control

First of all, this is a great repo! It seems a bit faster and more memory efficient than the original lua-based neural-style.

I’ve made a fork of this repo trying to add masked style transfer as described by Gatys, and going off of the gist you wrote for the lua version.

I’ve almost got it working, but my implementation is suffering from two bugs. The first is that, testing with two style images and segmentations, my implementation seems only to get gradients for the first mask but not the second.

So for example, the following command:

python neural_style.py -backend cudnn -style_image examples/inputs/cubist.jpg,examples/inputs/starry_night.jpg -style_seg examples/segments/cubist.png,examples/segments/starry_night.png -content_seg examples/segments/monalisa.png -color_codes white,black

produces the following output:

where the first style (cubist) and corresponding segmentation get good gradients and works in the mask provided, but the second mask (starry night) has little or no gradient signal.

By simply swapping the order of the style images, as in:

python neural_style.py -backend cudnn -style_image examples/inputs/starry_night.jpg,examples/inputs/cubist.jpg -style_seg examples/segments/starry_night.png,examples/segments/cubist.png -content_seg examples/segments/monalisa.png -color_codes white,black

I get the opposite effect where only the starry night style works and the cubist style in its mask is not there.

I have been trying to debug this, checking the masks, and everything seems right to me, and I can’t figure out the problem. This was almost a pytorch mirror of what you made in your gist, which does appear to work fine. I’m not sure if there’s some typo I’m missing or something deeper.

Additionally, loss.backward() without keeping the gradients with retain_graph=True produces a runtime error (RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.), which makes me think I setup the graph wrong.

If you are able to see what I’m doing wrong so that we can fix it, I’d love to see this implemented in PyTorch. I think it would be a really nice addition to the repo.

[genekogan]

I just updated my histogram_loss branch with your Histogram loss module, and made it support masking in the same way the Style Loss does. I also added a covariance loss option for the normal StyleLoss module. So far with limited tests, histogram loss does not seem to do much to fix the blending problem. I’m going to do some tests combining the style loss parameters and see if I can either improve that issue somehow or at least get the general style transfer to look nicer.

[genekogan]

Another note about transitional blending problem. I e-mailed Leon Gatys about it and he suggested that since covariance loss seems to reduce the smudging effect more than Gram, to try using covariance loss on the lower layers (where the differences between the styles are greatest) and using Gram on the higher layers to preserve better style reconstruction. Going to try that next.