Artifacts compared to Lua version

Coming back to this repo after a long break, interested in developing this further…

I was just comparing this implementation to the original Lua version and noticed something I hadn’t before. The two produce almost identical results, but the PyTorch version appears to produce very subtle artifacts.

The following is an example, using Hokusai as the style image. On the left side is neural-style (lua), on the right side is neural-style-pt.

Notice in scattered places the presence of high-frequency discolorations, often in almost checkerboard-like patterns. These do not appear in the Lua version. If you zoom in on a few parts of the neural-style-pt versions, you can see them clearly. Notice the pink and green checkers.

This generally happens consistently for any combination of content and style images, although for some style images the artifacts are more obvious. Sometimes obvious discolorations appear, other times they are smaller, giving the output an almost grainy appearance. The artifacts can be reduced by increasing -tv_weight but at the expense of content/style reconstruction, and even then it’s still visible.

I tried fixing it a few ways. Clamping the image between iterations (not just at the end) didn’t fix it. I tried playing with the TVLoss module. For example, changing

self.loss = self.strength * (torch.sum(torch.abs(self.x_diff)) + torch.sum(torch.abs(self.y_diff)))

to an L2-loss, i.e.

self.loss = self.strength * (torch.sum(torch.pow(torch.pow(self.x_diff, 2)) + torch.sum(torch.pow(self.y_diff, 2)), 2))

also did not get rid of the artifact (I tried this because my reading of the TVLoss formula is that they use L2-loss not absolute values. But I’m not sure this makes a big difference.

The artifact is very subtle, but I’m hoping to fix it, as I’d like to produce more print-quality images in the future, and multi-stage or multi-scaled techniques on top may amplify the artifact. I wonder if you have any idea what might be causing this or what could potentially fix it.

[Sankyuubigan]

Hi guys. I am not very good at your code, could you please provide the code of the neural_style.py file with ready changes maxpool2d or genekogan’s tv optimization? How to use this code?

[genekogan]

One hacky idea I have that could help balance the tradeoff between checkerboard artifacts/high-freq noise (which seem to reduce with increased TV regularization) and muddy regions (which reduce with decreased TV regularization) would be to modify the TVLoss by first multiplying it element-wise with a saturation map (or something similar) of the image before summing it all together.

So instead of:

 self.loss = self.strength * (torch.sum(torch.abs(self.x_diff)) + torch.sum(torch.abs(self.y_diff)))

Something like:

X = torch.abs(self.x_diff)) + torch.sum(torch.abs(self.y_diff)
self.loss = self.strength * (torch.sum(z))

xd = torch.mul(S, torch.abs(self.x_diff))
yd = torch.mul(S, torch.abs(self.y_diff))
self.loss = self.strength * (torch.sum(xd) + torch.sum(yd))

where S is some measure of the local saturation around that pixel (or rather desaturation, since you want to increase the TV penalty around the desaturated regions). For example, S(x,y) could be inverse of the standard deviation in the m*n square block centered at x,y.

Or even simpler would be to use L2 sum instead of L1, i.e. square/raise each element in torch.abs(self.x_diff) and torch.abs(self.y_diff) to some power to penalize really big differences between pixels.

I’m not sure if there’s much to gain from trying to optimize TV noise, as the effect is pretty subtle, and maybe the artifacts aren’t even my biggest problem anymore, but some untrained/half-baked food for thought.