Problem with multisample postprocessing

This ticket is the result of the discussion in Discord and its conclusions.

There appears to be an inconsistence within JME postprocessing stack, related to multisampling, that, according to my preliminary findings, encompasses such problems as:

1. Some of the stock shaders are not built to support multisampling (for example, FXAA), thus when multisampling is enabled, these shaders fail.
2. The failure looks like replacing the screenbuffer texture with some randrom texture from the loaded ones, thus effectively breaking the rendering pipeline.
3. There’s no warning or error that notifies the user that a particular shader is not compatible with multisampling and cannot be used.
4. There’s no documentation notice about what shaders allow that usage or not.
5. Unaware contributors make shaders that are incompatible with multisampling. That could have been avoided if the contribution guide included the relevant section for shaders on how to make them multisampling-compatible.
6. The problem can be silently “eliminated” by preceeding the line of post effects of the affected FilterPostProcessor with one of the shaders that was by chance made compatible with multisampling. That is not a real solution, however, it’s currently widely adopted among the community on a level of a belief that a lucky sequence of postprocessing effects will work and the user just needs to find it by randomly changing the order of the filters in the postprocessing stack until it works.

Reproducing the problem:

The simplest reproduction of the problem can be made by creating a FilterPostProcessor within any JME project (for example, TestPBRLighting or a barest project with a box and a sky sphere and some textures), setting the number of samples on application settings to, for example, 4, setting the number of samples on the FilterPostProcessor to the same amount, and assigning two filters to it:

1. FXAAFilter
2. ToneMapFilter

which will lead to the viewport screenbuffer to be replaced with the texture of the environment map:

while the reverse combination of the filters will seemingly work:

The issue is also reproduced and confirmed by Oxplay on his machine.

The statement of this issue implies that preceeding the FilterPostProcessor chain of effects with even a passthrough filter that simply does nothing except for treating the possibility of multisampling correctly does fix the symptom is confirmed.

This project https://github.com/noncom/jme-multisampling-stopgap contains such a proof of a passthrough filter that can be used at the start of the chain with any number of following filters that don’t support multisampling and eliminate the glitch.

Take note that that filter also requires

 #extension GL_ARB_texture_multisample : enable

which in turn requires specifying GLSL150 version requirement in its j3md.

However, such an “approach” is not a solution, since the problem clearly lies within the JME pipeline or FilterPostProcessor pipeline.

As of now I did not yet get into how the pipelines work inside so I don’t know why exactly this “approach” fixes the texture glitch. My current guess is that such a filter simply converges the multiple texture samples into one, thus providing a sampler2D as its output, that is later nonchalantly used by the following filters. If this is true, then the following filters are already silently receiving the color data collapsed to a single sample and might not always be desirable? This must be a problem?

I am not sure how this could have been missed until now, but this looks like a huge gap in the solidity of documentation and/or rendreing/postprocessing stacks and requires some attention.

[jayfella]

From what I know based on the conversation on discord:

If multisampling is enabled, the screen texture is passed to the multisampling texture:

uniform sampler2DMS m_Texture;

and at this stage, the unsampled texture is NULL.

uniform sampler2D m_Texture;

Therefore until a filter reads the sampled texture, the unsampled texture is always null.

If multisampling is enabled and no filter reads the multisampled texture, the FPP will return a null texture - which will result in the first(?) texture that was loaded being displayed (usually the BitmapFont texture - or in the OP’s case - the env map).

The problem can be patched by putting a Filter at the front (zero index) of the list that simply reads from the multisampled texture and outputs it straight back - i.e. a passthrough filter.

[stephengold]

• I reproduced the issue by modifying TestPBRLighting in master branch as you suggested.
• I pushed a change to TestPBRLighting to set the number of samples based on the rendering context, which should make it slightly easier for others to reproduce the issue. (They still must use the setting dialog and uncomment the FXAA filter.)
• I confirmed that disabling MSAA or swapping the order (of the FXAA and ToneMap filters) or inserting a MultiSamplingFix (before the FXAA) works around the issue.

I’m not very familiar with how filters work. Perhaps the solution is to require that all filters support MSAA? Or to recognize which ones don’t and coddle them appropriately.