LWJGL3: Switching screen mode triggers render() call

Issue details

If i try to toggle fullscreen mode while playing, i get an JVM crash.

Reproduction steps/code

Graphics.DisplayMode primaryMode = Gdx.graphics.getDisplayMode();
Gdx.graphics.setFullscreenMode(primaryMode);

Full Code:

//toggle fullscreen mode
if ((Gdx.input.isKeyPressed(Input.Keys.CONTROL_LEFT) || Gdx.input.isKeyPressed(Input.Keys.CONTROL_RIGHT)) && Gdx.input.isKeyJustPressed(Input.Keys.F)) {
    //toggle fullscreen mode
    if (Gdx.graphics.isFullscreen()) {
        Gdx.graphics.setWindowedMode(this.lastWidth, this.lastHeight);
    } else {
        Graphics.DisplayMode primaryMode = Gdx.graphics.getDisplayMode();
        Gdx.graphics.setFullscreenMode(primaryMode);
    }
}

Version of LibGDX and/or relevant dependencies

Please provide the version(s) affected.

Stacktrace

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffcbadcaceb, pid=10456, tid=1724
#
# JRE version: Java(TM) SE Runtime Environment (8.0_91-b15) (build 1.8.0_91-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.91-b15 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C  [glfw.dll+0xaceb]
#
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
#
# An error report file with more information is saved as:
# D:\Users\Justin\IdeaProjects\island-exploration-rts\hs_err_pid10456.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
AL lib: (EE) alc_cleanup: 1 device not closed

hs_err_pid10456.log

Please select the affected platforms

• Android
• iOS (robovm)
• iOS (MOE)
• HTML/GWT
• Windows
• Linux
• MacOS

[Darkyenus]

This is pretty interesting. What happened is clearly visible from the error log:

...
j  org.lwjgl.glfw.GLFWFramebufferSizeCallbackI.callback(J)V+13
v  ~StubRoutines::call_stub
j  org.lwjgl.system.JNI.invokePPV(JJJIIIII)V+0
j  org.lwjgl.glfw.GLFW.glfwSetWindowMonitor(JJIIIII)V+30
j  com.badlogic.gdx.backends.lwjgl3.Lwjgl3Graphics.setFullscreenMode(Lcom/badlogic/gdx/Graphics$DisplayMode;)Z+134
j  com.jukusoft.rts.gui.GameGUI.render()V+135
j  com.badlogic.gdx.backends.lwjgl3.Lwjgl3Graphics$1.invoke(JII)V+87
j  org.lwjgl.glfw.GLFWFramebufferSizeCallbackI.callback(J)V+13
v  ~StubRoutines::call_stub
j  org.lwjgl.system.JNI.invokePPV(JJJIIIII)V+0
j  org.lwjgl.glfw.GLFW.glfwSetWindowMonitor(JJIIIII)V+30
j  com.badlogic.gdx.backends.lwjgl3.Lwjgl3Graphics.setWindowedMode(II)Z+68
j  com.jukusoft.rts.gui.GameGUI.render()V+113
j  com.badlogic.gdx.backends.lwjgl3.Lwjgl3Graphics$1.invoke(JII)V+87
j  org.lwjgl.glfw.GLFWFramebufferSizeCallbackI.callback(J)V+13
v  ~StubRoutines::call_stub
j  org.lwjgl.system.JNI.invokePPV(JJJIIIII)V+0
j  org.lwjgl.glfw.GLFW.glfwSetWindowMonitor(JJIIIII)V+30
j  com.badlogic.gdx.backends.lwjgl3.Lwjgl3Graphics.setFullscreenMode(Lcom/badlogic/gdx/Graphics$DisplayMode;)Z+134
j  com.jukusoft.rts.gui.GameGUI.render()V+135
j  com.badlogic.gdx.backends.lwjgl3.Lwjgl3Window.update()Z+185
j  com.badlogic.gdx.backends.lwjgl3.Lwjgl3Application.loop()V+101
j  com.badlogic.gdx.backends.lwjgl3.Lwjgl3Application.<init>(Lcom/badlogic/gdx/ApplicationListener;Lcom/badlogic/gdx/backends/lwjgl3/Lwjgl3ApplicationConfiguration;)V+257
j  com.jukusoft.rts.desktop.DesktopLauncher.main([Ljava/lang/String;)V+56
v  ~StubRoutines::call_stub

In your render() method, you have invoked setFullscreenMode, which made window fullscreen, which invoked GLFW resize callback, which in turn called your render() method again (presumably so that resized window would show correctly rendered application), but because the key was still pressed, it invoked setWindowedMode, causing the cycle to repeat, alternating fullscreen and windowed mode, in single call stack. This probably lead to stack overflow, but maybe something else failed first.

While it is indeed a problem caused by incorrect reading of input, for which your fix is correct, it is rather strange that changing windowed/fullscreen should reinvoke application’s render() inside itself. We may want to postpone its effects to the start of next frame (with explicit disabling of render() invocation in framebuffer resize callback), so that render() isn’t called twice, as these methods are usually not reentrant. For example, this crashes after pressing F:

public class Repro extends ApplicationAdapter {

    private SpriteBatch batch;

    @Override
    public void create() {
        batch = new SpriteBatch();
    }

    @Override
    public void render() {
        batch.begin();
        // java.lang.IllegalStateException: SpriteBatch.end must be called before begin.

        if (Gdx.input.isKeyJustPressed(Input.Keys.F)) {
            Gdx.graphics.setFullscreenMode(Gdx.graphics.getDisplayMode());
        }

        batch.end();
    }

    public static void main(String[] args){
        new Lwjgl3Application(new Repro(), new Lwjgl3ApplicationConfiguration());
    }
}

but someone may encounter this as a much subtler bug.

[Darkyenus]

During normal window resizing, which happens through call to GLFW.glfwPollEvents(), normal rendering loop seems to stop, as it is blocked by GLFW.glfwPollEvents() for the duration of window resize dragging (at least on Mac). I don’t know if this blocking is a bug or intended, but resizing the window triggers the resize callback, and the window never shows garbage (LWJGL2 did, IIRC), so it works well.

However, setFullscreen/WindowedMode triggers the resize callback immediately, which, as it is usually called inside render(), causes nested render() invocation. This is likely not desired behavior.

Two ways to fix this come to mind:

1. Disable the resize callback when setFullscreen/WindowedMode is called
2. Postpone the real window mode change so that it happens at the beginning of next frame.

Option 1. feels more intuitive, but could still lead to bugs, as application may not expect screen resolution to change mid-frame. The frame may then still show incorrectly resized frame, as it is rendering for the original screen size. 2. therefore is probably more correct. Without any modifications to resize callback, render() would still get called twice per frame. I don’t know if GLFW expects glfwSwapBuffers to be called inside the callback, but if not, it may be worthwhile to disable the render() call from the resize callback when it happens during window mode change, so that it doesn’t delay the whole operation.