grafana-image-renderer crashing on Centos 8

What happened: Puppet upgraded my grafana-imagine-renderer plugin to the latest version, 3.2.1. When the grafana-server service starts it attempts to start the plugins and when this plugin fails it results in a failure of grafana as it restarts to try and load plugins.

I deploy grafana with puppet in all of my envs. I do not know exactly what version the problem started occurring on but it doesn’t seem to be just the latest version. I tested versions 3.2.0 and 3.1.0 and all had the same issue. I have this identical setup in many envs and found that my other systems (which are RHEL not Cent) had version 2.0.0 of the plugin on them and were working without issues so that is the version I installed on the problematic node and it resolved my issue and I was able to start my grafana service and load the plugin.

This is what I was seeing in my logs (note the systemd):

Oct  7 15:05:18 grafana-server.fqdn.com grafana-server[26730]: t=2021-10-07T15:05:18-0400 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=346 duration=891.432µs
Oct  7 15:05:18 grafana-server.fqdn.com grafana-server[26730]: t=2021-10-07T15:05:18-0400 lvl=info msg="Starting plugin search" logger=plugins
Oct  7 15:05:18 grafana-server.fqdn.com grafana-server[26730]: t=2021-10-07T15:05:18-0400 lvl=info msg="Registering plugin" logger=plugins id=input
Oct  7 15:05:21 grafana-server.fqdn.com grafana-server[26730]: t=2021-10-07T15:05:21-0400 lvl=info msg="Registering plugin" logger=plugins id=grafana-image-renderer
Oct  7 15:05:21 grafana-server.fqdn.com grafana-server[26730]: t=2021-10-07T15:05:21-0400 lvl=info msg="Registering plugin" logger=plugins id=grafana-polystat-panel
Oct  7 15:05:21 grafana-server.fqdn.com grafana-server[26730]: t=2021-10-07T15:05:21-0400 lvl=info msg="Live Push Gateway initialization" logger=live.push_http
Oct  7 15:05:21 grafana-server.fqdn.com grafana-server[26730]: t=2021-10-07T15:05:21-0400 lvl=info msg="Writing PID file" logger=server path=/var/run/grafana/grafana-server.pid pid=26730
Oct  7 15:05:21 grafana-server.fqdn.com systemd[1]: Started Grafana instance.
Oct  7 15:05:21 grafana-server.fqdn.com grafana-server[26730]: t=2021-10-07T15:05:21-0400 lvl=info msg="HTTP Server Listen" logger=http.server address=127.0.0.1:3000 protocol=http subUrl= socket=
Oct  7 15:05:21 grafana-server.fqdn.com kernel: traps: plugin_start_li[26740] trap invalid opcode ip:81cedf sp:7fffd0db07d8 error:0 in plugin_start_linux_amd64[80a000+12b8000]
Oct  7 15:05:21 grafana-server.fqdn.com systemd[1]: Started Process Core Dump (PID 26746/UID 0).
Oct  7 15:05:21 grafana-server.fqdn.com systemd-coredump[26747]: Removed old coredump core.plugin_start_li.989.05f1f509d3414345849eeeb8245c0972.5165.1633631082000000.lz4.
Oct  7 15:05:21 grafana-server.fqdn.com grafana-server[26730]: t=2021-10-07T15:05:21-0400 lvl=eror msg="Stopped background service *rendering.RenderingService" logger=server reason="Failed to start renderer plugin: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
Oct  7 15:05:21 grafana-server.fqdn.com grafana-server[26730]: t=2021-10-07T15:05:21-0400 lvl=warn msg="plugin failed to exit gracefully" logger=plugins.backend pluginId=grafana-image-renderer
**Oct  7 15:05:21 grafana-server.fqdn.com systemd-coredump[26747]: Process 26740 (plugin_start_li) of user 989 dumped core.#012#012Stack trace of thread 26740:#012#0  0x000000000081cedf n/a (/var/lib/grafana/plugins/grafana-image-renderer/plugin_start_linux_amd64)**
Oct  7 15:05:21 grafana-server.fqdn.com systemd[1]: [e-mail]: Succeeded.
Oct  7 15:05:21 grafana-server.fqdn.com grafana-server[26730]: t=2021-10-07T15:05:21-0400 lvl=eror msg="Server shutdown" logger=server error="*rendering.RenderingService run error: Failed to start renderer plugin: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
Oct  7 15:05:21 grafana-server.fqdn.com grafana-server[26730]: *rendering.RenderingService run error: Failed to start renderer plugin: Unrecognized remote plugin message:
Oct  7 15:05:21 grafana-server.fqdn.com grafana-server[26730]: This usually means that the plugin is either invalid or simply
Oct  7 15:05:21 grafana-server.fqdn.com grafana-server[26730]: needs to be recompiled to support the latest protocol.
Oct  7 15:05:21 grafana-server.fqdn.com systemd[1]: grafana-server.service: Main process exited, code=exited, status=1/FAILURE
Oct  7 15:05:21 grafana-server.fqdn.com systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Oct  7 15:05:21 grafana-server.fqdn.com systemd[1]: grafana-server.service: Service RestartSec=100ms expired, scheduling restart.
Oct  7 15:05:21 grafana-server.fqdn.com systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 11.
Oct  7 15:05:21 grafana-server.fqdn.com systemd[1]: Stopped Grafana instance.
Oct  7 15:05:21 grafana-server.fqdn.com systemd[1]: Starting Grafana instance...

What you expected to happen: The plugin to update to the latest version and start properly when called by the grafana service.

How to reproduce it (as minimally and precisely as possible): For me, it seems if I install any of the latest versions of this plugin, I immediately run into issues when restarting the grafana service.

Anything else we need to know?: Checking my system logs, the first occurrence of the systemd core dump from this plugin was today, Oct 7th, 2021 @ 13:02 EST. This was shortly after the 3.2.1 release. I was quite surprised to find that I ran into similar issues while working my way backwards through the previous two versions.

Environment:

• Grafana Image Renderer version: Recognized problem on 3.2.1 but 3.2.0 and 3.1.0 had same issue.
• Grafana version: Version 8.2.0 (commit: d7f71e9eae, branch: HEAD)
• Installed plugin or remote renderer service:

installed plugins:
grafana-image-renderer @ 2.0.0
grafana-polystat-panel @ 1.2.6

• OS Grafana Image Renderer is installed on: Centos 8.4.2105
• User OS & Browser: Chrome
• Others:

[AgnesToulet]

This issue is actually related to this one: https://github.com/grafana/grafana/issues/40162

The fix is similar to what is described here: https://github.com/grafana/grafana/issues/40162#issuecomment-938060240. As a temporary workaround, you can run:

systemctl edit grafana-server

And add in this line to the systemd override file that will be open in your editor:

[Service]
SystemCallFilter=
MemoryDenyWriteExecute=false

The fix for that should be included in 8.2.1.

[gsperry2011]

Thanks @AgnesToulet!