question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG] grafana 8 webhook integration

See original GitHub issue

Using the latest major release of grafana, i’m not able to use the existing webhook integration.

I receive the following error message in zulip-log:

2021-10-28 15:58:51.243 ERR  [zulip.zerver.webhooks] 'ruleName'
Traceback (most recent call last):
  File "./zerver/decorator.py", line 334, in _wrapped_func_arguments
    return view_func(request, user_profile, *args, **kwargs)
  File "./zerver/lib/request.py", line 390, in _wrapped_view_func
    return view_func(request, *args, **kwargs)
  File "./zerver/webhooks/grafana/view.py", line 57, in api_grafana_webhook
    rule_name=payload["ruleName"],
KeyError: 'ruleName'
Stack (most recent call last):
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/wsgi.py", line 133, in __call__
    response = self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 130, in get_response
    response = self._middleware_chain(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "./zerver/middleware.py", line 298, in __call__
    return self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django_otp/middleware.py", line 32, in __call__
    return self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/two_factor/middleware/threadlocals.py", line 19, in __call__
    return self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "./zerver/lib/request.py", line 390, in _wrapped_view_func
    return view_func(request, *args, **kwargs)
  File "./zerver/decorator.py", line 352, in _wrapped_func_arguments
    unsupported_event=isinstance(err, UnsupportedWebhookEventType),
  File "./zerver/decorator.py", line 299, in log_exception_to_webhook_logger
    webhook_logger.exception(summary, stack_info=True)
2021-10-28 15:58:51.245 ERR  [zerver.middleware.json_error_handler] Traceback (most recent call last):
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/srv/zulip-venv-cache/3738dadf748d3baa2eb0371a83c912149a50d964/zulip-py3-venv/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "./zerver/lib/request.py", line 390, in _wrapped_view_func
    return view_func(request, *args, **kwargs)
  File "./zerver/decorator.py", line 354, in _wrapped_func_arguments
    raise err
  File "./zerver/decorator.py", line 334, in _wrapped_func_arguments
    return view_func(request, user_profile, *args, **kwargs)
  File "./zerver/lib/request.py", line 390, in _wrapped_view_func
    return view_func(request, *args, **kwargs)
  File "./zerver/webhooks/grafana/view.py", line 57, in api_grafana_webhook
    rule_name=payload["ruleName"],
KeyError: 'ruleName'

2021-10-28 15:58:51.247 ERR  [django.request] Internal Server Error: /api/v1/external/grafana

The grafana log tells me:

t=2021-10-28T15:56:32+0000 lvl=info msg="level=error msg=\"Missing receiver\"" logger=alerting.notifier.webhook
t=2021-10-28T15:56:32+0000 lvl=info msg="level=error msg=\"Missing group labels\"" logger=alerting.notifier.webhook

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:6
  • Comments:11 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
tilpnercommented, Nov 29, 2021

Using the alertmanager integration seems to work… a bit. The “graph” link leads to the alert editor, but it’s better than no alerts. 😃

0reactions
timabbottcommented, Nov 15, 2022

This was fixed in #22882.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Configure the webhook notifier | Grafana documentation
WeCom · Click the “…” in the top right corner of a group chat that you want your alerts to be delivered to...
Read more >
Webhook not working - Alerting - Grafana Community
Configure webhook to get alerts. How are you trying to achieve it? Following the steps here. What happened? The notification doesnt work, see ......
Read more >
Configure outgoing webhooks for Grafana OnCall
In Grafana OnCall, navigate to Outgoing Webhooks and click + Create. This is also the place to edit and delete existing outgoing webhooks....
Read more >
Grafana 8 Alert not working as expected
I have upgraded Grafana from 7.4 to 8.1.5 with the new alerting mechanism. The 'new alerting' not working properly (also not so intuitive...
Read more >
Grafana alerting error
t=2017-04-29T00:16:04+0000 lvl=info msg="Sending notification" logger=alerting.notifier type=webhook id=0 isDefault=false ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found