Production - [Alerting] Service Hook Deliveries Alert
See original GitHub issue💔 Metric state changed to alerting
We detected service hooks that are failing to deliver at least half of their message in the last hour. The most usual cause is a rotation on the storage account key. Action points:
- Verify which service hooks are failing, this should be done on Service hooks internal and Service hooks public. To achieve that you can see the 7 day status or open the history and check the most recent attempts. Please take into account that the history is in ascending order.
- For the failing hook look into the ‘Action’ section in which you are going to find something like this
Queue Account helixexecution, Queue testreporting
- On the Azure portal look for the ‘Queue Account’, that should be a storage account. For example in this case we should search for
helixexecution
- Inside the storage account navigate to ‘Access key’, press ‘Show keys’ and copy the ‘Key’ value of the ‘key1’
- To update the secret: return to the service hook page, select the service hooks that need to be fixed, click on the ‘pen button’ (Edit the selected subscription). When the ‘Edit service hooks subscriptions’ pops up click on ‘Next’ and paste the ‘Key’ value on ‘Storage account key’. As a last step click on ‘Test’ to verify that it is working and then click on ‘Finish’
- Repeat the previous steps for all the Service hook that have failures
- Succeeded Rate {ServiceHookName=BuildMonitorWebhook} 0
@dotnet/dnceng, please investigate
Automation information below, do not change
Grafana-Automated-Alert-Id-b30a51409b1d461e8538ca9c6a1c4c79
Issue Analytics
- State:
- Created a year ago
- Comments:12 (9 by maintainers)
Top Results From Across the Web
Custom HTTP Delivery
Custom HTTP Alert Delivery Methods (i.e. Webhooks) allow you to send alert notification data to an external system via HTTP GET or POST ......
Read more >Write and deploy Cloud Monitoring alert notifications ...
A tutorial for writing and deploying customized Cloud Monitoring alert notifications to third party services.
Read more >Notification webhook payloads—ArcGIS Monitor
Once a webhook is triggered, a payload is delivered to the specified webhook URL. Each webhook payload follows a similar JSON schema, containing...
Read more >Receive change notifications through webhooks
Change notifications can be delivered via different channels, including webhooks and Azure Event Hubs. This article walks you through how to ...
Read more >Set up alerts and notifications | Apigee Edge
You can set up alerts at the environment, API proxy or target service, or region level. When an alert is triggered, you will...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I already fixed it.
💚 Metric state changed to ok
Go to rule