Production - [Alerting] Service Hook Deliveries Alert

💔 Metric state changed to alerting

We detected service hooks that are failing to deliver at least half of their message in the last hour. The most usual cause is a rotation on the storage account key. Action points:

1. Verify which service hooks are failing, this should be done on Service hooks internal and Service hooks public. To achieve that you can see the 7 day status or open the history and check the most recent attempts. Please take into account that the history is in ascending order.
2. For the failing hook look into the ‘Action’ section in which you are going to find something like this Queue Account helixexecution, Queue testreporting
3. On the Azure portal look for the ‘Queue Account’, that should be a storage account. For example in this case we should search for helixexecution
4. Inside the storage account navigate to ‘Access key’, press ‘Show keys’ and copy the ‘Key’ value of the ‘key1’
5. To update the secret: return to the service hook page, select the service hooks that need to be fixed, click on the ‘pen button’ (Edit the selected subscription). When the ‘Edit service hooks subscriptions’ pops up click on ‘Next’ and paste the ‘Key’ value on ‘Storage account key’. As a last step click on ‘Test’ to verify that it is working and then click on ‘Finish’
6. Repeat the previous steps for all the Service hook that have failures

• Succeeded Rate {ServiceHookName=BuildMonitorWebhook} 0

Go to rule

@dotnet/dnceng, please investigate

Automation information below, do not change

Grafana-Automated-Alert-Id-b30a51409b1d461e8538ca9c6a1c4c79

[AlitzelMendez]

I already fixed it.

[dotnet-eng-status[bot]]

💚 Metric state changed to ok

We detected service hooks that are failing to deliver at least half of their message in the last hour. The most usual cause is a rotation on the storage account key. Action points:

1. Verify which service hooks are failing, this should be done on Service hooks internal and Service hooks public. To achieve that you can see the 7 day status or open the history and check the most recent attempts. Please take into account that the history is in ascending order.
2. For the failing hook look into the ‘Action’ section in which you are going to find something like this Queue Account helixexecution, Queue testreporting
3. On the Azure portal look for the ‘Queue Account’, that should be a storage account. For example in this case we should search for helixexecution
4. Inside the storage account navigate to ‘Access key’, press ‘Show keys’ and copy the ‘Key’ value of the ‘key1’
5. To update the secret: return to the service hook page, select the service hooks that need to be fixed, click on the ‘pen button’ (Edit the selected subscription). When the ‘Edit service hooks subscriptions’ pops up click on ‘Next’ and paste the ‘Key’ value on ‘Storage account key’. As a last step click on ‘Test’ to verify that it is working and then click on ‘Finish’
6. Repeat the previous steps for all the Service hook that have failures

Go to rule