Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Too many management API calls

See original GitHub issue

For a while now we’ve been having bad throttling issues, where things kept crashing with errors. After a long debugging session where I went into the code to track where the bad calls are made, I saw that the offending calls are ones that are made to the management API, eg:

https://management.azure.com/subscriptions/[...]/providers/Microsoft.Storage/storageAccounts?api-version=2017-06-01

Later on, we found that it’s no wonder that we’ve been throttled, since apparently the limit for these calls is 100 per 5 minutes, which is easy to reach when we have a bunch of blobs to delete, which we have to do one by one (see Issue #3757). To make things worse, it looks like this limit is at the subscription level, and therefore it’s even easier to reach.

Digging more into the code, it looked like these calls are made to get the storage access key, so we’ve tried providing that (via $AZURE_STORAGE_KEY) and the throttling problems went away. To clarify, this became a bad blocking issue for us since we got to a point where we couldn’t create any new builds.

I also see that the help text claims that --account-name must be used with --account-key, which we haven’t followed.

So: I think that as a good short-term stopgap it would be best to clear the help text and say that it will retrieve the key, but that makes for another API call and therefore it slows things down and it might run into the much-tighter throttling limits on the management service.

A better long-term solution would be to cache the access key itself, possibly together with the cached login information. For example, it would retrieve and cache the key if it’s not there, and if it uses a cached key and that doesn’t work, only then try to re-retrieve the key in case it changed. This would make it possible to lift the requirement of using both the account name and the key, and still have a sane interaction.