Rate limiting of NS1 provider

Getting on really well with OctoDNS, checking out a number of providers whilst planning a DNS migration. But I have a problem with the NS1 provider because it gets rate limited. I don’t think I am hitting it that hard (my dataset contains 1022 records across 11 zones).

I have been able to get my dataset into NS1 but it’s taken me a few hours of retrying to do so. This is in contrast to other providers I have also tested which work pretty-much instantaneously.

Having discussed this with NS1, this is what I was told…

Each API call that makes a change to a zone or record causes a large number of messages to be generated internally in order to extremely quickly propagate changes to our edge nodes and is computationally expensive. In order to manage this computational cost and to protect the platform we have a token based ratelimitting system in place.

First let me break down how the ratelimitting system works: Basically for each HTTP method (PUT, GET, POST, DELETE) you have an independent bucket of tokens and an independent replenishment period for said bucket. Whenever you make an API request a token is consumed from the appropriate bucket. In your case the API rate limits by method are:

GET
< X-RateLimit-Limit: 900
< X-RateLimit-Period: 300
POST
< X-RateLimit-Limit: 300
< X-RateLimit-Period: 300
PUT
< X-RateLimit-Limit: 200
< X-RateLimit-Period: 300
DELETE
< X-RateLimit-Limit: 100
< X-RateLimit-Period: 200

DELETE has a bucket of 100 tokens meaning you can instantaneously send 100 delete requests without being rate limited. Those 100 tokens continuously replenish and it takes 200 seconds to fully replenish (approx 0.5 tokens/second). Whereas GET has 900 tokens in it’s bucket and a replenishment period of 300 seconds (approx 3 tokens/sec). This is approximately correlated to how computationally expensive each call is for us. The current status of your token buckets are returned in the response header from the API and our recommendation is to incorporate those headers as part of your rate limiting.

This means that in order to make changes to 900 records you can instantaneously burst through 300 changes, then apply the next 600 changes over the course of 600 seconds (10 minutes).

The problem with OctoDNS is that it is sending an individual PUT call to create each and every record on the platform and it is doing so without any ratelimitting. The ideal thing for it to do would be for it to use the zone file upload endpoint (docs: https://ns1.com/api#put-import-a-zone-from-a-zone-file) and supply an appropriately formatted plain text file. This is a single API call that would upload an entire zone which is much better for both the user and our platform. Another possibility is enforcing a rate limit within Octo when it is interacting with the NS1 API. In the case of zone/record creations this would be 1.5 PUTs per second (5,400 creations per hour). Another possibility yet would be to do the zone upload independently of Octo (using the zonefile upload endpoint) and then sync’ing with OctoDNS afterwards (I believe this is possible using Octo).

I’m just posting for interest / information and possible future enhancement to the NS1 provider.

[rupa]

I got the go-ahead to work on this yesterday, so as of now it’s my priority and I hope to have a PR ready within the next couple of days.

[rupa]

Hey, yeah, I can confirm we have some improvements to handling rate-limiting in the works, and we’re planning to make a PR to octodns about that pretty soon. The changes are mostly around preventing 429s from happening, but things can still take a while with large numbers of zones and records due to the number of API hits.

The zone import endpoint does seem like it could be useful to cut down on some of the traffic around initial setups, I’m going to look into it.