Adding Rate Limits

Fantastic work on this codebase. Since I’ve never done a pull request, I wanted to drop this potential improvement here as there is a lesser chance I’ll bork something.

Background: So my bot does a ton of requests in the background to get historical data and I was having issues getting banned, so I had created a temporary hack that I had to merge each time you updated.

Today, I figured it would be worth just cleaning up the code so others can benefit.

Refactored Request function to include rate limiting.

I’ve refactored the body of each of the request functions into one core function and added rate limiting.

	let callWeight, requestCount = 0, requestInitTime = Date.now();

	const makeRequest = function(opt, done){
		// If our last requestInitTime was more than a minute ago. Reset our count.
		if(Date.now() - requestInitTime > 60*1000){
			requestCount = 0;
			requestInitTime = Date.now();
		}

		if(opt.url.indexOf('depth') > -1){
			callWeight = 10;
		} else if(opt.url.indexOf('historicalTrades') > -1){
			callWeight = 5;
		} else {
			callWeight = 1;
		}

		options.log(opt.caller, requestCount, opt.url);

		if(requestCount < 1000 || opt.caller === 'signedRequest'){
			requestCount += callWeight;	
			request(opt, function(error, response, body) {
				if ( !done ) return;
				if ( error ) {
					options.log(Error(error));
					return done( error, {} );
				}

				if ( response && response.statusCode !== 200 ) {
					return done( response, {} );
				}
				
				return done( null, JSON.parse(body) );
			});
		} else {
			console.log('delaying:', opt.url)
			setTimeout(() => {
				makeRequest(opt, done);
			}, Date.now() - requestInitTime + 60*1000 );
		}
	}

Refactored requests:

Then I’ve refactored each of the “requests” to be as follows:

	const publicRequest = function(url, data, callback, method = 'GET') {
		if ( !data ) data = {};
		let opt = {
			caller: 'publicRequest',
			url: url,
			qs: data,
			method: method,
			timeout: options.recvWindow,
			agent: false,
			headers: {
				'User-Agent': userAgent,
				'Content-type': contentType
			}
		};
		makeRequest(opt, callback)
	};

The only thing that I added to each request was a “caller” to the opts so I could debug better.

Assumptions

I’ve set the rate limit to 1000 and then defer all requests that aren’t signed ones. The logic here is a bit weak as others may issue 200+ signed requests in a minute though I know mine won’t ever encounter that edge case.

Tested

I’ve tested this quiet thoroughly and can download data on a huge amount of markets without getting banned.

[bkrypt]

Thanks for sharing this @nickreese. I also handle rate limiting in a similar way in my wrapper of this library, and had it on my “if I had more time” list, to eventually add into the library itself.

I’m not 100% when @jaggedsoft will implement and merge this, but I’m sure if you opened a PR it would be much sooner, as it would require a lot less of his time. 👍

A couple of suggestions from my side though.

1. The library should get its limit from a call to exchangeInfo.
2. The depth endpoint should take into account its limit argument and set the weight as 1, 5, 10 appropriately.
3. signedRequest calls should not be segregated from the limiting.
4. the delaying setTimeout should be clamped to its interval window defined by requestInitTime . ie. if I’ve reached my limit at 55 seconds, the next request should be delayed 5 seconds, and not 65 seconds.

// Results in a (60000 + <the difference>)ms delay.
setTimeout(() => {}, Date.now() - requestInitTime + 60*1000 );

// Results in a <the difference>ms delay or it'll be scheduled
// immediately in the next event loop if <= 0.
setTimeout(() => {}, (60 * 1000) - (Date.now() - requestInitTime));

[jaggedsoft]

Thank you for your contribution! Excellent. I was wanting to add this at some point. I was originally wanting to do this under a flag that would default to true. But with your implementation, I don’t see any reason this shouldn’t just be merged in to the main codebase. Looks like it will just work