Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

rateLimits

See original GitHub issue

I was wondering how you come up with rateLimit values for given exchanges. They are usually > 1000 (as I understand this is in ms), but on exchanges which publish their limits its more like 1000/min and 1,000,000/day which comes to about 87ms rateLimit instead.

I would also like to spark conversation about implementing rateLimit natively into all libs so that we do not have to care about ddos protection in our code.

This is what I use now in my code:


let _EXCHANGES_API_LAST_CALLED = {};

function APICallTimeLog(exchange_id){
    _EXCHANGES_API_LAST_CALLED[exchange_id] = Date.now();
}

async function APISleep(exchange_id){

    let sleep = (ms) => new Promise (resolve => setTimeout (resolve, ms));

    let wait_time = APICallWait(exchange_id);

    if(wait_time)
        await sleep(wait_time);

    APICallTimeLog(exchange_id);

}

function APICallWait(exchange_id){
    if(!_EXCHANGES_API_LAST_CALLED[exchange_id])
        return 0;

    let last_called_time = parseInt(_EXCHANGES_API_LAST_CALLED[exchange_id]);

    let wait = last_called_time + config.exchanges[exchange_id].rateLimit - Date.now();

    if(wait < 0)
        return 0;

    return wait;
}

and then during API communications:


            let promise = (async () => {

                await APISleep(exchange_id);

                return await exchange.fetchTickers();
            })();

would be nice if all this were under the hood implemented. I am willing to look into this if there is interest, thanks for your opinion

Issue Analytics

State:
Created 6 years ago
Comments:11 (10 by maintainers)

Top GitHub Comments

3reactions

kroitorcommented, Jan 21, 2018

@michnovka here’s the plan:

check out the entire history, from how it all started to where it is now:

initially, we had a naive implementation like yours, that sleeps regular intervals between requests
it was slightly different from real rate limits that don’t limit the timing between requests and only limit the amount of requests per a period of time, thus allowing bursts
@samholt proposed the more advanced tokenBucket implementation in Python, here https://github.com/ccxt/ccxt/pull/373
it was ported to JS, the PHP implementation still remains semi-naive (to be reworked)

check out how current rate-limiter works:

(we are working on the js/base and some files may be moving around in the meantime, don’t panic 😉)

check out related issues and other important info:

you probably already know this: https://github.com/ccxt/ccxt/blob/master/CONTRIBUTING.md
ask questions if any )
your contributions and pull-requests are welcome 😉

Let me know if it answers your question )

0reactions

kroitorcommented, Mar 3, 2018

@Codewyvern

Does that mean the rate limit is fully automatically handled by the library

Yep.

do I still need to call it manually like here?

Nope. You should either do it manually, or enable and use the built-in rate-limiting in the library, but not both.

The code after 'enableRateLimit': true is:

// fetch trades for all markets one by one sequentially
// while still staying below the rate limit:

;(async () => {
    for (symbol in exchange.markets)
        console.log (await exchange.fetchTrades (symbol)); 
}) ()