binance 429 Too Many Requests - enableRateLimit: true

I’m trying to setup async batch fetching of OHLCV values and ran into the over maxCapacity error. I have enableRateLimit set to True and was hoping that the rate limiter would dynamically queue requests rather than just throw if I spawn too many. Seems it does the later?

EDIT: It definitely does the former, just was hitting the hard coded capacity of the throttler itself

I intentionally set up the fetch to be OOB for the crypto histories (20 years of history) to see the behavior of the API when I fetched data before the exchange. Though I anticipate that as I branch out the dimensionality of the fetch with valid times I’ll encounter the same error.

System Info

• OS: macOS Monterey (12.1)
• Programming Language version: Python 3.9.10
• CCXT version: 1.72.64

Code:

import ccxt.async_support as ccxt
from datetime import datetime, timedelta, timezone
import pandas as pd
import asyncio

EXCHANGES = ['binance', 'ftx', 'aax']
COIN = ['BTC', 'ETH']
LIMIT = 500

async def fetch_history_data(start_time: datetime):
    start_time = start_time.replace(second=0, microsecond=0)

    intervals = pd.date_range(start_time, datetime.now().replace(second=0, microsecond=0), freq='1min')

    reqs = {}
    exchanges = []

    # for exchange in ccxt.exchanges:
    for name in EXCHANGES:
        # Turn on rate limiting
        exchange = getattr(ccxt, name)({ 'enableRateLimit': True })
        exchanges.append(exchange)

        if not exchange.has["fetchOHLCV"]:
            print(f"Skipping {name}")
            continue
        if exchange.timeframes is None:
            print(f"Skipping {name}")
            continue
        if "1m" not in exchange.timeframes:
            print(f"Skipping {name}")
            continue

        if name not in reqs:
            reqs[name] = {}
        
        await exchange.load_markets()
        for coin in COIN:

            # Check fiat currency as well as USDT pricing
            symbol = f'{coin}/USD'
            if symbol not in exchange.symbols:
                symbol = f'{coin}/USDT'
                if symbol not in exchange.symbols:
                    continue

            data_reqs = []
            for idx in range(0, len(intervals), LIMIT):
                data_reqs.append(exchange.fetch_ohlcv(symbol, "1m", since=int(intervals[idx].replace(tzinfo=timezone.utc).timestamp() * 1000), limit=LIMIT))

            data = await asyncio.gather(*data_reqs)
            history = []
            for chunk in data:
                history += chunk
            
            reqs[name][coin] = history

    for exchange in exchanges:
        await exchange.close()
    
    columns = ['timestamp', 'open', 'high', 'low', 'close', 'volume']
    df = pd.DataFrame(columns=(['exchange', 'coin'] + columns))

    for e, cs in reqs.items():
        for c, history in cs.items():
            idf = pd.DataFrame(history, columns=columns)
            idf['exchange'] = e
            idf['coin'] = c
            df = pd.concat([df, idf])

    df = df.set_index(['timestamp', 'exchange', 'coin']).sort_index()

    return df
if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    data = loop.run_until_complete(fetch_history_data(datetime.now() - timedelta(weeks=1040)))
    print(data)

Output from python script.py

Traceback (most recent call last):
  File "/Users/jshcrowthe/Sandbox/script.py", line 76, in <module>
    data = loop.run_until_complete(fetch_history_data(datetime.now() - timedelta(weeks=1040)))
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/Users/jshcrowthe/Sandbox/script.py", line 51, in fetch_history_data
    data = await asyncio.gather(*data_reqs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml/lib/python3.9/site-packages/ccxt/async_support/binance.py", line 2170, in fetch_ohlcv
    response = await getattr(self, method)(self.extend(request, params))
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml/lib/python3.9/site-packages/ccxt/async_support/binance.py", line 4896, in request
    response = await self.fetch2(path, api, method, params, headers, body, config, context)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml/lib/python3.9/site-packages/ccxt/async_support/base/exchange.py", line 101, in fetch2
    await self.throttle(cost)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml/lib/python3.9/site-packages/ccxt/async_support/base/throttler.py", line 45, in __call__
    raise RuntimeError('throttle queue is over maxCapacity')
RuntimeError: throttle queue is over maxCapacity

[ttodua]

I confirm that I’ve recently met cases of DDOS/rate-limit errors with binance . It seems some revision is needed for binance RateLimit config.

[kroitor]

@jshcrowthe i’ll paste an alternative snippet here shortly.