Inconsistent candle data returned when using fetch_ohlcv() at 1 minute granuality for Huobi.

• OS: MacOS Montrey 12.2.1
• Programming Language version: Python 3.8.8
• CCXT version: 1.74.6
• Exchange: Huobi
• Method: fetch_ohlcv()

Here, I am trying to fetch the complete historical OHLCV data about a trading pair from an exchange using fetch_ohlcv() at a 1-minute interval.

import time

import ccxt
import pandas as pd

# set exchange
exchange = ccxt.huobi({
    'enableRateLimit': True,
    # 'verbose': True,
})

symbol = 'BTC/USDT'
tf = '1m'
from_datetime = '2018-06-24 00:00:00'
from_timestamp = exchange.parse8601(from_datetime)
# end time
now = exchange.milliseconds()

# set timeframe in msecs
tf_multi = 60 * 1000
hold = 30

# make list to hold data
data = []

candle_no = (int(now) - int(from_timestamp)) / tf_multi + 1
print('downloading...')
while from_timestamp < now:
    try:
        ohlcvs = exchange.fetch_ohlcv(symbol, tf, from_timestamp)
        from_timestamp += len(ohlcvs) * tf_multi
        print(from_timestamp)

        data += ohlcvs  # ←------------------------------- this particular line will mix them

        print(str(len(data)) + ' of ' + str(int(candle_no)) + ' candles loaded...')
    except (ccxt.ExchangeError, ccxt.AuthenticationError, ccxt.ExchangeNotAvailable, ccxt.RequestTimeout) as error:
        print('Got an error', type(error).__name__, error.args, ', retrying in', hold, 'seconds...')
        time.sleep(hold)

header = ['t', 'o', 'h', 'l', 'c', 'v']

#  ↓↓↓↓ added timestamp sorting of your mixed "data" to this line ↓↓↓↓

df = pd.DataFrame(sorted(data, key=lambda x: x[0]), columns=header)

The script seems to return the wrong data for Huobi with the trading_pair as BTC/USDT. For the first iteration, the query for the data is from 2018-06-24 but the data that is returned is from 2022-03-13. There is a lot of duplication of the rows if this script is executed for Huobi, while the same script executed for Binance, works completely fine.

>>> df = pd.read_csv('foo.csv')
>>> df
         Unnamed: 0              t         o         h         l         c         v
0                 0  1647210840000  38719.64  38720.56  38670.93  38670.93  3.840178
1                 1  1647210840000  38719.64  38720.56  38670.93  38670.93  3.840178
2                 2  1647210840000  38719.64  38720.56  38670.93  38670.93  3.840178
3                 3  1647210840000  38719.64  38720.56  38670.93  38670.93  3.840178
4                 4  1647210840000  38719.64  38720.56  38670.93  38670.93  3.840178
...             ...            ...       ...       ...       ...       ...       ...
1957878     1957878  1647271320000  39001.50  39029.00  38998.79  39021.37  6.081969
1957879     1957879  1647271320000  39001.50  39029.00  38998.79  39021.37  6.081969
1957880     1957880  1647271320000  39001.50  39029.00  38998.79  39017.19  6.082467
1957881     1957881  1647271320000  39001.50  39029.00  38998.79  39017.19  6.082467
1957882     1957882  1647271320000  39001.50  39029.00  38998.79  39017.19  6.117897

[1957883 rows x 7 columns]
>>> df['diff'] = df['t'].diff()
>>> df
         Unnamed: 0              t         o         h         l         c         v  diff
0                 0  1647210840000  38719.64  38720.56  38670.93  38670.93  3.840178   NaN
1                 1  1647210840000  38719.64  38720.56  38670.93  38670.93  3.840178   0.0
2                 2  1647210840000  38719.64  38720.56  38670.93  38670.93  3.840178   0.0
3                 3  1647210840000  38719.64  38720.56  38670.93  38670.93  3.840178   0.0
4                 4  1647210840000  38719.64  38720.56  38670.93  38670.93  3.840178   0.0
...             ...            ...       ...       ...       ...       ...       ...   ...
1957878     1957878  1647271320000  39001.50  39029.00  38998.79  39021.37  6.081969   0.0
1957879     1957879  1647271320000  39001.50  39029.00  38998.79  39021.37  6.081969   0.0
1957880     1957880  1647271320000  39001.50  39029.00  38998.79  39017.19  6.082467   0.0
1957881     1957881  1647271320000  39001.50  39029.00  38998.79  39017.19  6.082467   0.0
1957882     1957882  1647271320000  39001.50  39029.00  38998.79  39017.19  6.117897   0.0

[1957883 rows x 8 columns]
>>> df.value_counts('diff')
diff
0.0        1956874
60000.0       1008
dtype: int64
>>> 

Not sure what is causing this behavior or how to fix it? This error does not appear to occur when I use binance instead of huobi.

[ttodua]

@YuriyTigiev it’s hard to verbally describe, maybe you can take a look here - https://github.com/ccxt/ccxt/blob/5b1adf0938f18ec4157164a1ef99b1557681856b/js/ftx.js#L944

[ttodua]

@YuriyTigiev thanks for sharing useful bits of infos! I will keep note of it, as I didn’t expect FTX returned delayed data and misbehaved at that level.