Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

can't resample data

See original GitHub issue

Hi there,

I’m trying to add 1 minute data to CCXTFeed, and then resample it to 5 minutes, so I can get MACD of 1m and 5m. But failed.

Here’s the code:

# !/usr/bin/env python
# -*- coding: utf-8; py-indent-offset:4 -*-
from __future__ import (absolute_import, division, print_function,
                        unicode_literals)
import sys
import time
from datetime import datetime, timedelta

import backtrader as bt
from ccxtbt import CCXTFeed

class TestStrategy(bt.Strategy):
    params = (
        ('printlog', True),
    )

    def log(self, txt, dt=None, doprint=False):
        ''' Logging function fot this strategy'''
        if self.params.printlog or doprint:
            dt = dt or bt.num2date(self.data.datetime[0])
            print('%s, %s' % (dt, txt))

    def start(self):
        self.counter = 0
        print('START')

    def prenext(self):
        self.counter += 1
        print('prenext len %d - counter %d' % (len(self), self.counter))

    def __init__(self):
        self.macd = bt.indicators.MACDHisto(self.datas[0])
        self.macd2 = bt.indicators.MACDHisto(self.datas[1])


    def next(self):
        self.counter += 1
        price_txt = "Counter: " + str(self.counter) + " Open/Close/High/Low/Volume: " + str(self.data0.open[0]) + " - "+ str(self.data0.close[0]) + " - " + str(self.data0.high[0]) + " - " + str(self.data0.low[0])  + " - " + str(self.data0.volume[0]) + " Len: "+ str(len(self.data0))# + " Time Frame:" + bt.TimeFrame.getname(self.data0._timeframe) + " Len: "+ str(len(self.data0))
        self.log(price_txt)
        macd_txt = "MACD: {:.2f}, Histo: {:.2f}".format(self.macd.macd[0],self.macd.histo[0])
        self.log("MACD#1: " + macd_txt)
        macd2_txt = "MACD: {:.2f}, Histo: {:.2f}".format(self.macd2.macd[0],self.macd2.histo[0])
        self.log("MACD#2: " + macd2_txt)
    
    def notify_data(self, data, status, *args, **kwargs):
        dn = data._name
        dt = datetime.now()
        msg= 'Data Status: {}'.format(data._getstatusname(status))
        print(dt,dn,msg)
        if data._getstatusname(status) == 'LIVE':
            self.live_data = True
        else:
            self.live_data = False

if __name__ == '__main__':
    #cerebro = bt.Cerebro(quicknotify=True)
    cerebro = bt.Cerebro()

    

    #exchange = sys.argv[1] if len(sys.argv) > 1 else 'gdax'
    exchange = sys.argv[1] if len(sys.argv) > 1 else 'binance'
    symbol = sys.argv[2] if len(sys.argv) > 2 else 'ETH/USDT'

    #store = CCXTStore(exchange=exchange, currency='LTC', config=config, retries=5, debug=False)

    hist_start_date = datetime.utcnow() - timedelta(minutes=10)
    print('UTC NOW: ', datetime.utcnow())
    print('hist_start_data: ', hist_start_date)
    print('Using symbol: ', symbol)

    data = CCXTFeed(exchange=exchange,
                             dataname=symbol,
                             timeframe=bt.TimeFrame.Minutes,
                             fromdate=hist_start_date,
                             #todate=datetime(2019, 1, 1, 0, 2),
                             compression=1,
                             ohlcv_limit=2,
                             currency='USDT',
                             retries=5,

                             # 'apiKey' and 'secret' are skipped
                             config={'enableRateLimit': True, 'nonce': lambda: str(int(time.time() * 1000))})

    cerebro.adddata(data)
    cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=2)

    cerebro.addstrategy(TestStrategy)
    # Run the strategy
    cerebro.run()

And here’s the output:

python ccxtbttest.py             
UTC NOW:  2019-06-21 07:19:54.687534
hist_start_data:  2019-06-21 07:09:54.687358
Using symbol:  ETH/USDT
START
2019-06-21 15:20:01.656669 ETH/USDT Data Status: DELAYED
prenext len 1 - counter 1
prenext len 2 - counter 2
2019-06-21 15:20:03.363956 ETH/USDT Data Status: LIVE
prenext len 3 - counter 3
prenext len 4 - counter 4
prenext len 5 - counter 5
prenext len 6 - counter 6
prenext len 7 - counter 7
prenext len 8 - counter 8
prenext len 9 - counter 9
prenext len 10 - counter 10
prenext len 11 - counter 11
prenext len 12 - counter 12
Traceback (most recent call last):
  File "ccxtbttest.py", line 90, in <module>
    cerebro.run()
  File "/Users/michael/.venv/bt-ccxt-store/lib/python3.6/site-packages/backtrader-1.9.74.123-py3.6.egg/backtrader/cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "/Users/michael/.venv/bt-ccxt-store/lib/python3.6/site-packages/backtrader-1.9.74.123-py3.6.egg/backtrader/cerebro.py", line 1298, in runstrategies
    self._runnext(runstrats)
  File "/Users/michael/.venv/bt-ccxt-store/lib/python3.6/site-packages/backtrader-1.9.74.123-py3.6.egg/backtrader/cerebro.py", line 1557, in _runnext
    dt0 = min((d for i, d in enumerate(dts)
ValueError: min() arg is an empty sequence

Any idea?

Thank you.

Issue Analytics

State:
Created 4 years ago
Reactions:2
Comments:8 (1 by maintainers)

Top GitHub Comments

1reaction

LatchRazucommented, Dec 7, 2020

This works for me. Thanks. But are there any specific reasons to use 15 seconds? I used 5 seconds and it seems to work as well.

No specific reason. Excessive caution.

1reaction

LatchRazucommented, Mar 16, 2020

After testing it seems that this bug only triggers when a bar which needs to be resampled is only a few seconds old. I haven’t been able to trigger it while walking through the code.

I came up with this ugly fix: Inside of ccxtfeed.py -> _load

                    start = datetime.utcnow()
                    if start.second < 15:
                        time.sleep(15 - start.second)

                    self._fetch_ohlcv()

Log snippets from some testing I did with 1 and 2 minute bars:

2020-03-16 16:22:22.377627 - fetch_ohlcv - Attempt 0
03-16 16:22 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:24:22.207071 - fetch_ohlcv - Attempt 0
03-16 16:24 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:26:21.639287 - fetch_ohlcv - Attempt 0
03-16 16:26 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:32:11.773225 - fetch_ohlcv - Attempt 0
03-16 16:32 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:42:17.367415 - fetch_ohlcv - Attempt 0
03-16 16:42 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:46:17.382720 - fetch_ohlcv - Attempt 0
03-16 16:46 - custom_classes.f_cerebro: WARNING  works as intended



2020-03-16 16:34:08.293936 - fetch_ohlcv - Attempt 0
03-16 16:34 - custom_classes.f_cerebro: WARNING  resampled bug triggered
2020-03-16 16:40:03.369245 - fetch_ohlcv - Attempt 0
03-16 16:40 - custom_classes.f_cerebro: WARNING  resampled bug triggered
2020-03-16 16:44:05.686620 - fetch_ohlcv - Attempt 0
03-16 16:44 - custom_classes.f_cerebro: WARNING  resampled bug triggered
2020-03-16 16:48:07.024158 - fetch_ohlcv - Attempt 0
03-16 16:48 - custom_classes.f_cerebro: WARNING  resampled bug triggered