How to properly use `watchTrades()` in union with `buildOHLCVC()`?

Hi! I’m trying to stream the OHLCV data from Binance using CCXT.pro using NodeJS, and reading the manual it says that is better to use the 1st order data (trades) and convert it client side in 2nd order data (OHLCV).

All good till here, but what I didn’t understand completely is how you would suggest to do this exactly? I can’t find an example if not the one posted on the manual in Python. Which is understandable, but also a bit too dead simple.

What I’m specifically trying to do is:

• for each market:
• pull trades data from binance in realtime
• with trades data build OHLCV data
• distinguish between a closed candle and a still opened one
• emit an event through the system with finalKlineUpdate or nonFinalKlineUpdate

No matter how I try to rewrite this thing (using rxjs, using promises or with the sync while(true) loop) it always ends up doing the following:

• after a short (sometimes a minute sometimes less) it starts throwing NetworkErrors & TimeOut errors.

• I’d be expecting the log with Final Kline Update to trigger massively for each market ad the ticking of the minute on the clock (or even some seconds after), but what happens instead is that for the whole minute the script keeps logging “final kline update”, which is strage. Am I doing something wrong?

In other words I’m trying to reproduce with CCXT.pro the behaviour of some libraries for NodeJS (specific for Binance) that allowed me to subscribe to klines updates, and have returned 2 types of updates:

• Closed candle (marked as “final” by Binance)
• Non closed candle updates (marked as “final: false” by Binance)

I’ve tried for two days in a row, but I can’t seem to get it working how I’d like to 🥺 Could you please help me? 🙏🏻

• OS: Mac OS Monterey 12.2.1
• Programming Language version: Node 16.x
• CCXT version: ccxt.pro 1.0.66

The code I’ve managed to write is the followning:

// Client instantiation
this.client = new ccxt.binance({
  newUpdates: true,
  enableRateLimit: true,
  apiKey: beamConfig.keys.key,
  secret: beamConfig.keys.sec,
});

private scanAllMarkets(){
  filteredMarkets.forEach((market) =>
      this.scanExchangeMarket(
      market,
      this.beamConfigService.thisBeam.exchange.usedInterval,
      ),
  );
}

private async scanExchangeMarket(
    market: ccxt.Market,
    timeframe: string,
  ): Promise<boolean> {
    // symHistory fetching omitted since not relevant 
    while (true) {
      try {
        const trades = await this.client.watchTrades(market.symbol);
        this.handleTrades(market, timeframe, symHistory, trades);
      } catch (e) {
        this.dbLogger.error({
          logData: true,
          title: CCXTBinanceClientService.name,
          message: `Error getting ${market.symbol} trades data`,
          data: {
            market,
            interval: timeframe,
            error: e,
          },
        });
      }
    }
  }
  
  private async handleTrades(
    market: ccxt.Market,
    timeframe: string,
    symHistory: IBinanceSymbolHistory,
    trades: ccxt.Trade[],
  ) {
    // client is instantiated with "newUpdates: true" so the build result should always be 1 element
    const ohlcv = this.client['buildOHLCVC'](trades, '1m');
    const mappedOHLCV = this.mapOHLCV(ohlcv[0]);

    // if current update === last infra kline
    // update infra kline
    if (
      !symHistory.lastInfraKline ||
      symHistory.lastInfraKline.id === mappedOHLCV.id
    ) {
      symHistory.lastInfraKline = mappedOHLCV;
      this.onInfraKlineUpdate$.next({
        interval: timeframe,
        symbol: market.symbol,
        exchange: Exchanges.binance,
        history: symHistory.history,
        lastKline: symHistory.lastInfraKline,
      });
      return;
    }

    // if current update != than last infra kline
    // move last kline to history
    // register last infra kline to this kline
    symHistory.history.shift();
    symHistory.lastInfraKline = mappedOHLCV;
    symHistory.history.push(symHistory.lastInfraKline);

    this.onFinalKlineUpdate$.next({
      interval: timeframe,
      symbol: market.symbol,
      exchange: Exchanges.binance,
      history: symHistory.history,
    });
    this.onInfraKlineUpdate$.next({
      interval: timeframe,
      symbol: market.symbol,
      exchange: Exchanges.binance,
      history: symHistory.history,
      lastKline: symHistory.lastInfraKline,
    });
    this.logger.debug(`[${market.symbol}] Final Kline Update`);
  }

[caiuscitiriga-zupit]

@kroitor Hi! I’ve followed your suggestions and the example you’ve linked, and now my system can handle virtually the whole exchange markets, accounting for all the limitations.

With that being said, there’s still a shady spot in the example you’ve provided. I’m using the buildOHLCV() method, but it seems to “break” when there are periods without trades.

Let me explain better:

I have a gateway that collects trades from multiple scanners (different servers, all pushing updates to the gateway) The other part of the system hooks up to the gateway, and receives with a throttling time of (500ms) trades updates in chunks (collected by the gateway and dispatched every 500ms)

Once the trades chunks arrives to the consumer, the consumer takes the trades, pushes them into the specific symbol “currentKlineTrades” array and from that builds the OHLCV.

If the resulting built OHLCV has more than one element I can deduce that the previous kline has closed, and a new one has begun.

So i take the first value of the built OHLCV array and push it into the symbol’s “closedKlines” array. (this array is pre-filled with 500 history klines upon startup, so we keep pushing and shifting klines from it to keep it always at 500 of len)

The problem comes here: if a symbol does not receive updates for a long period of time (for example a couple of minutes on the 1m timeframe), when the trades resumes, the kline pushed in the history will have a “gap” of a couple of minutes, so my final array of closed klines will result incorrect, and when further computation is performed on it (like EMA calcs) the result is not correct.

I saw that TradingView for example fills the gaps with “dashes”, so I’m wondering if is as simple as initialising an empty candle with volume value set to 0 and all the other values for prices set to the last kline except for the timestamp, and push it x times as the gaps.

I’ve tried to search around the internet how to “clean” this, but with no luck. Do you have any suggestions? Or some readings that I could look at? Or the solution I’ve came up with looks safe enough to you?

Thank you again for the kind support ❤️

If it can clarify my problem here’s the simplified version of the code that handles the building of OHLCV from trades:

  private async handleTradesInternal(
    cumulativeTrades: ICumulativeTradesEventPayload,
  ) {
    const { market, trades } = cumulativeTrades;
    this.marketsHistory[market].currentKlineTrades.push(...trades);
    const ohlcvc = this.exchange.buildOHLCV(
      this.marketsHistory[market].currentKlineTrades,
      this.beamConfig.thisBeam.usedInterval,
    );

    const lastHistoryKline =
      this.marketsHistory[market].closedKlines[
        this.marketsHistory[market].closedKlines.length - 1
      ];

    if (ohlcvc.length > 1) {
      this.marketsHistory[market].currentKlineTrades = [];
      if (lastHistoryKline[0] === ohlcvc[0][0]) {
        this.marketsHistory[market].closedKlines[
          this.marketsHistory[market].closedKlines.length - 1
        ] = ohlcvc[0];
      } else {
        this.marketsHistory[market].closedKlines.push(ohlcvc[0]);
      }
      this.handleFinalKlineUpdate(market);
    } else {
      this.marketsHistory[market].lastInfraKline = ohlcvc[0];
      this.handleInfraKlineUpdate(market);
    }
  }

[kroitor]

@caiusCitiriga pardon for the long wait, we will add an example asap.