Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Simple fix for "Error in data stream"

See original GitHub issue

In case others are also encountering this error on batch downloads, here’s a simple fix.

diff --git a/packages/dukascopy-node/src/buffer-fetcher/index.ts b/packages/dukascopy-node/src/buffer-fetcher/index.ts
index fbe14ce..0818e7b 100644
--- a/packages/dukascopy-node/src/buffer-fetcher/index.ts
+++ b/packages/dukascopy-node/src/buffer-fetcher/index.ts
@@ -78,7 +78,10 @@ export class BufferFetcher {
       return this.fetcherFn(url);
     }
 
-    const data = await fetch(url);
-    return data.buffer();
+    while (1) {
+      const data = await fetch(url);
+      if (data.status==200) return data.buffer();
+      await wait(100);   // Back-off before retry
+    }
   }
 }

Issue Analytics

State:
Created 2 years ago
Reactions:2
Comments:5 (3 by maintainers)

Top GitHub Comments

1reaction

straitsboycommented, Aug 8, 2021

Hi Leonid,

This error happens randomly. Sometimes everything goes smoothly with no errors. Other times it gets loads of 503 responses. I believe it is because Dukascopy is rate-throttling on their end. So there’s nothing much we can do about it, except to retry the fetch when it happens.

Here’s an example that I just ran. I ran it three times. Twice it ended with no errors. On the third run, it gets several 503 errors. I’ve attached an excerpt of the logs…

  {
    instrument: 'usdtry',
    dates: { from: 1572271200000, to: 1628276401000 },
    timeframe: 'h1',
    format: 'json',
    pauseBetweenBatchesMs: 1000
  }

  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2020/02/BID_candles_hour_1.bi5
  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2020/05/BID_candles_hour_1.bi5
  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2020/06/BID_candles_hour_1.bi5
  200 OK https://datafeed.dukascopy.com/datafeed/USDTRY/2019/09/BID_candles_hour_1.bi5
  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2020/01/BID_candles_hour_1.bi5
  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2020/03/BID_candles_hour_1.bi5
  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2019/10/BID_candles_hour_1.bi5
  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2019/11/BID_candles_hour_1.bi5
  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2020/04/BID_candles_hour_1.bi5
  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2020/02/BID_candles_hour_1.bi5
  200 OK https://datafeed.dukascopy.com/datafeed/USDTRY/2020/00/BID_candles_hour_1.bi5
  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2020/06/BID_candles_hour_1.bi5
  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2020/05/BID_candles_hour_1.bi5
  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2020/03/BID_candles_hour_1.bi5
  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2020/01/BID_candles_hour_1.bi5
  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2019/10/BID_candles_hour_1.bi5
  200 OK https://datafeed.dukascopy.com/datafeed/USDTRY/2019/11/BID_candles_hour_1.bi5
  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2020/05/BID_candles_hour_1.bi5
  503 Service Temporarily Unavailable https://datafeed.dukascopy.com/datafeed/USDTRY/2020/02/BID_candles_hour_1.bi5
  ...

0reactions

straitsboycommented, Aug 23, 2021

That’s great! Thank you very much. 😃