Bitstamp order book using ccxt.pro gets corrupted after a while (asks[0] <= bids[0])

Hi,

We’ve been running all sorts of exchanges using ccxt.pro for a while now, and have this simple validation for the book: ob['asks'][0] > ob['bids'][0]. This should never fail, as it means the exchange are publishing match-able orders to users.

At some point after a few minutes of running, this validation failed for Bitstamp. We thought that it might be a glitch that gets fixed immediately after, but unfortunately this issue persists even hours after the first encounter.

We’ve reproduced it with a slightly modified watch-all-symbols.py example from the ccxt repo. Code attached below. It would be great if you help us figure this out.

Thanks!

Output:

# python examples/ccxt.pro/py/watch-all-symbols.py
2020-12-09T19:54:19.522Z 0 BTC/USD 0 [18253.18, 0.10377839] [18244.37, 0.27081588]
2020-12-09T19:54:21.326Z 1 BTC/EUR 0 [15133.77, 0.28455782] [15130.0, 0.06298595]
2020-12-09T19:54:21.327Z 2 BTC/GBP 0 [13642.96, 0.165] [13634.75, 0.165]
2020-12-09T19:54:24.531Z 4 BTC/USDC 0 [18258.75, 0.00548] [18241.85, 0.01]
2020-12-09T19:54:25.517Z 3 BTC/PAX 0 [18285.96, 0.55] [18213.92, 0.01]
...
2020-12-09T20:08:53.078Z 9 XRP/EUR 3600 [0.46353, 1001.0] [0.463, 10000.0]
2020-12-09T20:08:53.684Z 5 GBP/USD 100 [1.33827, 1120.02987] [1.33718, 1119.61187]
2020-12-09T20:08:54.469Z 12 XRP/PAX 1100 [0.5628, 2632.19616203] [0.55687, 1871.01424728]
2020-12-09T20:08:55.529Z 0 BTC/USD 7500 [18324.02, 0.35] [18311.44, 0.20004647]
Failed book validation for symbol XRP/EUR, orderbook: {'bids': [[0.4633, 2392.14110199], [0.46305, 1001.0], [0.46304, 10000.0], [0.46298, 19601.75431991], [0.46295, 14378.14], [0.46293, 23004.35160215], [0.46291, 17884.932], [0.46287, 7354.05], [0.4628, 2000.0], [0.46276, 30001.0], [0.46259, 500.0], [0.46258, 17897.845], [0.46254, 800.0], [0.46252, 25938.499], [0.46245, 900.0], [0.46228, 6540.0], [0.46226, 1300.0], [0.4622, 3665.0], [0.46218, 8820.0], [0.46217, 1700.0], [0.46198, 11178.0], [0.46197, 21076.85761233], [0.46194, 2200.0], [0.46175, 3000.0], [0.46152, 35937.0], [0.4599, 43351.2981345], [0.45964, 375626.0], [0.45812, 178405.0], [0.45782, 77588.0], [0.45732, 130.34906714], [0.4556, 60.7330992], [0.45521, 165038.0], [0.45467, 131.10879404], [0.45323, 9146.9334], [0.45231, 35674.0], [0.45204, 131.87159408], [0.452, 711.57079646], [0.45193, 123138.0], [0.45111, 600.84275895], [0.45078, 57.67780291], [0.45001, 1222.19506233], [0.45, 41693.9321731], [0.44976, 505.22500889], [0.44942, 132.64037067], [0.448, 5980.60267857], [0.44681, 133.41517734], [0.44571, 500.0], [0.445, 11147.28089887], [0.44427, 159763.0], [0.44423, 134.19002631], [0.44412, 37120.0], [0.444, 500.0], [0.44304, 100.0], [0.443, 1105.0], [0.44229, 280.42528293], [0.44165, 134.97392819], [0.44149, 500.0], [0.4411, 1000.0], [0.441, 120.0], [0.44049, 56.75542837], [0.44035, 778.42625184], [0.44003, 648817.0], [0.44001, 1249.97159155], [0.44, 11430.43181817], [0.43909, 135.76085856], [0.4389, 100.0], [0.438, 135.0], [0.4375, 955.25], [0.43713, 100.0], [0.43662, 36652.0], [0.43659, 59.5524405], [0.43655, 136.55076254], [0.4356, 57.0], [0.43545, 221.51796991], [0.43501, 2258.15498494], [0.435, 12150.00037931], [0.43484, 5286.48238432], [0.4343, 7746.58070458], [0.43402, 137.34674758], [0.43349, 148.19257653], [0.433, 500.0], [0.43299, 31741.0], [0.4322, 100.0], [0.432, 16743.56481481], [0.43151, 138.1456638], [0.43149, 500.0], [0.43075, 35970.0], [0.43023, 60.0], [0.43003, 60.0], [0.43001, 1279.04002233], [0.43, 21741.32569161], [0.42965, 201.69905737], [0.42901, 138.95068969], [0.4288, 80.0], [0.42867, 100.0], [0.42828, 2021.77313906], [0.4275, 100.0], [0.42652, 139.76187608], [0.4261, 60.0], [0.42584, 5256.71613751]], 'asks': [[0.4633, 360.9544], [0.4638, 1001.0], [0.46381, 9242.58947503], [0.46382, 2000.0], [0.46394, 13413.339], [0.46395, 10000.0], [0.46401, 6695.98], [0.46405, 25033.843], [0.46406, 12192.2], [0.46407, 17884.451], [0.46421, 39338.895], [0.46427, 30001.0], [0.466, 275.0], [0.46607, 12915.72489504], [0.46608, 38543.61120377], [0.46609, 14843.0], [0.4661, 58.0], [0.46665, 526.62363533], [0.46666, 200.28091965], [0.467, 114.0168996], [0.46723, 154174.4448151], [0.46724, 64438.0], [0.46726, 129.59527672], [0.46728, 35937.0], [0.468, 90.0], [0.46828, 5209.99024361], [0.4685, 117.60315242], [0.46874, 415.8500914], [0.469, 750.0], [0.46951, 375626.0], [0.46999, 10329.12271917], [0.47, 39573.72923679], [0.47054, 3600.0], [0.47072, 60.5538126], [0.471, 1867.22793719], [0.4712, 10000.0], [0.472, 60000.0], [0.47237, 327.96204], [0.4725, 194.02085648], [0.47272, 128.09703323], [0.47333, 200.28091965], [0.474, 4318.75359245], [0.47404, 35674.0], [0.475, 15776.75666749], [0.47513, 264.02571224], [0.4753, 55.22841922], [0.47547, 127.3553857], [0.4761, 56.0], [0.47614, 115.71606556], [0.47634, 1900.12766939], [0.47666, 200.28091965], [0.477, 345.0], [0.4778, 54.0], [0.478, 555.0], [0.47825, 126.61689759], [0.47829, 269.60276691], [0.4789, 500.0], [0.479, 500.0], [0.47999, 200.28091965], [0.48, 38808.70081192], [0.48022, 19053.17462], [0.48103, 125.88426614], [0.48112, 100.0], [0.4815, 100.0], [0.48317, 114.03323455], [0.48333, 200.28091965], [0.48383, 125.1548087], [0.4839, 5000.0], [0.484, 10000.0], [0.48434, 2066.0], [0.485, 28251.53351857], [0.4851, 63.0], [0.48637, 100.0], [0.48666, 324.70948145], [0.487, 47335.77006507], [0.4876, 312.48256156], [0.488, 386.55662172], [0.48817, 317.0108], [0.48892, 526.12921372], [0.4891, 10095.9721], [0.48922, 68.50787752], [0.48949, 123.70812747], [0.48956, 200.1937856
4], [0.48971, 1621.86807937], [0.48999, 200.28091965], [0.49, 30110.66058515], [0.49045, 310.67031804], [0.49076, 350.0], [0.49107, 53.4547826], [0.49132, 264.02571223], [0.49172, 500.0], [0.492, 500.0], [0.49235, 122.99091232], [0.4924, 10000.0], [0.49299, 111.76203319], [0.49332, 308.85989241], [0.49333, 600.28091965], [0.49367, 1939.65921803], [0.494, 61.57], [0.4944, 51.0]], 'timestamp': 1607544539048, 'datetime': '2020-12-09T20:08:59.048Z', 'nonce': 1607544539048954}

The code that I ran:

# -*- coding: utf-8 -*-

import sys
import asyncio
import ccxtpro


async def loop(exchange, symbol, n):
    i = 0
    while True:
        try:
            orderbook = await exchange.watch_order_book(symbol)
            if orderbook['asks'][0] <= orderbook['bids'][0]:
                print(f"Failed book validation for symbol {symbol}, orderbook: {orderbook}")
                sys.exit(1)
            # print every 100th bidask to avoid wasting CPU cycles on printing
            if not i % 100:
                # i = how many updates there were in total
                # n = the number of the pair to count subscriptions
                now = exchange.milliseconds()
                print(exchange.iso8601(now), n, symbol, i, orderbook['asks'][0], orderbook['bids'][0])
            i += 1
        except Exception as e:
            print(str(e))
            # raise e  # uncomment to break all loops in case of an error in any one of them
            # break  # you can also break just this one loop if it fails


async def main():
    exchange = ccxtpro.bitstamp({'enableRateLimit': True})
    await exchange.load_markets()
    markets = list(exchange.markets.values())
    symbols = [market['symbol'] for market in markets]
    await asyncio.gather(*[loop(exchange, symbol, n) for n, symbol in enumerate(symbols)])
    await exchange.close()


if __name__ == '__main__':
    asyncio.get_event_loop().run_until_complete(main())

[nudabagana]

I can confirm, that this indeed happens. I’m using Node v15.2.1 + ccxt v1.39.1 (but it happened in earlier versions too). To add some info: this happens when using:

bitstamp.subscribeLevel2Snapshots(market);

The problem- order is not being removed. You have an order, that should no longer be there, that’s what causes this to happen. However I also found no good way to reproduce it: sometimes in happens after 2-3hours, sometimes it works perfectly for 5 days until this happens.

Temporary solution is when ob[‘asks’][0] > ob[‘bids’][0], remove older order (it’s out of sync). But of course it’s a dirty fix and finding the real cause would be better.

[langabi]

Unless you’ve overwritten it, the default ccxtpro connection mode to bitstamp is: ‘type’ => ‘order_book’, which per the documentation here: https://www.bitstamp.net/websocket/v2/ just sends a snapshot of top 100 bids and asks with each packet. So there is NO local state that needs to be reset. So if you get an order book returned from the library that has inverted bid/asks, just ignore it and wait for the next order book event.

I have the same issue, but I think it’s a bitstamp issue: I’ve checked the actual incoming packet contents, and it indeed has an inverted bid/ask spread. So my guess it’s some race condition in bitstamp’s servers (eg sends orderbook snapshot before running the matching algorithm). Happens a few times a day, typically, no pattern I’ve seen. Cost me quite some cash though!