API Error 500, Probable Cause is this Mod

https://github.com/Moulberry/NotEnoughUpdates/blob/4369156ef9e0a96e0f789b27fa88fff9b2f0f2c7/src/main/java/io/github/moulberry/notenoughupdates/auction/APIManager.java#L283-L301

I’ll break it down so people don’t have to wonder what they are looking at.

• For every single user running this mod (every 5 seconds):
  • If they haven’t queried the auctions API yet, query page 0.
  • If the query lastUpdated value is not identical to the saved lastApiRequest:
    • If config value quickAHUpdate is enabled (not enabled by default), and user is currently looking at the custom auction house screen, set doFullUpdate to true, and query every single page on the auction house on next call (< 5 seconds away)
    • Set all auction pages as needing an update.
  • Set lastApiRequest to lastUpdated
  • If doFullUpdate:
    • Set doFullUpdate to false
    • Call getPageFromAPI for every page in the auctions endpoint, all at once, (dozens of requests in a second)
      • If any of these requests reach the (If the query lastUpdated value is not identical to the saved lastApiRequest), then repeat the above.

Depending on how often the skyblock/auctions endpoint is updated, this is the most likely cause of the API problems currently. It was advertised on YouTube and I estimate it’s got a few thousand users, and maybe hundreds of active users.

So if we take a generous estimate and assume 500 active users. Every 5 seconds, all 500 users request at least 1 auction house page, regardless of if the custom auction house screen is visible. If the custom AH screen is visible, then for every user looking at the auction house, they are also sending out several dozen requests, every 5 seconds.

Let’s assume of those 500, 50% (1/2) have the quick AH feature enabled and are using the remote auction house feature at the same time. That’s 250 users sending 45 requests each, all at once, for a total of ~11,250 requests every X seconds.

X is either 5 seconds (the delay in his mod), or how often the skyblock/auctions endpoint is set to update.

[puppy0cam]

The solution to this is going to be to have a system for receiving updates to data in real-time instead of requiring clients to constantly poll for, and retrieve a large amount of unchanged data. I would propose a public WebSocket or Kafka server for clients to be able to connect to and receive data from as it happens. In this case, the API would only need to process data once and could then distribute a small payload to all connected clients to indicate the changed resource. The RESTful endpoint should be kept to ensure clients are able to sync their state at startup, and just listen for changes for the duration of the program instead of constantly pinging the API to receive large payloads of data.

Such a system has been used successfully before for similar purposes which you can see from https://chatwars.github.io/chatwars-api-docs/public-exchanges.html. Historically the public data from this similar use case was hosted on RabbitMQ, however it proved to be too much of a hassle hurdle and the number of users was too much for RabbitMQ’s poor authentication service. As such it was migrated to Kafka which has proven to be extremely durable and useful for real-time data and even retrieving historical records of up to a month old.

[jecsham]

API is not working properly right now