Priority based requests queue

Describe the feature Add a { priority: number } parameter when enqueueing a request. I imagine it to be lowest number means higher priority.

Motivation Make sure items are pushed to the dataset as soon as possible.

I’ve got a use case where:

• a run might take 6-12 hours if not a day
• an item pushed to the dataset after a nested list of crawled links
• items have expiring content
• the actor’s user reads the dataset during the run to have the lest freshly pushed items

Right now I’ve got the case where 80 to 100% of these items end up being pushe to the dataset in the last 10% of time of the run, because all links end up treated as last.

Constraints I believe there is no simple drop-in solution with the requestQueue/queueing system as no kind of prioritizing is being done so far so it would require to rethink a little bit the system.

Thoughts?

[mnmkng]

Words of a senior dev 😄 👍

[gahabeen]

@mnmkng It’s in fact mostly in-memory for now (only backing it up to KVS). I’ll see to optimise it at scale when time comes.