question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

JetStream - filtered consumer not unique on workqueue stream - queue consumers

See original GitHub issue

I’m hitting a filtered consumer not unique on workqueue stream error when trying to setup a WorkQueue policy on a stream with load balanced consumers. I read the documentation a couple times but I guess there’s something I don’t get.

If I understand correctly, as per the docs on retention policy, it is clearly stated that:

The WorkQueuePolicy mode is a specialized mode where a message, once consumed and acknowledged, is discarded from the Stream. In this mode, there are a few limits on consumers. Inherently it's about 1 message to one consumer, this means you cannot have overlapping consumers defined on the Stream - needs unique filter subjects.

I tried the following options:

const natsSubscriptionOpts = consumerOpts();
natsSubscriptionOpts.ackExplicit();
natsSubscriptionOpts.queue('myqueue');
natsSubscriptionOpts.deliverTo('test.sub');

According to this issue response, you also have to set the durable name (which in my opinion goes against what the documentation says), but I also tried to set it with

natsSubscriptionOpts.durable('worker');

Anyway I always hit the same error once I try to span more than 1 instance of my consumer service, is there any solution to this? Thanks a lot!

Edit: Added some code for example

// Stream creation
jsm.streams.add({
  name: 'ocr-analysis',
  subjects: [
    'ocr.*',
  ],
  retention: RetentionPolicy.Workqueue,
});

// Subscriber creation
const options = natsSubscriptionOpts; // Defined earlier
const subscription = await js.subscribe('ocr.analysis', options);
(async (sub) => {
  for await (const msg of sub) {
    // ...
    msg.ack();
  }
})(subscription);

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:9 (7 by maintainers)

github_iconTop GitHub Comments

3reactions
aricartcommented, Oct 21, 2021

@ovesco when you have a work queue there’s a restriction that you cannot have consumer definitions that overlap. You can filter the subject, but the filtering must not overlap. So if consumer A filters on ocr.A, you cannot have consumer B filter on ocr.*, you can however do ocr.B.

2reactions
derekcollisoncommented, Oct 22, 2021

WorkQueue policy tends to pair best with pull based consumers, of which you can have as many apps as needed to bind to that and request new messages.

If you want a push based consumer bound to a distributed queue that is also possible with a retention based stream like a workqueue but less common pattern for that.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Work-queue Stream in JetStream - NATS by Example
Exclusive non -filtered consumer. As noted in the description above, work-queue streams can only have at most one consumer with interest on a...
Read more >
JetStream Model Deep Dive - NATS Docs
Messages are kept as long as there are Consumers on the stream (matching the message's subject if they are filtered consumers) for which...
Read more >
Get messages history - Google Groups
If the stream is a workqueue stream or has interest policy retention then ... error: multiple non-filtered consumers not allowed on workqueue stream...
Read more >
mod.rs - source - Docs.rs
Source of the Rust file `src/jetstream/mod.rs`. ... Filtered consumer not unique on workqueue stream ConsumerWQConsumerNotUnique = 10100, /// Consumer must ...
Read more >
NATS C Client with JetStream and Streaming support: status.h ...
Multiple non-filtered consumers not allowed on workqueue stream. JSConsumerWQConsumerNotUniqueErr. Filtered consumer not unique on workqueue stream.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found