JetStream - filtered consumer not unique on workqueue stream - queue consumers
See original GitHub issueI’m hitting a filtered consumer not unique on workqueue stream
error when trying to setup a WorkQueue policy on a stream with load balanced consumers. I read the documentation a couple times but I guess there’s something I don’t get.
If I understand correctly, as per the docs on retention policy, it is clearly stated that:
The WorkQueuePolicy mode is a specialized mode where a message, once consumed and acknowledged, is discarded from the Stream. In this mode, there are a few limits on consumers. Inherently it's about 1 message to one consumer, this means you cannot have overlapping consumers defined on the Stream - needs unique filter subjects.
I tried the following options:
const natsSubscriptionOpts = consumerOpts();
natsSubscriptionOpts.ackExplicit();
natsSubscriptionOpts.queue('myqueue');
natsSubscriptionOpts.deliverTo('test.sub');
According to this issue response, you also have to set the durable name (which in my opinion goes against what the documentation says), but I also tried to set it with
natsSubscriptionOpts.durable('worker');
Anyway I always hit the same error once I try to span more than 1 instance of my consumer service, is there any solution to this? Thanks a lot!
Edit: Added some code for example
// Stream creation
jsm.streams.add({
name: 'ocr-analysis',
subjects: [
'ocr.*',
],
retention: RetentionPolicy.Workqueue,
});
// Subscriber creation
const options = natsSubscriptionOpts; // Defined earlier
const subscription = await js.subscribe('ocr.analysis', options);
(async (sub) => {
for await (const msg of sub) {
// ...
msg.ack();
}
})(subscription);
Issue Analytics
- State:
- Created 2 years ago
- Comments:9 (7 by maintainers)
Top Results From Across the Web
Work-queue Stream in JetStream - NATS by Example
Exclusive non -filtered consumer. As noted in the description above, work-queue streams can only have at most one consumer with interest on a...
Read more >JetStream Model Deep Dive - NATS Docs
Messages are kept as long as there are Consumers on the stream (matching the message's subject if they are filtered consumers) for which...
Read more >Get messages history - Google Groups
If the stream is a workqueue stream or has interest policy retention then ... error: multiple non-filtered consumers not allowed on workqueue stream...
Read more >mod.rs - source - Docs.rs
Source of the Rust file `src/jetstream/mod.rs`. ... Filtered consumer not unique on workqueue stream ConsumerWQConsumerNotUnique = 10100, /// Consumer must ...
Read more >NATS C Client with JetStream and Streaming support: status.h ...
Multiple non-filtered consumers not allowed on workqueue stream. JSConsumerWQConsumerNotUniqueErr. Filtered consumer not unique on workqueue stream.
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
@ovesco when you have a work queue there’s a restriction that you cannot have consumer definitions that overlap. You can filter the subject, but the filtering must not overlap. So if consumer A filters on
ocr.A
, you cannot have consumer B filter onocr.*
, you can however doocr.B
.WorkQueue policy tends to pair best with pull based consumers, of which you can have as many apps as needed to bind to that and request new messages.
If you want a push based consumer bound to a distributed queue that is also possible with a retention based stream like a workqueue but less common pattern for that.