Programatically getting all messages from a JetStream stream

What would a reasonable way of doing the same as nats str view streamName --subject filter_subject programmatically be?

I’ve been using JetStreamManager.streams.info(streamName) to get the number of messages in the stream to then use jetstreamClient.fetch(streamName, temporaryDurableConsumerThatUsesFilterSubject, { batch: numberOfMessagesInStream, no_wait: true}) to get to all messages.

And after this, deleting the temporaryDurableConsumerThatUsesFilterSubject

Are there any dangers that might not be obvious in doing this, or is this a reasonable approach?

This is to enable projections over a specific stream, for example: users.someUserId that has a message of type Registered, and then EmailConfirmed from which I can “project” {hasUserConfirmedEmail: true}

[ripienaar]

You probably dont want to pull the full stream in one go as it could overwhelm our client. Do it in batches of 10 or whatever.

Using a durable is annoying, but its the only way we can do pull.

[ruidfigueiredo]

Just experienced an instance where num_pending in the consumer is 0 when there are actual messages, last_active really does seem like the only option which is unfortunate. Here’s the consumer info:

subscription consumerInfo: {
  type: 'io.nats.jetstream.api.v1.consumer_info_response',
  stream_name: 'dev-check-audio-spam',
  name: 'qmXnZGeU',
  created: '2021-12-14T15:14:14.93922231Z',
  config: {
    deliver_subject: '_INBOX.CHND2NFODYEGAGELC0W4K5',
    deliver_policy: 'all',
    ack_policy: 'none',
    ack_wait: 79200000000000,
    max_deliver: 1,
    filter_subject: 'dev.check-audio-spam.61b8b292899bc00029201488.file',
    replay_policy: 'instant',
    idle_heartbeat: 5000000000,
    flow_control: true
  },
  delivered: {
    consumer_seq: 16,
    stream_seq: 2208,
    last_active: '2021-12-14T15:14:14.941194031Z'
  },
  ack_floor: { consumer_seq: 16, stream_seq: 2204 },
  num_ack_pending: 0,
  num_redelivered: 0,
  num_waiting: 0,
  num_pending: 0,
  cluster: { name: 'nats', leader: '....' },
  push_bound: true
}