Does this lib support rabbitmq Consitent Hash Exchange?

Hi, thanks for your work. Did not find related discussions in the issues so rising a new one.

I take it the lib exposes the Routing and Exchanges functionality that was not exposed in default Nestjs support classes due to protocol generalization. Does the lib also allow to express a setup with Consistent Hash Exchange? I.e. a setup to distribute messages with different calculatable hashes among an array of queues.

I took a brief look in README and the source and saw that it allows specifying multiple exchanges, but no option to specify queues that are bound to these exchanges. Am I missing something, or does this lib assume there can only be one queue bound to an exchange? Or is queues/exchanges setup intended to be performed manually by the host app perhaps?

The kind of setup I’m looking for:

ch.exchange_declare(exchange="e", exchange_type="x-consistent-hash", durable=True)

for q in ["q1", "q2", "q3", "q4"]:
    ch.queue_declare(queue=q, durable=True)
    ch.queue_purge(queue=q)

for q in ["q1", "q2"]:
    ch.queue_bind(exchange="e", queue=q, routing_key="1")

for q in ["q3", "q4"]:
    ch.queue_bind(exchange="e", queue=q, routing_key="2")

The kind of setup I see expressable through your lib:

RabbitMQModule.forRoot(RabbitMQModule, {
  exchanges: [
    { name: 'e', type: 'x-consistent-hash' },
  ],
  // queues: ???,
  uri: 'amqp://rabbitmq:rabbitmq@localhost:5672',
}),
...
  @RabbitRPC({exchange: 'e', routingKey: '1', queue: 'q1'})
  public async rpcHandler1(msg: {}) {}

  @RabbitRPC({exchange: 'e', routingKey: '1', queue: 'q2'})
  public async rpcHandler2(msg: {}) {}

  @RabbitRPC({exchange: 'e', routingKey: '2', queue: 'q3'})
  public async rpcHandler3(msg: {}) {}

  @RabbitRPC({exchange: 'e', routingKey: '2', queue: 'q4'})
  public async rpcHandler4(msg: {}) {}

Is it the way to go? If so, one obvious drawback is that queues/routingKyes can’t be created dynamically (each needs a separate method declaration in my understanding), can that be leveraged somehow? If Consistent Hash Exchange is possible to setup with this lib, an example proper usage would be highly appreciated, as I’m totally noob!

[WonderPanda]

Hey @klesun. I’ve never used hash exchanges myself but in theory it should work.

This library definitely does not restrict you to using a single queue per exchange. It’s recommended that you would have many queues if you’re really going to adopt RabbitMQ as a central message bus in your architecture.

When you use either the RabbitRPC or RabbitSubscribe and include a named queue property the library will assert the queue so that it’s created if it does not already exist and bind it to the exchange using the routing key in the decorator. So in your example @RabbitRPC({exchange: 'e', routingKey: '1', queue: 'q1'}) this will create a queue called q1 if it does not exist.

Usually I’ve found its convenient that you can tie the queue configuration and the handler together in one spot. Are you suggesting that you think it would be easier to also be able to provide queue binding configuration up front as part of initializing the whole RabbMQModule?

It’s also worth noting that the module also provides access to a NestJS provider that can be injected in your app called AmqpConnection which is a wrapper around the underlying channel so you could easily interact with it directly if that makes more sense for your app and bind things imperatively in your code instead of using decorators. Let me know if that’s helpful to get you started. If things work out with the hash exchange I’ll look into updating the docs

[klesun]

As promised, here is my update: I ended up writing my own implementation of ServerRMQ and ClientRMQ. I was half-way of finishing them when I opened this ticket, but had a small hope that the functionality for working with dynamic number of queues that Nest lacked would be present here together with Exchanges and would so much fit my use case that I would happily discard my own writings…

The code I wrote is proprietary, but the basic idea can be seen in the configuration:

Collapsed Code

const app = await NestFactory.createMicroservice<MicroserviceOptions>(AppModule, {
  strategy: new ServerRMQMultiqueue(serverAmqpOptions),

export const SPLIT_DIVERSIFY_QUEUES = 64;

type AppConfig =
  | typeof appConfig
  | {
      // in tests just few fields enough
      noAck: boolean;
      prefetchCount: number;
    };

/** the numbering in name starts with "1" to match topic numbers as in rabbitmq_consistent_hash_exchange examples */
const makeSerialQueueName = (i: number | string) => 'split_hash_serial_queue_' + (+i + 1);

const getSplitBindings = () => {
  return [...Array(SPLIT_DIVERSIFY_QUEUES).keys()].map(i => ({
    queue: makeSerialQueueName(i),
    exchange: SPLIT_EXCHANGE_NAME,
    routingKey: +i + 1 + '',
  }));
};

export const getQueuesDefinition = (): QueuesDefinition => {
  return {
    exchanges: [{ name: SPLIT_EXCHANGE_NAME, type: 'direct' }],
    queues: [
      {
        queueOptions: { durable: true },
        name: amqpConfig.queues.execution,
      },
      ...getSplitBindings().map(({ queue }) => ({
        queueOptions: { durable: true },
        name: queue,
      })),
    ],
    // main queue does not need binding as it belongs to default '' exchange
    bindings: [...getSplitBindings()],
  };
};

export const getServerAmqpOptions = (appConfigData: AppConfig): ServerAmqpMultiqueueOptions => {
  const { noAck, prefetchCount } = appConfigData;

  return {
    urls: [amqpConfig.url],
    ...getQueuesDefinition(),
    channels: [
      {
        noAck: noAck,
        prefetchCount: prefetchCount,
        consumers: [
          {
            queue: amqpConfig.queues.execution,
            contentFormat: 'json-native',
          },
        ],
      },
      {
        noAck: noAck,
        prefetchCount: prefetchCount,
        consumers: getSplitBindings().map(({ queue }) => ({
          queue: queue,
          contentFormat: 'json-bigint',
        })),
      },
    ],
  };
};

const hexToBigInt = (hex: string) => BigInt('0x' + hex);

const makeRoutingKey = (params: SplitElementData) => {
  const { groupValuesPerSplit, triggerId } = params;
  const hashData = { groupValuesPerSplit, triggerId };
  const hash = crypto.createHash('md5').update(JSON.stringify(hashData)).digest('hex');
  const intHash = hexToBigInt(hash);
  // we would like to eventually move to Consistent Hashing Exchange and
  // remove dependency on this constant from client, but it is not this day
  const hashMod = intHash % BigInt(SPLIT_DIVERSIFY_QUEUES);

  return (hashMod + 1n).toString();
};
...
await return this.executionClient.emitToExchange({
  pattern: 'executeSplitProcess',
  exchange: SPLIT_EXCHANGE_NAME,
  routingKey: makeRoutingKey(params),
  data: params,
});

In the end, I did not try the this.amqpConnection.createSubscriber() solution with your lib, very likely it would have reduced the amount of custom code I’d have to write to make things work, but sadly I already wrote the code, researching how amqlib works along the way. As wise men say, “You can only understand how awesome a framework is after writing your shitty version of it” =-D

I guess, I’m closing this issue as you answered my question. Thank you very much!