Add support for defining DLQ/expiry queue for an individual queue

Description

Add support for defining DLQ and expiry queue for an individual queue.

At a high level, the user needs to be able to

1. associate queue or subscription with a dead letter and expiry address.
2. configure redelivery / poison message handling parameters.
3. consume messages from the dead letter/expiry address so that an operation (business operation to retry an operation etc).

Design

Address Plan

addressplans.admin.enmasse.io will be extended to support DLQ/expiry queues.

Dead Letter Address Plans

The plan for the deadletter type follows the existing schema however, the resources section will be ignored.

metadata:
   name: standard-deadletter
spec:
   displayName: Dead Letter Queue
   addressType: deadletter

Addressplans of type deadletter will support the existing messageTtl section. Dead letter address instances of that plan will automatically purge dead lettered messages older than a certain time.

metadata:
   name: standard-deadletter-ttl
spec:
   displayName: Dead Letter Queue
   addressType: deadletter
   messageTtl:
      maximum: 86400000

Queue/Subscription Address Plans

The address plan definitions for subscriptions and queues types will support a new field messageRedelivery (optiona) defining the redelivery characteristics.

spec:
   displayName: Small Queue
   addressType: queue
   messageRedelivery:
     maxDeliveryAttempts: 10
     redeliveryDelayMultiplier: 1.5
     redeliveryDelay: 5000
    ...

Address

addresses.enmasse.io will be extended to support DLQ/expiry queues.

The address crd will support a new address type deadletter. This will allow instances of a special address type deadletter to be defined. Instances will serve as dead letter destinations and will be referred to be other addresses. It will be possible for client applications to receive from deadletter address instances, but sending will be disallowed. The deadLetter queue will exist on every broker in the address space. This will be achieved in router configuration.

metadata:
   name: myspace.mydlqaddr
spec:
   plan: standard-deadletter
   type: deadletter
   address: mydlqaddr

Addresses of type deadletter will support the existing messageTtl section, allowing the user to config a deadletter address that automatically purges dead lettered messages older than a certain time.

metadata:
   name: myspace.mydlqaddrttl
spec:
   plan: standard-deadletter
   type: deadletter
   address: mydlqaddr
   messageTtl:
      maximum: 86400000

Address types queue and subscription will support a new fields deadLetterAddress and expiryAddress. If defined, these must reference the address of a previously defined deadletter address instance. If the reference(s) doesn’t exist or is of the wrong type, this will be reported via the status section of the address, much like topic references behave today. An optional fields allowing the redelivery/maxdelivery settings to be specified (overriding any values set at the plan).

The status section will report message redelivery settings that are in force for the address.

metadata:
   name: myqueue
spec:
   plan: standard-small-queue
   type: queue
   deadLetterAddress: mydeadletterqueue
   expiryAddress: myexpiryqueue
   messageRedelivery:
     maxDeliveryAttempts: 10
     redeliveryDelayMultiplier: 1.5
     redeliveryDelay: 5000
status:
   ...
   messageRedelivery:
     maxDeliveryAttempts: 10
     redeliveryDelayMultiplier: 1.5
     redeliveryDelay: 5000
  ...

The Agent will apply the dead letter settings to the Broker(s) in the same way as address size/TTL are configured today.

Tasklist

Enumerate the sub-tasks required to complete the task:

• Extend addressplan
• Extend addresses
• Controller logic to enforce integrity between queues/subscriptions and dead letter instances (standard and brokered)
• Agent to create dead letter queues on all broker(s)
• Agent to configure address settings
• Router configuration to expose dead letter queues as a consume only address
• Make dead letter queues valid for purge operator
• [UI] Add ability for create/delete dead letter queue
• [UI] Add ability to associate queue/subscription with an existing dead letter queue.
• [UI] Enable dead letter purge
• End to end system tests (testing ability to configure dead lettering from the resource including messaging behaviour).
• End to end system tests (ability to create dlqs and assign them to queues/subscriptions).

[lulf]

@sophokles73 This depends on the broker configuration applied, but the default should be (@k-wall probably good if you could confirm):

modified (delivery-failed) | released | rejected -> retry until max retries -> DLQ

There is a tunable for ‘rejected’ that will cause messages to go straight to DLQ (part of .spec.broker in StandardInfraConfig).

By default, the max retries should be infinite with a back-off, but with this PR, it is tunable per address and/or plan, as well as the DLQ address.

[k-wall]

@k-wall So if I understand everything right, from the UI perspective, these are the expected changes:

Create Address Wizard:

• Address Configuration: When the user selects the type as subscription or queue , two new drop down options will appear, which will let the user select dlq and expiry queue (Like the current implementation topics for Subscription)

  • Q: Should the fields for dlq and expiry queue be drop downs or a checkbox makes more sense?

Dropdowns. The backend will need to query for all addresses of type deadletter within the addressspace. This is the same query you already run for topics on the create subscription workflow.

• Q: Are both of these drop down selections (dlq and expiry queue) mandatory?

They are both optional (and independent of one and other).

• Preview: The Preview step will display the type, plan and dlq/expiry queue selections. The selected choices will also reflect in the configuration details.

Yup.

Address Detail Page

• The Header in the Address Detail page will display the Address’s Plan, dlq and expiry queue.

Make sense.

• Q: Should the header also display the selected type?
• The Kebab on the Header would enable the user to Edit/Delete/Purge the address of Queue/Subscription type (Existing feature)

The user needs to be able to purge addresses of type deadletter too.

Address List

The Kebab on each row of the address list table and the bulk selection kebab will enable user to Edit/Delete/Purge the address of Queue/Subscription type (Existing feature)

Q: Is there any change that should be implemented in UI for messageRedelivery?

No - I think opening up the messageRedelivery options (and messageTtl) would be a separate feature required (not planned at the moment).