Changing address (space) plans

why

One much desired feature of EnMasse is to be able to dynamically change the plan of address space plans and address plans. This applies only to the standard address space type in the first instance, since there are not plan limit enforcement or scaling performed in the brokered address space.

Proposal

Address space plans

Changing the address space plan (i.e. from standard-small to standard-medium) need to happen without losing data. Making sure that the address space plan change is applied is the responsibility of the address-space-controller. The AddressSpace should have an annotation enmasse.io/applied-plan that tracks the currently applied plan, vs. the ‘desired’ plan specified in .spec.plan.

Scaling up

Moving from a ‘small’ to a ‘larger’ plan involves increasing the number (until we have some way of enforcing address space level quotas) of min routers specified by new plan. Alternatively, if we merge #1917, we don’t need to think about this as its part of the infra config instead.

Scaling down

Moving from a larger plan to a smaller one should be possible assuming the amount of currently used resources can fit in the new plan. If the amount of used resources can fit the new plan, we should catch this in the api-server or the address-space-controller. We can either

a) Reject the plan change in api-server and tell the user to try again when resource consumption is reduced b) Accept the plan change in api-server, but have address-space-controller set a status error message on the address space resource explaining why it can’t change the plan. The address-space-controller will then apply the plan change as soon as resource usage is below the limit.

Address plans

Changing the address plan for anycast and multicast address types does not require any broker changes, only that quota is checked and have the change rejected by standard controller if the new plan is above quota limits.

In general, for “stateful” address types, the following changes are needed:

• Add enmasse.io/applied-plan annotation to Address resource to indicate the current plan in use
• Update standard-controller to check enmasse.io/applied-plan against .spec.plan and start the migration process.
• Add a new phase ‘Migrating’ to indicate data “migration” is in progress
• If quota won’t allow the migration, don’t do anything and add error message to address status
• Deprecated the enmasse.io/broker-id annotation and introduce an array .status.brokers containing the list of brokers managing that address. The .status.brokers array would contain the following objects: {.id="same value as old annotation", .state=Active|Draining}
• Use .status.brokers for sharded queues as well, i.e. change standard-controller to treat pooled and sharded queues in the same way in terms of ‘allocating’ brokers (separate deployments for sharded). This means that a queue sharded over 2 brokers would have two brokers in the array, belonging to the same deployment.

Queues

For the queue address type, allocation changes can be done by creating new brokers and marking the old to be decomissioned when no longer active.

Moving from a standard-small-queue to a standard-medium-queue may cause the need to allocate one or more brokers. The following changes can be made to allow that to happen reliably:

• Standard-controller: If new plan can fit within the existing broker, just update applied-plan
• Standard-controller: If new plan requires moving to another broker, set the .status.phase field of the Address to Migrating, update the .status.brokers list with the current broker state set to Draining, and add the new broker(s) to the list.
• Agent: for addresses in Migrating phase, the agent will delete the router incoming traffic autolinks on brokers in Draining state.
• Agent: for addresses in Migrating phase and number of messages in queue is 0, delete the router outgoing traffic autolinks
• Standard-controller: Delete brokers containing no addresses or where addresses addresses are all Draining on that broker and without any active autolinks on the router (in both directions). Remove broker from address list.
• Standard-controller: For addresses in Migrating phase with not brokers in Draining, move back to Active phase and update applied-plan.

The above procedure should work moving from a pooled to a sharded queue and back.

Topics

When a topic is scaled up, it may need to switch to a different set of brokers. The process will work the same as for queues, with brokers being marked as Draining for the address which is going to be moved to another address. All durable subscriptions of a topic must also migrate to the new broker. This may cause ordering issues in the same way as for queues.

• Standard-controller: If new plan can fit within the existing broker, just update applied-plan
• Standard-controller: If new plan requires moving to another broker, set the .status.phase field of the Address and all subscriptions for topic to Migrating, update the .status.brokers list with the current broker state set to Draining, and add the new broker(s) to the list.
• Agent: for topics in Migrating phase, the agent will delete the router incoming linkRoute/autolink on brokers in Draining state.
• Agent: for durable subscriptions in Migrating phase and number of messages for subscription is 0, delete the router outgoing traffic autolinks
• Standard-controller: Delete brokers containing no topics and subscriptions, or where topics and subscriptions are all Draining on that broker and without any active linkroutes+autolinks on the router (in both directions). Remove broker from address list.
• Standard-controller: For addresses in Migrating phase with not brokers in Draining, move back to Active phase and update applied-plan.

Durable subscriptions

Changing the plan of a durable subscription is possible as long as its within the range of the topic credits (as today). There should not be any need for migrating a durable subscription from scaling up. Only when the topic plan is changed, the subscription may need to be migrated to a different broker.

For scaling down, a durable subscription may be moved to a different broker, in which case the same steps as for a queue should apply.

Testing

What is the test plan for the feature/change?

Documentation

What documentation will be created for this feature?

Tasklist

For changing address space plans:

• Add applied-plan annotation on address space and allow plan field to change
• Check quota to determine if new plan can be applied
• Add phase field for address space status which can be Active or Pending (if not created or while plan change is taking effect).
• Apply new plan and change annotation when sucessful
• Allow plan field to be changed in rest api

For changing plans for queues

• Move broker assignment to .status.brokers[] where the field status.brokers[].id is the expected broker id, and change agent to first look for .status.brokers[] before looking for the annotation (for backwards compatibility)
• Change standard-controller to write .status.brokers[] field.
• Add new field .status.brokers[].state which can be Active, Draining
• Add applied-plan field to address annotation and set it when changing plan
• Allow plan field to be changed in rest api
• Set address draining state when plan is migrated to other broker
• Cut off incoming links for migrating addresses

[k-wall]

One orthogonal thought. It should be possible for the user to understand the progress (or lack of it) on their change addressspace/space request. We should have the controllers emit events about the migrations so that the user can do this in a standard manner.

[lulf]

Done