Requirements for shared infrastructure

Overview

Address spaces in EnMasse are currently isolated, i.e. they don’t share infrastructure (router network, brokers). Moreover, the standard-controller and agent deployments are created per address space. Sharing infrastructure allows sharing resources and reducing the footprint when configuring multiple address spaces.

Before writing a design proposal up, some discussion about the general direction we want to take needs to take place. Once there is a quorum, I will write down a more detailed proposal.

Here is an initial set of questions that I feel we need to discuss:

1. Which address space types should shared infra apply to?
2. Do we support only 1 or multiple instances of shared infra?
3. How does the service-admin configure the shared infrastructure?
4. How does the messaging tenant use the shared infrastructure?
5. How is the shared infrastructure managed? (I.e. by which components)
6. How are plans applied in shared infra?

Changes are required to be backwards-compatible with existing APIs, but the design will consider the ‘ideal’ architecture to set an initial goal to work towards and propose tasks that can be carried out once a backward-incompatible change can be made.

I will provide my views on the above questions, and throw out a few ideas to trigger the discussion.

1. Which address space types does shared infra apply to?

I think both standard and brokered address space types are good candidates for shared infra. standard because it allows brokers to scale horizontally and can support many addresses only limited by the router. brokered because it would allow resource sharing of a broker/broker-cluster which would make for a feature-rich deployment and allow bridging for the brokered space.

For brokered, I think shared infrastructure means we have to deploy the dispatch router to get the ‘virtual broker’ feature. Deploying the router would make the standard and brokered address spaces more similar and maybe we should rethink that. I think a key difference is that for brokered, the broker(s) are configured as a ‘broker cluster’, and there is no scaling logic part of the enmasse operator. With the router features for always doing link forwarding, one would get most of the broker features as today.

Should we still support todays brokered address space deploying a single broker? Or is that use case better served via a broker operator instead?

An attractive option is that we leave both existing address space types as is, and introduce a new address space type for anything ‘shared’ that supports the features of both ‘standard’ and ‘brokered’. In that context, the desired semantics could be specified as a property of the address instead (i.e. ‘transactions’), and the operator would take care to ensure its allocated to the link-routed broker cluster. Once the shared type supports all desired address semantics, the existing standard and brokered infra can go away, and we get rid of address space types…

2. Do we support 1 or multiple instances of shared infra?

In the future, with the move the CRDs for all resources, I think we are in a better position to allow multiple EnMasse instances on a cluster. The question is then if multiple shared infra instances is better handled that way than needing to handle that within a single instance.

However, if we are to eventually replace ‘brokered’ and ‘standard’ with the new ‘shared’ infra type, then we probably need a way to create dedicated instances for some installations where installing multiple EnMasse instances is not an option.

3. How does the service admin configure the infrastructure?

For compatibility reasons, I think the existing StandardInfraConfig and BrokeredInfraConfig should be used to configure the existing address space types. A new address space type shared combined with a SharedInfraConfig can be used to configure the shared infrastructure(s).

However, in a world without backward-compatibility and with support for multiple EnMasse instances/operators per cluster, I would like to consider the following: A new CRD enmasse, which would contain what is currently part of the singleton resources ConsoleService (under .spec.console), IoTConfig(under .spec.iot) andStandardInfraConfig/BrokeredInfraConfig` (under .spec.infra.standard?) etc. A service-admin can then decide to either deploy 1 operator + 1 CR instance as today, or deploy multiple operators managing multiple instances (each independent of eachother).

4. How does the messaging tenant use the shared infrastructure?

IMO this should work exactly the same way as before. The plans and infra from the service admin dictate if dedicated or shared infra will be used.

5. How is the shared infrastructure managed? (I.e. by which components)

I think the end-goal should be that that all infra is managed by enmasse-operator. However, the path to that end goal I think is:

• Phase 1: Create initial shared address space type purely managed by enmasse-operator. This address space type supports all features from standard address space except MQTT. Deprecate standard address space. Use router operator to deploy routers.
• Step 2: Add support for deploying broker clusters using broker operator and introduce address properties to select ‘broker’-semantics like ‘transactions’. These addresses will be put on the broker cluster. Deprecate brokered address space
• Step 3: Remove standard and brokered address space once ‘shared’ supports the desired set of semantics.

I think splitting it up like this will allow us to perform the work in pieces while maintaining a stable and functional system for existing deployments.

6. How are plans applied in shared infra?

With shared infra, the limits in plans will have to translate to ‘something else’ than infra. IMO I think they can make a good use as a quota system, i.e. a shared infra with 3 routers can support 10 address space plans with resource limit ‘router: 0.1’). The credits of address space plans and address plans would have to be adjusted/lowered to take into account shared infra.

In example-plans, we can provide example plans that ends up using the shared infra.

I think once the router and broker supports a more fine-grained level of limitations, we should map the current set of credits to connection, link limits, per-address memory limits in broker etc. We should consider expanding the address space plan with settings not specific to infra, like max connections, sessions, links etc. in order to allow defining different limits for different tenants in the address space.

[rgodfrey]

The other (potential) advantage for shared infrastructure is that in order to create an address space (potentially for just short term use - e.g. running a test) no new pods need to be deployed, etc… which I would hope would reduce the amount of time for an address space to become ready.

– Rob

On Tue, 5 Nov 2019, 12:15 grs, [e-mail] wrote:

I think a realistic aggregate throughput is that you can sort of divide the max throughput of a dedicated instance by the number of address spaces

Agreed. Do we know what that max throughput is? (That would determine how many address-spaces you could reasonably expect to share).

I do think that the those two configurations can give comparable memory usage in cases where you have many addresses and/or connections, because the constant base cost of a router would be insignificant to the cost of addresses.

What would be the point at which the cost of the addresses would start to outweigh the base footprint? That would start to define the envelope within which the sharing of messaging infrastructure would have some benefit. (e.g. if you have one address per address-space, you could save the full router footprint on each extra address space up to some quite high number. For 10 addresses per address space the saving would be slightly less etc).

Is the intention for address-spaces from different ‘owners’ to share infrastructure? Or just for a given ‘owner’ to be able to define lower overhead namespaces?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/EnMasseProject/enmasse/issues/3420?email_source=notifications&email_token=AAAREQZ2QAKFCEBOU5UKTOLQSFIWHA5CNFSM4JITT4EKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDCPPPY#issuecomment-549779391, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAREQ3KU7I4YSMIMEFIU6DQSFIWHANCNFSM4JITT4EA .

[grs]

That is not too bad for address count (and possibly it is even higher? is the factor for link capacity really the 18k for the link base cost?). It does seem to me that the throughput is more likely to be the problem (1-2k msgs/sec for queues per address space seems more restrictive than 160+ addresses). However I guess there are many and varied cases. Though I remain somewhat sceptical I’m much less concerned if this is simply an option to allow for that variation.