New addresses take long to get into ready state

We have the problem that newly created addresses take quite long to get into the “ready” state - it can be a matter of minutes here.

Looking at the log from the StandardController container log output, it becomes obvious that AddressController.resourcesUpdated() invocations take quite a long time. Here is a log excerpt (minikube environment, 600 addresses, half of them anycast, half brokered), where the method takes roughly 1 minute:

2018-03-07 09:40:24.688 [Thread-3] INFO  AddressController:83 - resourcesUpdated: Check addresses in address space controller: [ ... all 600 addresses here ...]
2018-03-07 09:40:24.764 [Thread-3] DEBUG AddressController:165 - checkStatuses
2018-03-07 09:40:24.767 [Thread-3] DEBUG AddressController:225 - Checking router status of router 172.17.0.11
2018-03-07 09:40:24.824 [Thread-3] DEBUG AddressController:293 - checkRouter: request took 56ms
2018-03-07 09:40:24.855 [Thread-3] DEBUG AddressController:293 - checkRouter: request took 30ms
2018-03-07 09:40:24.880 [Thread-3] DEBUG AddressController:293 - checkRouter: request took 25ms
... 56s ...
2018-03-07 09:41:18.028 [Thread-3] DEBUG AddressController:184 - checkStatuses - done
2018-03-07 09:41:18.085 [Thread-3] INFO  AddressController:114 - Deprovisioning unused addresses in 1 clusters
2018-03-07 09:41:18.086 [Thread-3] INFO  AddressController:117 - Checking cluster with id broker
2018-03-07 09:41:18.087 [Thread-3] INFO  AddressController:119 - Checking cluster broker against address {address=telemetry/ST_818a49cb-047c-43b1-9eae-b827c88531f2_102,name=telemetryst818a49cb-047c-43b1-9eae-b827c88531f2102,uuid=bd5a8ce2-7d63-3a24-bd66-f01649208cbf,annotations={},type=anycast,plan=standard-anycast,status={isReady=true,phase=Active},version=716519}
.. repeated for each address ...
2018-03-07 09:41:18.130 [Thread-3] INFO  AddressController:119 - Checking cluster broker against address {address=event/ST_ae26e212-f378-4790-8d5d-49fa423f780a_21,name=eventstae26e212-f378-4790-8d5d-49fa423f780a21,uuid=2b3155c8-91f3-3efb-92d6-7012b23992bd,annotations={enmasse.io/broker-id=broker-0, cluster_id=broker},type=queue,plan=pooled-queue,status={isReady=true,phase=Active},version=716351}
2018-03-07 09:41:18.132 [Thread-3] DEBUG ConfigMapAddressApi:112 - replaceAddress: get() took 1ms
2018-03-07 09:41:18.138 [Thread-3] DEBUG ConfigMapAddressApi:120 - replaceAddress: replace() took 6ms
.. two lines repeated for each address ...
2018-03-07 09:41:22.716 [Thread-3] DEBUG ConfigMapAddressApi:112 - replaceAddress: get() took 1ms
2018-03-07 09:41:22.721 [Thread-3] DEBUG ConfigMapAddressApi:120 - replaceAddress: replace() took 4ms
2018-03-07 09:41:22.722 [Thread-3] DEBUG AddressController:165 - checkStatuses
2018-03-07 09:41:22.727 [Thread-3] DEBUG AddressController:225 - Checking router status of router 172.17.0.11
2018-03-07 09:41:22.778 [Thread-3] DEBUG AddressController:293 - checkRouter: request took 51ms
2018-03-07 09:41:22.810 [Thread-3] DEBUG AddressController:293 - checkRouter: request took 31ms
2018-03-07 09:41:22.835 [Thread-3] DEBUG AddressController:293 - checkRouter: request took 24ms
2018-03-07 09:41:22.835 [Thread-3] DEBUG AddressController:184 - checkStatuses - done
2018-03-07 09:41:22.835 [Thread-3] INFO  AddressController:109 - resourcesUpdated: done

Observations: resourcesUpdated is called with all addresses as params - although these weren’t really changed. I guess it’s because replaceAddress is invoked on all addresses.

In the 56s period above, there should be quite a lot checkClusterStatus invocations with nested K8s api calls for the isDestinationClusterReady method. I guess a majority of these calls are not necessary.

This whole processing also seems to require alot of CPU: kubectl top pod -n hono returns:

NAME                                           CPU(cores)   MEMORY(bytes)   
address-controller-79d4c4d6c7-8767b            11m          236Mi           
admin-6c786858ff-cprkm                         462m         450Mi           
broker-0                                       6m           981Mi                     
qdrouterd-d8585894f-zqzqj                      8m           60Mi 

A look at the java process: And sampler output:

[calohmn]

@lulf Thumbs up for the changes! Times are looking good now.

[calohmn]

@lulf Thanks! I’ll do the tests based on master then.