New addresses take long to get into ready state
See original GitHub issueWe have the problem that newly created addresses take quite long to get into the “ready” state - it can be a matter of minutes here.
Looking at the log from the StandardController container log output, it becomes obvious that AddressController.resourcesUpdated()
invocations take quite a long time.
Here is a log excerpt (minikube environment, 600 addresses, half of them anycast, half brokered), where the method takes roughly 1 minute:
2018-03-07 09:40:24.688 [Thread-3] INFO AddressController:83 - resourcesUpdated: Check addresses in address space controller: [ ... all 600 addresses here ...]
2018-03-07 09:40:24.764 [Thread-3] DEBUG AddressController:165 - checkStatuses
2018-03-07 09:40:24.767 [Thread-3] DEBUG AddressController:225 - Checking router status of router 172.17.0.11
2018-03-07 09:40:24.824 [Thread-3] DEBUG AddressController:293 - checkRouter: request took 56ms
2018-03-07 09:40:24.855 [Thread-3] DEBUG AddressController:293 - checkRouter: request took 30ms
2018-03-07 09:40:24.880 [Thread-3] DEBUG AddressController:293 - checkRouter: request took 25ms
... 56s ...
2018-03-07 09:41:18.028 [Thread-3] DEBUG AddressController:184 - checkStatuses - done
2018-03-07 09:41:18.085 [Thread-3] INFO AddressController:114 - Deprovisioning unused addresses in 1 clusters
2018-03-07 09:41:18.086 [Thread-3] INFO AddressController:117 - Checking cluster with id broker
2018-03-07 09:41:18.087 [Thread-3] INFO AddressController:119 - Checking cluster broker against address {address=telemetry/ST_818a49cb-047c-43b1-9eae-b827c88531f2_102,name=telemetryst818a49cb-047c-43b1-9eae-b827c88531f2102,uuid=bd5a8ce2-7d63-3a24-bd66-f01649208cbf,annotations={},type=anycast,plan=standard-anycast,status={isReady=true,phase=Active},version=716519}
.. repeated for each address ...
2018-03-07 09:41:18.130 [Thread-3] INFO AddressController:119 - Checking cluster broker against address {address=event/ST_ae26e212-f378-4790-8d5d-49fa423f780a_21,name=eventstae26e212-f378-4790-8d5d-49fa423f780a21,uuid=2b3155c8-91f3-3efb-92d6-7012b23992bd,annotations={enmasse.io/broker-id=broker-0, cluster_id=broker},type=queue,plan=pooled-queue,status={isReady=true,phase=Active},version=716351}
2018-03-07 09:41:18.132 [Thread-3] DEBUG ConfigMapAddressApi:112 - replaceAddress: get() took 1ms
2018-03-07 09:41:18.138 [Thread-3] DEBUG ConfigMapAddressApi:120 - replaceAddress: replace() took 6ms
.. two lines repeated for each address ...
2018-03-07 09:41:22.716 [Thread-3] DEBUG ConfigMapAddressApi:112 - replaceAddress: get() took 1ms
2018-03-07 09:41:22.721 [Thread-3] DEBUG ConfigMapAddressApi:120 - replaceAddress: replace() took 4ms
2018-03-07 09:41:22.722 [Thread-3] DEBUG AddressController:165 - checkStatuses
2018-03-07 09:41:22.727 [Thread-3] DEBUG AddressController:225 - Checking router status of router 172.17.0.11
2018-03-07 09:41:22.778 [Thread-3] DEBUG AddressController:293 - checkRouter: request took 51ms
2018-03-07 09:41:22.810 [Thread-3] DEBUG AddressController:293 - checkRouter: request took 31ms
2018-03-07 09:41:22.835 [Thread-3] DEBUG AddressController:293 - checkRouter: request took 24ms
2018-03-07 09:41:22.835 [Thread-3] DEBUG AddressController:184 - checkStatuses - done
2018-03-07 09:41:22.835 [Thread-3] INFO AddressController:109 - resourcesUpdated: done
Observations:
resourcesUpdated
is called with all addresses as params - although these weren’t really changed. I guess it’s because replaceAddress
is invoked on all addresses.
In the 56s period above, there should be quite a lot checkClusterStatus
invocations with nested K8s api calls for the isDestinationClusterReady
method. I guess a majority of these calls are not necessary.
This whole processing also seems to require alot of CPU:
kubectl top pod -n hono
returns:
NAME CPU(cores) MEMORY(bytes)
address-controller-79d4c4d6c7-8767b 11m 236Mi
admin-6c786858ff-cprkm 462m 450Mi
broker-0 6m 981Mi
qdrouterd-d8585894f-zqzqj 8m 60Mi
A look at the java process:
And sampler output:
Issue Analytics
- State:
- Created 6 years ago
- Reactions:1
- Comments:6 (6 by maintainers)
@lulf Thumbs up for the changes! Times are looking good now.
@lulf Thanks! I’ll do the tests based on master then.