Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Misconfigured coordinator service in helm chart

See original GitHub issue

Unable to connect router to coordinator service.

Templating the helm chart a bit (getting rid of bitnami’s amd64 zookeeper, and postgres mainly) and deploying the official helm chart I can’t connect the router to any other services apparently.

image

However the coordinator seems to be connected to other services, image image

And the druid doctor shows the following image

Although all my pods seem to be running smoothly image

druid-zookeeper-headless       ClusterIP   None            <none>        2181/TCP,3888/TCP,2888/TCP   3h4m
druid-zookeeper                ClusterIP   10.43.127.52    <none>        2181/TCP                     3h4m
druid-broker                   ClusterIP   10.43.35.55     <none>        8082/TCP                     3h4m
druid-coordinator              ClusterIP   10.43.38.218    <none>        8081/TCP                     3h4m
druid-historical               ClusterIP   10.43.173.83    <none>        8083/TCP                     3h4m
druid-middle-manager           ClusterIP   10.43.146.5     <none>        8091/TCP                     3h4m
druid-router                   ClusterIP   10.43.135.124   <none>        8888/TCP                     3h4m

Affected Version

arm64 build 0.22.0 @98957be0443b669cc7464886ef9ee21d3d21f762

Description

Please include as much detailed information about the problem as possible.

  • Cluster size: 3 nodes 4cpu each
  • Configurations in use: helm v3.7.1, k3s v1.22.2, longhorn v1.2.2 as storageclass
  • Steps to reproduce the problem: bumping zookeeper image to v3.5.8+ from the helm dependency, and setting an external psql instance for runtime metadata
                 List of relations
 Schema |         Name          | Type  |  Owner   
--------+-----------------------+-------+----------
 public | druid_audit           | table | druid
 public | druid_config          | table | druid
 public | druid_datasource      | table | druid
 public | druid_pendingsegments | table | druid
 public | druid_rules           | table | druid
 public | druid_segments        | table | druid
 public | druid_supervisors     | table | druid
 public | druid_tasklocks       | table | druid
 public | druid_tasklogs        | table | druid
 public | druid_tasks           | table | druid
(10 rows)
  • The error message or stack traces encountered. Providing more context, such as nearby log messages or even entire logs, can be helpful.
  • Any debugging that you have already done Router trace
druid-router-7f6bcfb79c-9wdh9 druid 2021-11-26T04:07:54,891 ERROR [qtp555864569-122] org.apache.druid.server.router.QueryHostFinder - No server found for serviceName[druid/broker]. Using backup
druid-router-7f6bcfb79c-9wdh9 druid 2021-11-26T04:07:54,892 ERROR [qtp555864569-122] org.apache.druid.server.router.QueryHostFinder - No backup found for serviceName[druid/broker]. Using default[druid/broker]
druid-router-7f6bcfb79c-9wdh9 druid 2021-11-26T04:07:54,893 ERROR [qtp555864569-124] org.apache.druid.server.router.QueryHostFinder - No server found for serviceName[druid/broker]. Using backup
druid-router-7f6bcfb79c-9wdh9 druid 2021-11-26T04:07:54,893 ERROR [qtp555864569-124] org.apache.druid.server.router.QueryHostFinder - No backup found for serviceName[druid/broker]. Using default[druid/broker]
druid-router-7f6bcfb79c-9wdh9 druid 2021-11-26T04:07:54,893 ERROR [qtp555864569-124] org.apache.druid.server.router.QueryHostFinder - Catastrophic failure! No servers found at all! Failing request!: {class=org.apache.druid.server.router.QueryHostFinder}
druid-router-7f6bcfb79c-9wdh9 druid 2021-11-26T04:07:54,893 ERROR [qtp555864569-122] org.apache.druid.server.router.QueryHostFinder - Catastrophic failure! No servers found at all! Failing request!: {class=org.apache.druid.server.router.QueryHostFinder}
druid-router-7f6bcfb79c-9wdh9 druid 2021-11-26T04:07:54,893 WARN [qtp555864569-124] org.eclipse.jetty.server.HttpChannel - /druid/v2/sql
druid-router-7f6bcfb79c-9wdh9 druid org.apache.druid.java.util.common.ISE: No default server found!
druid-router-7f6bcfb79c-9wdh9 druid 	at org.apache.druid.server.router.QueryHostFinder.assertServerFound(QueryHostFinder.java:158) ~[druid-services-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.apache.druid.server.router.QueryHostFinder.pickDefaultServer(QueryHostFinder.java:109) ~[druid-services-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.apache.druid.server.AsyncQueryForwardingServlet.service(AsyncQueryForwardingServlet.java:284) ~[druid-services-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-router-7f6bcfb79c-9wdh9 druid 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:82) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.apache.druid.server.security.AllowHttpMethodsResourceFilter.doFilter(AllowHttpMethodsResourceFilter.java:78) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:75) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.apache.druid.server.security.AllowAllAuthenticator$1.doFilter(AllowAllAuthenticator.java:84) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-router-7f6bcfb79c-9wdh9 druid 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]

Coordinator trace

druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:42,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Load Queues:
druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:42,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Server[:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served.
druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:47,842 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.LogUsedSegments - Found [0] used segments.
druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:47,843 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.DruidCoordinator - Caught exception, ignoring so that schedule keeps going.: {class=org.apache.druid.server.coordinator.DruidCoordinator, exceptionType=class java.lang.RuntimeException, exceptionMessage=org.apache.druid.java.util.common.IOE: No known server}
druid-coordinator-f4cd776f5-xj6g9 druid java.lang.RuntimeException: org.apache.druid.java.util.common.IOE: No known server
druid-coordinator-f4cd776f5-xj6g9 druid 	at org.apache.druid.client.indexing.HttpIndexingServiceClient.getTasks(HttpIndexingServiceClient.java:266) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-coordinator-f4cd776f5-xj6g9 druid 	at org.apache.druid.client.indexing.HttpIndexingServiceClient.getActiveTasks(HttpIndexingServiceClient.java:231) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-coordinator-f4cd776f5-xj6g9 druid 	at org.apache.druid.server.coordinator.KillStalePendingSegments.run(KillStalePendingSegments.java:55) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-coordinator-f4cd776f5-xj6g9 druid 	at org.apache.druid.server.coordinator.DruidCoordinator$DutiesRunnable.run(DruidCoordinator.java:910) [druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-coordinator-f4cd776f5-xj6g9 druid 	at org.apache.druid.server.coordinator.DruidCoordinator$2.call(DruidCoordinator.java:720) [druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-coordinator-f4cd776f5-xj6g9 druid 	at org.apache.druid.server.coordinator.DruidCoordinator$2.call(DruidCoordinator.java:713) [druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-coordinator-f4cd776f5-xj6g9 druid 	at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$4.run(ScheduledExecutors.java:163) [druid-core-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-coordinator-f4cd776f5-xj6g9 druid 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
druid-coordinator-f4cd776f5-xj6g9 druid 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_312]
druid-coordinator-f4cd776f5-xj6g9 druid 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
druid-coordinator-f4cd776f5-xj6g9 druid 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_312]
druid-coordinator-f4cd776f5-xj6g9 druid 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
druid-coordinator-f4cd776f5-xj6g9 druid 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
druid-coordinator-f4cd776f5-xj6g9 druid 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
druid-coordinator-f4cd776f5-xj6g9 druid Caused by: org.apache.druid.java.util.common.IOE: No known server
druid-coordinator-f4cd776f5-xj6g9 druid 	at org.apache.druid.discovery.DruidLeaderClient.getCurrentKnownLeader(DruidLeaderClient.java:267) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-coordinator-f4cd776f5-xj6g9 druid 	at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:122) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-coordinator-f4cd776f5-xj6g9 druid 	at org.apache.druid.client.indexing.HttpIndexingServiceClient.getTasks(HttpIndexingServiceClient.java:251) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-coordinator-f4cd776f5-xj6g9 druid 	... 13 more
druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:47,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.LogUsedSegments - Found [0] used segments.
druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:47,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.
druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:47,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - Metadata segments are not available. Cannot balance.
druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:47,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Load Queues:
druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:47,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Server[:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served.
druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:52,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.LogUsedSegments - Found [0] used segments.
druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:52,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.
druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:52,963 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - Metadata segments are not available. Cannot balance.

Broker trace

druid-broker-579ffb6b97-jtc5b druid 2021-11-26T01:24:36,788 INFO [ServerInventoryView-0] org.apache.druid.client.BatchServerInventoryView - Server Disappeared[DruidServerMetadata{name=':8083', hostAndPort=':8083', hostAndTlsPort='null', maxSize=300000000000, tier='_default_tier', type=historical, priority=0}]
druid-broker-579ffb6b97-jtc5b druid 2021-11-26T01:24:36,788 INFO [ServerInventoryView-0] org.apache.druid.client.BatchServerInventoryView - Server Disappeared[DruidServerMetadata{name=':8083', hostAndPort=':8083', hostAndTlsPort='null', maxSize=300000000000, tier='_default_tier', type=historical, priority=0}]
druid-broker-579ffb6b97-jtc5b druid 2021-11-26T01:24:36,832 ERROR [NodeRoleWatcher[COORDINATOR]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher - Unknown error in node watcher of role[coordinator].
druid-broker-579ffb6b97-jtc5b druid java.lang.RuntimeException: java.net.URISyntaxException: Expected hostname at index 7: http://:8081
druid-broker-579ffb6b97-jtc5b druid 	at org.apache.druid.server.DruidNode.getUriToUse(DruidNode.java:292) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-broker-579ffb6b97-jtc5b druid 	at org.apache.druid.discovery.BaseNodeRoleWatcher.childAdded(BaseNodeRoleWatcher.java:131) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-broker-579ffb6b97-jtc5b druid 	at org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher.childAdded(CuratorDruidNodeDiscoveryProvider.java:271) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-broker-579ffb6b97-jtc5b druid 	at org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher.handleChildEvent(CuratorDruidNodeDiscoveryProvider.java:237) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-broker-579ffb6b97-jtc5b druid 	at org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher.lambda$new$0(CuratorDruidNodeDiscoveryProvider.java:205) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-broker-579ffb6b97-jtc5b druid 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:538) [curator-recipes-4.3.0.jar:4.3.0]
druid-broker-579ffb6b97-jtc5b druid 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:532) [curator-recipes-4.3.0.jar:4.3.0]
druid-broker-579ffb6b97-jtc5b druid 	at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100) [curator-framework-4.3.0.jar:4.3.0]
druid-broker-579ffb6b97-jtc5b druid 	at org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) [curator-client-4.3.0.jar:?]
druid-broker-579ffb6b97-jtc5b druid 	at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92) [curator-framework-4.3.0.jar:4.3.0]
druid-broker-579ffb6b97-jtc5b druid 	at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:530) [curator-recipes-4.3.0.jar:4.3.0]
druid-broker-579ffb6b97-jtc5b druid 	at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-4.3.0.jar:4.3.0]
druid-broker-579ffb6b97-jtc5b druid 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:808) [curator-recipes-4.3.0.jar:4.3.0]
druid-broker-579ffb6b97-jtc5b druid 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
druid-broker-579ffb6b97-jtc5b druid 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_312]
druid-broker-579ffb6b97-jtc5b druid 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
druid-broker-579ffb6b97-jtc5b druid 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_312]
druid-broker-579ffb6b97-jtc5b druid 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
druid-broker-579ffb6b97-jtc5b druid 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
druid-broker-579ffb6b97-jtc5b druid 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
druid-broker-579ffb6b97-jtc5b druid Caused by: java.net.URISyntaxException: Expected hostname at index 7: http://:8081
druid-broker-579ffb6b97-jtc5b druid 	at java.net.URI$Parser.fail(URI.java:2847) ~[?:1.8.0_312]
druid-broker-579ffb6b97-jtc5b druid 	at java.net.URI$Parser.failExpecting(URI.java:2853) ~[?:1.8.0_312]
druid-broker-579ffb6b97-jtc5b druid 	at java.net.URI$Parser.parseHostname(URI.java:3389) ~[?:1.8.0_312]
druid-broker-579ffb6b97-jtc5b druid 	at java.net.URI$Parser.parseServer(URI.java:3235) ~[?:1.8.0_312]
druid-broker-579ffb6b97-jtc5b druid 	at java.net.URI$Parser.parseAuthority(URI.java:3154) ~[?:1.8.0_312]
druid-broker-579ffb6b97-jtc5b druid 	at java.net.URI$Parser.parseHierarchical(URI.java:3096) ~[?:1.8.0_312]
druid-broker-579ffb6b97-jtc5b druid 	at java.net.URI$Parser.parse(URI.java:3052) ~[?:1.8.0_312]
druid-broker-579ffb6b97-jtc5b druid 	at java.net.URI.<init>(URI.java:673) ~[?:1.8.0_312]
druid-broker-579ffb6b97-jtc5b druid 	at org.apache.druid.server.DruidNode.getUriToUse(DruidNode.java:289) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-broker-579ffb6b97-jtc5b druid 	... 19 more
druid-broker-579ffb6b97-jtc5b druid 2021-11-26T01:24:36,838 INFO [ServerInventoryView-0] org.apache.druid.client.BatchServerInventoryView - New Server[DruidServerMetadata{name=':8083', hostAndPort=':8083', hostAndTlsPort='null', maxSize=300000000000, tier='_default_tier', type=historical, priority=0}]
druid-broker-579ffb6b97-jtc5b druid 2021-11-26T01:24:36,855 INFO [ServerInventoryView-0] org.apache.druid.client.BatchServerInventoryView - New Server[DruidServerMetadata{name=':8083', hostAndPort=':8083', hostAndTlsPort='null', maxSize=300000000000, tier='_default_tier', type=historical, priority=0}]

Historical

druid-historical-0 druid 2021-11-26T01:24:36,660 INFO [Announcer-0] org.apache.druid.curator.announcement.Announcer - Node[/druid/announcements/:8083] dropped, reinstating.
druid-historical-0 druid 2021-11-26T01:24:36,799 ERROR [NodeRoleWatcher[COORDINATOR]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher - Unknown error in node watcher of role[coordinator].
druid-historical-0 druid java.lang.RuntimeException: java.net.URISyntaxException: Expected hostname at index 7: http://:8081
druid-historical-0 druid 	at org.apache.druid.server.DruidNode.getUriToUse(DruidNode.java:292) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-historical-0 druid 	at org.apache.druid.discovery.BaseNodeRoleWatcher.childAdded(BaseNodeRoleWatcher.java:131) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-historical-0 druid 	at org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher.childAdded(CuratorDruidNodeDiscoveryProvider.java:271) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-historical-0 druid 	at org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher.handleChildEvent(CuratorDruidNodeDiscoveryProvider.java:237) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-historical-0 druid 	at org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher.lambda$new$0(CuratorDruidNodeDiscoveryProvider.java:205) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-historical-0 druid 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:538) [curator-recipes-4.3.0.jar:4.3.0]
druid-historical-0 druid 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:532) [curator-recipes-4.3.0.jar:4.3.0]
druid-historical-0 druid 	at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100) [curator-framework-4.3.0.jar:4.3.0]
druid-historical-0 druid 	at org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) [curator-client-4.3.0.jar:?]
druid-historical-0 druid 	at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92) [curator-framework-4.3.0.jar:4.3.0]
druid-historical-0 druid 	at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:530) [curator-recipes-4.3.0.jar:4.3.0]
druid-historical-0 druid 	at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-4.3.0.jar:4.3.0]
druid-historical-0 druid 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:808) [curator-recipes-4.3.0.jar:4.3.0]
druid-historical-0 druid 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
druid-historical-0 druid 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_312]
druid-historical-0 druid 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
druid-historical-0 druid 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_312]
druid-historical-0 druid 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
druid-historical-0 druid 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
druid-historical-0 druid 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
druid-historical-0 druid Caused by: java.net.URISyntaxException: Expected hostname at index 7: http://:8081
druid-historical-0 druid 	at java.net.URI$Parser.fail(URI.java:2847) ~[?:1.8.0_312]
druid-historical-0 druid 	at java.net.URI$Parser.failExpecting(URI.java:2853) ~[?:1.8.0_312]
druid-historical-0 druid 	at java.net.URI$Parser.parseHostname(URI.java:3389) ~[?:1.8.0_312]
druid-historical-0 druid 	at java.net.URI$Parser.parseServer(URI.java:3235) ~[?:1.8.0_312]
druid-historical-0 druid 	at java.net.URI$Parser.parseAuthority(URI.java:3154) ~[?:1.8.0_312]
druid-historical-0 druid 	at java.net.URI$Parser.parseHierarchical(URI.java:3096) ~[?:1.8.0_312]
druid-historical-0 druid 	at java.net.URI$Parser.parse(URI.java:3052) ~[?:1.8.0_312]
druid-historical-0 druid 	at java.net.URI.<init>(URI.java:673) ~[?:1.8.0_312]
druid-historical-0 druid 	at org.apache.druid.server.DruidNode.getUriToUse(DruidNode.java:289) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT]
druid-historical-0 druid 	... 19 more

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:6 (2 by maintainers)

github_iconTop GitHub Comments

2reactions
alborotogarciacommented, Dec 4, 2021

You were absolutely right @henriquekops, though i used oracle jdk container instead ! I’ve just figured out running the container, and it complained wrt ip commands…

Thanks @asdf2014 @henriquekops !

0reactions
henriquekopscommented, Dec 3, 2021

I got into this same problem when creating a custom image for Druid v0.20.0.

You’re using a custom Druid image as well, right?

...
containers:
        - name: druid
          image: "alvarogg777/druid:0.22.0"
...

Is your image Ubuntu based? Because if it is, you should check if druid.sh dependencies are fulfilled.

In my case, I had to apt install iproute2 for ip commands, otherwise I would get this exact same error:

...
...    Caused by: java.net.URISyntaxException: Expected hostname at index 7: http://:8081
...
Read more comments on GitHub >

github_iconTop Results From Across the Web

Scan Helm charts for Kubernetes misconfigurations with ...
Learn how to automate security scanning of Helm charts for Kubernetes misconfigurations with Bridgecrew's open-source scanner, Checkov.
Read more >
5 common Kubernetes misconfigs and how to fix them
In addition to analyzing the state of open source Helm charts, we wanted to dig into the most common misconfigurations found in Kubernetes ......
Read more >
Applying Kubernetes Security Best Practices to Helm Charts
Applying Kubernetes Security Best Practices to Helm Charts. Most Helm charts are not secure by default. It's vital for developers to secure Helm ......
Read more >
Make Your Own Helm Chart. Part III - Level Up Coding
Part III : Deployment Using Helm Package Manager ... cost you time and probably some misconfiguration that leads to some problem later on....
Read more >
Charts - Helm V2
NOTE: Whereas Helm Classic and Deployment Manager were both very GitHub oriented when it ... A description of the application or service the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

[Proposal] Improve memory estimations in `OnHeapIncrementalIndex`

Next page

Batch ingestion using SQL INSERT