Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Reducing Partition count - server does not host this topic-partition

See original GitHub issue

I had 4 partitions and 3 Pinot servers running. And noticed that only 3 partitions are getting consumed as I only have 3 servers. So I reduced the partition count from 4 to 3 in kafka. After that I am seeing errors in one of the Pinot server, other servers are fine. Restarts not fixing this issue.

my-pinot-server-2 server 2021/02/03 21:22:40.400 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:40.503 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:40.605 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:40.707 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:40.809 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:40.911 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:41.014 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:41.116 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:41.218 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:41.328 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:41.430 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:41.532 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:41.635 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.

Issue Analytics

State:
Created 3 years ago
Comments:6 (3 by maintainers)

Top GitHub Comments

1reaction

xiangfu0commented, Feb 4, 2021

Thanks @fx19880617 for the suggestions. I tried Ideal state and can see all the 4 partitions are getting consumed as shown below which is good. However there is an issue when you shrink Kafka partition i.e delete and recreate a topic with less number of partition than before. When that happens Pinot server keeps on ERROR-ing. Even though everything works the error is cluttering the console also not sure whether this have any other performance impact.
    "mylog__0__314__20210203T1856Z" : {
      "Server_my-pinot-server-2.my-pinot-server-headless.logging.svc.cluster.local_8098" : "CONSUMING"
    },
    "mylog__1__7__20210204T0043Z" : {
      "Server_my-pinot-server-0.my-pinot-server-headless.logging.svc.cluster.local_8098" : "CONSUMING"
    },
    "mylog__2__7__20210204T0043Z" : {
      "Server_my-pinot-server-1.my-pinot-server-headless.logging.svc.cluster.local_8098" : "CONSUMING"
    },
    "mylog__3__0__20210203T1927Z" : {
      "Server_my-pinot-server-2.my-pinot-server-headless.logging.svc.cluster.local_8098" : "CONSUMING"
    }

By design, Pinot is not handling the shrinking case. I think you need to manually delete the consuming segment mylog__3__0__20210203T1927Z. Also delete topic and recreate may mess up the consumer offset, @npawar may have more saying about this.

0reactions

pabrahamusacommented, Feb 5, 2021

Closing ticket as this is not supported feature.